[ruby-core:67371] Re: Future of test suites for Ruby

On Tue, Jan 6, 2015 at 1:50 AM, "Martin J. D=C3=BCrst"
<[e-mail]> wrote:
> Ruby always has had an inclusive rather than prescriptive culture (e.g. w=
hen
> compared to Python), so the fact that there are two test suites may not b=
e
> that bad. Also, mixing and matching things, and combining stuff from
> different sources, is something that is very frequent in Japanese culture=
.

I'm getting the impression that half the folks like RubySpec better
and half the folks like the MRI suite better, and both have fair
reasons.

At the moment there's a tremendous amount of inertia behind the MRI
suite and far more tests go there than to RubySpec over time. New
features, modified features, and bugs all have tests added to MRI's
suite first...probably because it's "there" in the repository.

That is, of course, why JRuby considers MRI its primary compatibility suite=
.

I have run coverage reports for both projects on JRuby.

MRI: http://headius.github.io/mri-coverage/

RubySpec: http://headius.github.io/rubyspec/-coverage/

Note that the aggregate numbers are not useful because RubySpec no
longer includes any tests for the standard library (they were
transplanted into Rubinius's "rubysl" projects) and JRuby does not run
all tests/specs from either suite.

Individual classes and methods do have useful numbers though. For
example, RubyString (which implements String) is 85% covered by MRI's
suite and 79% covered by RubySpec.

RubyString under MRI suite:
http://headius.github.io/mri-coverage/org.jruby/RubyString.html#dn-a
Highlighted source:
http://headius.github.io/mri-coverage/org.jruby/RubyString.java.html

RubyString under RubySpec:
http://headius.github.io/rubyspec-coverage/org.jruby/RubyString.html#dn-a
Highlighted source:
http://headius.github.io/rubyspec-coverage/org.jruby/RubyString.java.html

Other cases have slightly more coverage running RubySpec.

Overall, coverage of the core classes seems roughly the same between
MRI's suite and RubySpec, but if either suite is to be the "winner" we
need reports like this to ensure we maintain test coverage.

> I think contributions to both MRI's test suite and to RubySpec are very
> welcome. We are getting lots of small but valuable contributions to the
> documentation from what I understand are people who use this as a way to
> study Ruby. Maybe something similar could work for tests; it would be
> somewhat more difficult than working on documentation but may have a simi=
lar
> educational effect and likewise produce very valuable improvements.

Perhaps the biggest benefit of RubySpec is that all specs are small
and contain an English specdoc string describing what they test.
Compare that to MRI where you may have one method called
"test_function" that tests 100 different behaviors and you are left
reading the code to figure it out.

The specdoc requirement is not necessarily a *good* think, though.
Today I was fixing some bugs in JRuby and found that the related
RubySpec's English description described the exact *opposite*
behavior. English descriptions, just like comments, must be
maintained. And if all specs must have a specdoc string, then
non-English speakers are required to compose formal specdoc English
just to write a test.

That said, if we broke up the large test methods in MRI's suite and
added one-liner comments indicating what they do (perhaps with rdoc
formatting?) it would go a long way toward better readability.

I'm taking notes on what an "MRI test suite coding guide" might look like.

- Charlie