[ruby-core:39414] Re: RubySpec vs CRuby's test/...

I realize that I'm a small fish in this ocean, but for every release
of Ruby 1.8.6 that I have done, I have tested against RubySpec prior
to making the release.

Also, in my role at Engine Yard, I am often building packages for the
various Rubies (CRuby 1.8.x/1.9.x, REE, JRuby, Rubinius). Sometimes
these are straight builds. Sometimes we add patches. It depends on the
build. I try to test these builds thoroughly, and that is especially
important if it is a build that has patches applied to it. For
CRubies, this generally involves running both the test/* suite, and
RubySpec. I have found that I need to run both, because there have
been times when differing test coverage has resulted in one suite
running cleanly while the other has an unexpected failure. I have
always found that somewhat frustrating, and I have wondered for a long
time why Ruby behavioral tests are not being added directly to
RubySpec.

The benefit to doing so seems clear to me. While the testing
antecedents relied on test/* in order to regression test Ruby, back
when there was only a single implementation, the world has changed
since then. Ruby the language is bigger than any one implementation.
So while tests that are specific to MRI should clearly go in test/*,
tests which verify that an implementation has the correct Ruby
behavior would be more globally useful if they were added to RubySpec,
as that test suite is intended to support the language as a whole, and
not any one specific implementation.

If there is an untested old behavior, or a new behavior that needs a
corresponding test, placing that test in RubySpec benefits all Ruby
implementations. I've done it in the past, when working on a 1.8.6
release. That is actually what I originally got RubySpec commit access
for.

I was thinking about Urabe's comment that RubySpec only describes
current CRuby behavior, but doesn't proscribe what future behavior
should be. That sort of prescriptive behavior could be implemented
using a guard. A spec that isn't expected to pass currently, on any
version of Ruby, but that describes a behavior that is desired could
be flagged as such. Then, at such a time as an implementation of Ruby
has that behavior implemented, the guard is changed to reflect this.

I have been contemplating a role to take on with ruby-core after the
1.8.6 era closes for good. If there would be value in it, maybe I
could start working on identifying and duplicating tests that are in
test/*, but are lacking in RubySpec.

However, I struggle to understand the fundamental perspective that it
is better for Ruby (not MRI-specific) tests to go into test/* instead
of into RubySpec. Perhaps I am misunderstanding some important
component to the objections?

I understand not being comfortable with git. I will admit that I curse
git sometimes. But for all of the basic capabilities, git is trivial
to learn and to use.

I also understand not liking RSpec, and preferring the declarative
test/unit style. I prefer the test/unit style, personally. But RSpec
is also pretty trivial to learn sufficiently to make it a useful tool.

I am far from being a RubySpec expert, RSpec expert, or git expert,
but I would also offer to help anyone in ruby-core who wants to try
using RubySpec or adding a test to RubySpec, if they need someone who
has done it before to provide some assistance.


Thanks much,

Kirk Haines