[ruby-core:39325] Re: RubySpec vs CRuby's test/...

I'll jump in with some context from the JRuby perspective.

JRuby runs more tests/suites than any other Ruby implementation. We
run the old Rubicon tests...MRI's tests...RubySpec...ruby_test...old
minirunit tests from JRuby's past...even BFTS until a year ago or so.
We are very sensitive to the issues of dealing with many test suites.

RubySpec is by far the easiest and cleanest to:

* use
* modify
* filter for things we don't pass yet (mspec's "tags" are
beautiful...Brian Ford did an amazing job putting mspec together)
* understand

No other test suite even comes close to the ease of use and utility of
RubySpec today. Because of that, we have actively (but slowly) been
trying to transition to using only RubySpec for all our testing, and
as Ruby behavioral bugs are reported we add them to RubySpec.

There are many reasons why tests that live solely in MRI's repository
are problematic for other implementations:

* Impl-specific tests, as Matz mentioned
* Test harnesses specific to MRI's build
* Inability to easily filter specific tests to get a green partial run
* Nobody but MRI committers can contribute easily (!!!!!!!!!)

A perfect example of this is the Psych library. JRuby implements Psych
atop SnakeYAML. We reuse all the Psych .rb files as-is and only
reimplement what would be in C. And we run the Psych tests, of which
we have a copy in the JRuby repository.

But all the issues above affect us. Some of Psych's tests have minor
impl-specific details. They're test/unit based, so we need a test/unit
harness to run them. There's no way to filter out any impl-specific,
unsupported, or undefined behavior tests. And there's no way for us to
add new tests other than submitting pull requests and praying.

The result of these issues is that JRuby and Rubinius simply do not
contribute tests to MRI, and run MRI's tests less and less (Rubinius
doesn't run *any* as far as I know). It means that JRuby and Rubinius
contributors must do the test-writing work of all the MRI contributors
*again* to keep RubySpec up to date. And it means that MRI is always
drifting away from RubySpec, making a moving target even more
difficult to chase for JRuby and Rubinius.

It is my sincere belief that if ruby-core cares about having a rich
Ruby implementation ecosystem, more effort should be put into making
RubySpec the standard suite of tests, with MRI's tests reserved for
impl-specific tests (similar to how I see JRuby going over time).
Conversely, I worry that the failure of ruby-core to adopt RubySpec
indicates a complete disregard for other Ruby implementations and the
contributions they make to the Ruby community. This makes me very sad,
and I hope it's not true.

I understand the desire for ruby-core to maintain the status quo, and
I sympathize wrt the language barrier involved in contributing to
RubySpec. But maintaining Ruby behavioral test suites in the current
ruby-core way makes life much more difficult for alternative
implementations...and we have enough challenges already.

- Charlie