Re: Smoke testing Ruby

Hugh Sasse wrote:
> Ruby used to have the Triple-R project based on Rubicon: see
> 
> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/24820
> 
> but this seems to no longer exist.  The nearest thing we have to 
> platform coverage now is Tattle
> 
> http://tattle.rubygarden.org./
> 
> but this isn't a complete list of which tests of some suite succeeded,
> just which platforms support Ruby and have tattle installed.
> 
> Basically, there are a number of projects for testing ruby, and maybe
> when there is a winner the idea will resurface.  I hope so.

Rubicon still exists on RubyForge, but the version we've modified in 
JRuby source is more clean and up-to-date (and we've added a lot to it).

Daniel Berger has a selection of tests somewhere, I forget where. We 
have a copy in JRuby.

BFTS is another selection of tests, but only hits a few core classes 
(albeit pretty well).

Wilson mentioned Rubinius's specs.

And then JRuby takes all these plus a bunch of our own and runs them as 
part of our build and on the continuous integration server. The tests 
are run once with the compiler enabled and once without.

Of course there's the tests in Ruby's own source.

The CI server is here: http://jruby.thresher.com/start.action

It got a little flaky over the summer but it's settled back down in the 
past couple weeks. It's been a big help for us tracking stability as we 
improve compatibility and performance.

JRuby's entire test suite is available here:

http://svn.codehaus.org/jruby/trunk/jruby/test/

It's not super organized at the moment. Those under the root of the dir 
are a mix of minirunit and test/unit tests and were almost all created 
by JRuby developers over the years. Some of the tests are 
JRuby-specific, testing Java functionality and the like. There are some 
tests in Java code for the Java-facing JRuby APIs. Then there are tests 
under test/rubicon (our cleaned-up, converted, expanded rubicon tests), 
MRI (test/mri/sample/test.rb and test/externals/mri/*), BFTS 
(test/externals/bfts/*), Daniel Berger's stuff 
(test/externals/ruby_test), Rubinius specs 
(test/externals/rubinius/spec), and our set of benchmarks, including a 
copy of YARVs (test/bench/*).

See also this article comparing test suites for Ruby used in JRuby:

http://chneukirchen.org/blog/archive/2007/05/analyzing-the-jruby-test-suite.html

I'd love to see this all some day move to a single standard location.

- Charlie