Re: Smoke testing Ruby

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Charles Oliver Nutter wrote:
> Hugh Sasse wrote:
>> Ruby used to have the Triple-R project based on Rubicon: see
>>
>> http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/24820
>>
>> but this seems to no longer exist.  The nearest thing we have to
>> platform coverage now is Tattle
>>
>> http://tattle.rubygarden.org./
>>
>> but this isn't a complete list of which tests of some suite succeeded,
>> just which platforms support Ruby and have tattle installed.
>>
>> Basically, there are a number of projects for testing ruby, and maybe
>> when there is a winner the idea will resurface.  I hope so.
> 
> Rubicon still exists on RubyForge, but the version we've modified in
> JRuby source is more clean and up-to-date (and we've added a lot to it).
> 
> Daniel Berger has a selection of tests somewhere, I forget where. We
> have a copy in JRuby.
> 
> BFTS is another selection of tests, but only hits a few core classes
> (albeit pretty well).
> 
> Wilson mentioned Rubinius's specs.
> 
> And then JRuby takes all these plus a bunch of our own and runs them as
> part of our build and on the continuous integration server. The tests
> are run once with the compiler enabled and once without.
> 
> Of course there's the tests in Ruby's own source.
> 
> The CI server is here: http://jruby.thresher.com/start.action
> 
> It got a little flaky over the summer but it's settled back down in the
> past couple weeks. It's been a big help for us tracking stability as we
> improve compatibility and performance.
> 
> JRuby's entire test suite is available here:
> 
> http://svn.codehaus.org/jruby/trunk/jruby/test/
> 
> It's not super organized at the moment. Those under the root of the dir
> are a mix of minirunit and test/unit tests and were almost all created
> by JRuby developers over the years. Some of the tests are
> JRuby-specific, testing Java functionality and the like. There are some
> tests in Java code for the Java-facing JRuby APIs. Then there are tests
> under test/rubicon (our cleaned-up, converted, expanded rubicon tests),
> MRI (test/mri/sample/test.rb and test/externals/mri/*), BFTS
> (test/externals/bfts/*), Daniel Berger's stuff
> (test/externals/ruby_test), Rubinius specs
> (test/externals/rubinius/spec), and our set of benchmarks, including a
> copy of YARVs (test/bench/*).
> 
> See also this article comparing test suites for Ruby used in JRuby:
> 
> http://chneukirchen.org/blog/archive/2007/05/analyzing-the-jruby-test-suite.html
> 
> 
> I'd love to see this all some day move to a single standard location.
> 
> - Charlie
> 
> 

I thought that was the plan -- a merge of all the Ruby test suites,
including the BFTS. Maybe the trick is to get whoever has the largest
test suite to import the smaller ones??

It shouldn't matter that there's a mix of RSpec-style, Test::Unit style,
etc. tests. And while we're on the subject, I also think it's time for a
Big Formal Benchmark Suite. In fact, maybe the two should be the same --
add timer calls to all the tests and assertions/"shoulds" to all the
benchmarks.:)

Your list appears to have everything I know about, so perhaps it is
already done.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG1XzC8fKMegVjSM8RAkTFAJ4n32iAezG5Ugcj7QCU/mih7QkkvACfVcRI
jiZ5W2qnXO7F6bUYvVILm/I=
=kwZo
-----END PGP SIGNATURE-----