From: nathaniel@...
Date: 2003-01-28T15:09:45+09:00
Subject: Re: Test::Unit -> order of tests?

Matt Armstrong [mailto:matt@lickey.com] wrote:

> > The problem is that there is no good way to run the tests in the
order 
> > defined.
> 
> class TestCase
>   class << self
>     def method_added(symbol)
>       # how would matz do this Perlish test?
>       if symbol.id2name =~ /^test_/
> 	@test_methods ||= []
> 	@test_methods << symbol
>       end
>     end
>     attr_reader :test_methods
>   end
> end
> 
> class MyTestCase < TestCase
>   def test_zzz; end
>   def test_mmm; end
>   def test_aaa; end
> end
> 
> class MyTestCase2 < TestCase
>   def test_zzz2; end
>   def test_mmm2; end
>   def test_aaa2; end
> end
> 
> p MyTestCase.test_methods
> #=> [:test_zzz, :test_mmm, :test_aaa]
> p MyTestCase2.test_methods
> #=> [:test_zzz2, :test_mmm2, :test_aaa2]

That's pretty slick! I had a feeling as soon as I said it wouldn't work
that someone would propose a solution. There might be an issue with
doing this, though - after a little IRB'ing here, it appears that
including a module will not trigger #method_added for each method of the
module. But perhaps there's a way around that, too :-)

Also, and it could just be me being over-cautious, but adding test
methods by trapping their definition seems to be more vulnerable to
problems than waiting until it's time to run them and _then_ gathering
them from where I expect them to be. I don't have to worry about missing
anything, as Ruby is collecting it all for me. But I'm a worrywart.


> > Anyhow, on running tests in (pseudo-)random order, that's one of the

> > test orderings I'd like to implement in the future. The only problem

> > with it is, if things fail when you're running them randomly, you'll

> > want to know what order they were run in that particular time so you

> > can fix the problem. Otherwise it might be more frustrating than 
> > helpful.
> 
> I can think of 2 solutions here:
> 
>     - Test::Unit supports saving the order of the tests it runs in a
>       text file, and can parse that file and run the tests in the same
>       order.
>     - Test::Unit includes its own random number generator, spits
>       out the random seed it is using, and can support seeding its RNG
>       to a particular number.  (you can't use Ruby's RNG since tests
>       themselves might use it)

I think I like option #2 best, as it appears at first blush to be
simplest to provide.

Finally, as there seems to be so much interest in this, I'll mentally
bump it up a few levels in priority.

Thanks to everyone for their excellent input,


Nathaniel

<:((><
+ - -
| RoleModel Software, Inc.
| EQUIP VI