From: Nathaniel Talbott <nathaniel@...>
Date: 2003-01-28T11:57:34+09:00
Subject: Re: Test::Unit -> order of tests?

Warren Brown [mailto:wkb@airmail.net] wrote:

> > To implement this, I would suggest renaming all your test_XXX
methods 
> > to subtest_XXX and have a single test_all method that calls the 
> > subtest_XXX methods in the proper order.  Have your setup method 
> > create the database and your teardown method delete it.
> 
>     This is a *great* solution for test cases that must be 
> run in a certain order.

Amen!


>     The problem that I run into is when I have some test 
> cases that test simple building-block methods, and another 
> big test case that tests a method that calls those 
> building-block methods.  If a bug pops up in one of the 
> building-block methods, it would be nice to have the 
> building-block's test case catch it rather than the big test 
> case, since the big test case may fail in an obscure way if 
> the building-blocks it depends on are untrustworthy.  The 
> tests *can* be run in any order, it's just more helpful to 
> have them run in the given order.
> 
>     My suggestion is to have the test cases execute in the 
> order they are encountered, but add a 'randomize' capability 
> that can be specified from the command line to run the test 
> cases in random order.  I can see no benefit in having the 
> test cases execute in alphabetical order, but if there is a 
> benefit, that too could be a command line option.

The problem is that there is no good way to run the tests in the order
defined. IIRC, Ruby doesn't make an guarantees about the ordering of
methods returned from calls like Module#public_instance_methods, so for
consistencies sake it's much better to run methods in alphabetical
order. While it's not good for the code (and tests) to depend on test
ordering, it's still good to have tests run in a consistent, predictable
order so that us programmers know what to expect. I hope the distinction
between those two things is at least as clear as muddy water :-/.

Anyhow, on running tests in (pseudo-)random order, that's one of the
test orderings I'd like to implement in the future. The only problem
with it is, if things fail when you're running them randomly, you'll
want to know what order they were run in that particular time so you can
fix the problem. Otherwise it might be more frustrating than helpful.


Nathaniel

<:((><
+ - -
| RoleModel Software, Inc.
| EQUIP VI