From: Ryan Davis <ryand-ruby@...>
Date: 2010-10-07T09:53:21+09:00
Subject: Re: minitest randomization


On Oct 6, 2010, at 15:03 , Alex Chaffee wrote:

> This has no practical effect on an all-succeeding suite, or one in which a single test is failing.

Really? I think preventing test order dependency has a very practical effect.

> If I submit a patch to do the following, will you accept it?
> 
> * make test randomization an option ("randomize")

It is an option on a class by class basis. See `ri MiniTest::Unit::TestCase::test_order`.

> * make the default for that option be "off"

I actually disagree with this, but I think that is less important atm.

Why aren't you using --seed when you rerun your specs? If you use --seed with the previous value, all of your complaint about having to "squint" to find your previous failure goes away.

Phrogz contributed some patches for 1.6.0 that ensured that specs could be run in defined order by serializing their names. Combine that with ::test_order above and you have exactly what you want. Easy-peasy!

I could have _sworn_ that Spec already overrode ::test_order to be :sorted, but I have absolutely no commit to that effect. I guess Phrogz was explicitly defining that on his specs. 

I still think that random tests/specs are stronger tests/specs and completely disagree with you that "most well-factored OO [test] code these days does not exhibit isolation problems" on the basis that most OO [test] code is not well-factored. minitest's test dir flays at 535. Wrong's tests flay at 1150. 

Anecdotally (unfortunately, nothing I can go into in great detail), I've seen far too many projects with test order dependencies, which is the reason that feature went into the library in the first place.

In the cases where I'm doing big refactorings and getting huge swaths of errors and the randomization is bugging me, I'll temporarily pop in test_order to sort them and get through my work. But I always remove it before committing.

> * output the command-line options only if one of the following is true:
> ** "verbose" is on
> ** "randomize" is on AND there was a failure

Unfortunately, we need to output the seed value at the beginning in the case that your tests not only fail, but crash (like when you're Aaron Patterson and you're working on C extensions instead of writing ruby like a good person).

I can see it going the verbose route, but maybe there needs to be a middle level verbosity? Full verbosity is damn noisy/annoying unless you need those test method times (and then it is _awesome_).