From: Ryan Davis <ryand-ruby@...>
Date: 2010-10-10T19:25:06+09:00
Subject: Re: minitest randomization


On Oct 9, 2010, at 16:50 , Alex Chaffee wrote:

> I guess this comes down to idempotency. I expect that if I do
> something twice in a row I will get the same result. Randomizing by
> default breaks this expectation. It's astonishing, therefore bad, no
> matter how good from a theoretical standpoint, and especially
> astonishing when people have 10+ years of xUnit and its heirs building
> these expectations.

Idempotency is a red-herring. There is nothing about the xunit family/philosophy of tools (or any other test tool that I have used or studied--except rspec) that suggests that test order must be run in the order defined (or must be run in any order at all). Just look at the new tools coming out that distribute and multithread/process your tests and you can see that right there, the notion has to be thrown out the window by design.

As for your astonishment, I thought it was pretty well addressed in the first line of my reply: "Really? I think preventing test order dependency has a very practical effect". If you're still astonished after that, then you're probably misusing the word.

-----

At this point I'm going to cut much of your reply and everything I've written so far in response and cut to the chase:

> And I'm not even disagreeing with your observation! I agree that there
> should be a randomizing mode, and that people should run it fairly
> often. Just not all the time and not without a config or command-line
> option to turn it off.

Apparently this is the crux of our disagreement:

I __do__ think that people should randomize their tests a MAJORITY of the time and turn it off TEMPORARILY when they need to sort out an issue. If it wasn't random by default, it wouldn't happen at all.

If you disagree with that (and still want to use minitest), you ALREADY have multiple ways to fix it for your own tests:

1) define ::test_order to return :sorted. That'd be your "config" suggestion above.
2) use --seed when you want the order to be fixed, via TESTOPTS if you're using rake. And that'd be your command line option...

If you want a third option available, feel free to propose it and I'll gladly consider it.

P.S. I'm still mulling over Steve Klabnik's suggestion that the output be sorted. I think it could be very confusing when you do have test dependency errors but that there might be some way to mitigate the confusion. I'd like to hear what you think about his suggestion.