From: Alex Chaffee <alexch@...>
Date: 2010-10-10T13:38:50+09:00
Subject: Re: minitest randomization

--Apple-Mail-40--876620571
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

I guess this comes down to idempotency. I expect that if I do
something twice in a row I will get the same result. Randomizing by
default breaks this expectation. It's astonishing, therefore bad, no
matter how good from a theoretical standpoint, and especially
astonishing when people have 10+ years of xUnit and its heirs building
these expectations.

Thanks for your detailed reply, but I should point out that your
responses didn't address either my idempotency or astonishment
arguments.

> Why aren't you using --seed when you rerun your specs?

Because I say "rake" to run my tests, and seed is not preserved. How
are you running yours?

> If you use --seed with the previous value, all of your complaint about =
having to "squint" to find your previous failure goes away.

Well, no, not *all* of them :-)

And I wasn't being metaphorical. When I'm poring over a console full
of fail, I squint. And sometimes I sigh.

>> * make test randomization an option ("randomize")
>=20
> It is an option on a class by class basis. See `ri =
MiniTest::Unit::TestCase::test_order`.

Yeah, I saw that before, but that doesn't fix my core complaint unless
I hack minitest to always run in consistent order. Hence my request
for a patch.

And you're right, it seems that that patch is trivial for
MiniTest::Spec, and probably not too hard for
MiniTest::Unit::TestCase.

BTW your docs should reflect that for MiniTest::Specs, :alpha and
:sorted both really mean :defined -- or whatever you want to call the
traditional "order in which they occur in the file" (which is often
roughly in order of complexity, so earlier failures are often more
essential, so should be fixed first). (And for
MiniTest::Unit::TestCases, too, but only in Ruby 1.9.)

> Anecdotally (unfortunately, nothing I can go into in great detail), =
I've seen far too many projects with test order dependencies, which is =
the reason that feature went into the library in the first place.

I'm sure you thought you had a good reason for being inconsistent. But
in this case, where the library ships with core Ruby and is supposed
to be an invisible drop-in replacement for Test::Unit, it's a bridge
too far.

And I'm not even disagreeing with your observation! I agree that there
should be a randomizing mode, and that people should run it fairly
often. Just not all the time and not without a config or command-line
option to turn it off.

> Unfortunately, we need to output the seed value at the beginning in =
the case that your tests not only fail, but crash (like when you're =
Aaron Patterson and you're working on C extensions instead of writing =
ruby like a good person).

Wait, are you saying the reason we all have to look at console spam is
so that Aaron doesn't have to type "--verbose" when he's writing C in
Ruby? Does he have compromising photos of you with the maid or
something? :-)

> I still think that random tests/specs are stronger tests/specs and =
completely disagree with you that "most well-factored OO [test] code =
these days does not exhibit isolation problems" on the basis that most =
OO [test] code is not well-factored. minitest's test dir flays at 535. =
Wrong's tests flay at 1150.

That's a nice debate trick -- insert a qualifier and then disagree
with it, not with what I actually said :-)

Anecdotally, when you see test order dependency problems, are they
because the tests are not isolated or because the production code
isn't? I was talking about production OO code, not test code (which is
more like a bunch of functions than like an object anyway).

Flay's a nice tool but the threshold for DRY in test code is higher
than in production code. Test code needs to be understandable above
all else, so you can hone in on the scenario leading up to a
file-and-line failure, and that can mean duplication to some extent is
desirable. That doesn't mean the tests aren't isolated.

- A

--=20
Alex Chaffee - alex@stinky.com - http://alexch.github.com
Stalk me: http://friendfeed.com/alexch | http://twitter.com/alexch |
http://alexch.tumblr.com


--Apple-Mail-40--876620571--