Re: Test::Unit run order

On Jul 16, 9:55=A0pm, Ryan Davis <[e-mail]> wrote:
> On Jul 16, 2009, at 15:30 , Dylan wrote:
>
>
>
> > On Jul 16, 12:16 pm, Ryan Davis <[e-mail]> wrote:
> >> On Jul 15, 2009, at 10:15 , Dylan wrote:
>
> >>> Hey there! I had a question about the order tests are run in
> >>> Test::Unit. I have a class with about 10 tests and I'd like to be =A0
> >>> able
> >>> to better control the order they run in. Right now they just run
> >>> alphabetically [...]
>
> >> order dependent tests are a bug. every single test should be able to
> >> run and pass in isolation. eg:
>
> >> % ruby -Ilib test/test_something.rb -n test_one_test
>
> >> should always pass no matter what the file or test name is.
>
> > Which is nice in theory, but not always the best option. In my case,
> > I'm testing a web app, and I need to test different things in
> > different logins, so I'd like to order it so I log in to a normal user
> > account, run some tests, then log in to an admin account, run some
> > other tests, etc... without having to re-login for every test, which
> > wastes a lot of time and doesn't give me anything in return other than
> > being able to say "Ooh, my tests are independent" (I already have a
> > separate test for the login system). If the login system doesn't work,
> > none of the tests are going to pass even if I had separate login
> > scripts for each one.
>
> > So I either need a way to order the tests so I can use certain logins
> > for certain tests, or I need a way to create only 1 XML file from
> > CI_reporter from multiple suites [...]
>
> ..or you could deal with the real problem of having slow tests and =A0
> not the symptoms of it. I've talked about this a lot in the past[*], =A0
> so I'm not going to repeat myself here. I'll simply reassert:
>
> =A0 =A0_all_ tests should _always_ pass when run by themselves.
>
> and I'll expand on that:
>
> =A0 =A0_all_ tests should run fast enough that they're a pleasure to run =
=A0
> _all_ the time.
>
> [*] See my talks on Ruby Sadism, Electric Kool-ade Acid Testing, and =A0
> many others for related infos.

I would love to do that, but I have no idea how. The "real problem" of
my slow tests is caused by the very real requirement of logging in
every time, were they to be truly independent. I'm not in charge of
the bandwidth to the server, I'm not in charge of the log-in code, and
I'm not in charge of the "Lets make the tests take less than a day to
run" requirement. I'm testing a web app, and web apps by their very
nature have to transmit data back and forth, which takes more time
than any local machine doing operations. In fact, the "slow" tests
aren't all that slow were it a real user in a real situation. Running
so many tests takes long enough as it is, and I still don't see the
benefit of independence in my case (even so, I would still prefer it,
as I've said, but in my situation the tradeoffs do not seem to be
worth it. Nothing further gets tested, they run just the same as if I
logged in every time unless there is some major screw up with cookies/
cache, and the other tests aren't going to screw up because of a
previous one (or on the off-chance they do it would be easily
traceable back to the actual problem). Yes, I do realize that there is
a chance that not re-logging in every time might screw up some test,
but the chance of it is remote enough, and the problems that would
occur if it did happen are small enough, that the added benefit of
being able to get results back relatively quickly is (to me) worth it.

I am in fact trying to solve your second rule by removing the first.
Were I to make them truly independent it would take well over 4-5
times as long to run, which means no one would bother running them/
waiting around for their results, which makes them useless. I would
welcome a solution that could fulfill both your rules there, but until
I find one I have to err on the side of usability.

-Dylan