From: Johan Holmberg <holmberg@...>
Date: 2003-09-16T00:26:49+09:00
Subject: Re: Test::Unit -- multiple errors in test method ???


On Mon, 15 Sep 2003, Nathaniel Talbott wrote:
>
> I don't understand why the tests become useless to you. If we have a test
> for strip:
>
>   def test_strip
>     assert_equal("name", " name ")
>     assert_equal("name", "\tname\n")
>   end
>
> ...and the first assertion fails, we immediately know that there's a bug in
> strip. Why is the test useless if the second assertion isn't executed?
>

I didn't mean *completely useless*.
(but see below for an example)

>
> Perhaps a better word is "related". All of those tests are related, aren't
> they? If the first one fails, there's a high probability that the second
> one, and the third one, etc., will fail, too. So the testing framework only
> worries you with the first one.
>

I felt that I got a too narrow "window" of what was wrong.
A too "boolean" (yes/no) answer.

Let me give an example:

<example>
A while ago I began to think that Rubys File.dirname/File.basename
was too UNIX-centric and mis-behaved on Windows. One of the things I
tried to do was to run the relevant tests in Rubicon. Since these
test also were too UNIX-centric I started adding more tests in the
same style as the ones already there.

So I added a number of more assertions about "basename".
Before there were about 25 of these (in one "test method") and I
added about 30 more Windows-specific assertions.

When I started to run the tests I got errors.
So yes, I know that there *are errors* in File.basename (at least
according to my assumtions as codified in my new assertions).

But it was hard to understand *how much* the current Ruby
implementation deviated from what I thought about as the "right"
bahaviour.

After a while I gave up running the tests under Rubicon, and moved
my assertion code to a small "standalone-program" that just tested
basename and reported *all* errors it found.

Then when I saw all errrors at once (a wider "window"), I began to
understand more about the nature of the problem and could start
thinking about how to fix it.
</example>

I understand that the current Test::Unit framwork is intentionally
designed as it is, and that it normally is a good thing to abort a
test method when the first assertion fails.

I just don't know yet how "unusual" my scenario above is, and if
there are other ways to cope with the situation by using features
already present in Test::Unit.

>
> So, both for acceptance testing and for those who feel that they need it, I
> plan to add the ability to keep running when an assertion fails. I've
> actually planned on it for a while, but haven't gotten around to it yet. For
> the time being, you can use some variation of what Guy posted in
> [ruby-talk:82066].
>

That sounds promising.
I'll try Guys / David Alan Blacks approach until then.

/Johan Holmberg