From: Kent Beck <kentbeck@...>
Date: 2001-12-10T17:59:17+09:00
Subject: [ruby-talk:28069] Re: John Roth dolt (  Re: A challenge to proponents of Unit Testing. )


Thaddeus L Olczyk <olczyk@interaccess.com> wrote in message
news:3c0fca7e.119078828@nntp.interaccess.com...
> On Fri, 07 Dec 2001 15:28:15 GMT, "David B Lightstone"
> <david.lightstone@prodigy.net> wrote:
>
> >
> >"Keith Ray" <k1e2i3t4h5r6a7y@1m2a3c4.5c6o7m> wrote in message
> >> Kent Beck's response to that example:
> >>
> >> http://groups.yahoo.com/group/extremeprogramming/message/37242
> >>
> >> "The biggest problem is that it doesn't balance the cost and benefits
> >> of tests. He has a triangle example which he writes (I think) 29
> >> tests for. The test-first version can be done confidently with 5
> >> tests."
> ...doesn't balance the cost and benefits...
> That's exactly what the anti-testing people say.
> So what does it appear he does in typical managerial fashion he
> tries to balance the equation by lowering the costs but doesn't
> consider lowering the benifits. One might be tempted to say
> that he is now getting caught up in the general XP cult like
> behaviour; but I say: Everyone has an off day, and this might have
> been one for him. Let us give him a chance to clarify his statement.
>
> Oh yeah. And describe the five tests.

I didn't want to respond until I'd done the experiment myself. I apologize
for the length of what follows, but simple sequences of words appear to be
failing me, communication-wise.

I assume the interface is a function that returns 1 if equilateral, 2 if
isoceles, and 3 if scalene. Here is the first test. I don't know where to
put the function yet, so I'll just implement it in the test class and
refactor later.

TriangleTest>>testScalene
    self assert: (self evaluate: 1 side: 2 side: 3) = 3

Using Uncle Bob's excellent trick (immortalized as Duplicate Removing Set in
the Smalltalk Best Practice Patterns), this can be implemented as:

TriangleTest>>evaluate: aNumber1 side: aNumber2 side: aNumber3
    | sides |
    sides := Set
        with: aNumber1
        with: aNumber2
        with: aNumber3.
    ^sides size

Since I know Sets are insensitive to the order in which elements are added,
I conclude I don't have to test alternative orders for the arguments.

The implementation should suffice for isoceles and equilateral triangles
without modification. However, I want to communicate that to posterity, so I
write:

TriangleTest>>testEquilateral
    self assert: (self evaluate: 2 side: 2 side: 2) = 1

TriangleTest>>testIsoceles
    self assert: (self evaluate: 1 side: 2 side: 2) = 2

They both run first time.

Now I suppose we want to "idiot-proof" our code, so an exception is thrown
if bad arguments are passed. My code does not seem to suffer from the lack
of such checking code, but I have two tests to burn.

Now the implementation is:

TriangleTest>>evaluate: aNumber1 side: aNumber2 side: aNumber3
    | sides |
    sides := Set
        with: aNumber1
        with: aNumber2
        with: aNumber3.
    sides asSortedCollection first <= 0 ifTrue: [self fail].
    ^sides size

This implementation will also work if a nil is passed, because the
asSortedCollection will throw an exception trying to compare a number and a
nil. The truly pathological case is if someone were to pass all strings,
which can compare to each other. I'll implement this one, but we're already
wandering in Silly-land.

TriangleTest>>testStrings
    [self evaluate: 'a' side: 'b' side: 'c']
        on: Exception
        do: [:ex | ^self].
    self fail

That one already works, too, because of the comparison with 0.

There you have it--five tests that give me great confidence in the MTBF of
my function.

>
> >
> > It is rather hard to access either your response or Mr Beck's intent.
> >What to you believe?
> >
> >(1) Mr Beck may have read the 29 test cases and concluded that
> >the 5 tests he favors are adequate. (the original 29 hasd redundancy)
> >
> >(2) Mr Beck may have read the 29 test cases and concludes that
> >pruning a number of them was appropriate (impossible situations
> >by virtue of a priori knowledge about other aspects of the system).
> >(Hence he is doing integration testing, rather than unit testing)
> >
> >(3) Mr Beck may have determined that the customer can afford
> >to take the risk because the expected loss associated with
> >the pruned test cases is acceptable.
> >
> 4) He merges several test cases into one test case. Thus requiring the
> same amount of work.
>
> The one thing I notice about each case is that they require you to
> think of and dispose of cases. In others even though you don't write
> them, you do have to expend considerable effort on them.
>
> The things is though ( and a major part of my original post ) is that
> "only 5" sounds good until you think about it.

What defects am I missing with the above 5 tests? Put another way, what
further tests could improve the MTBF of my function?

> Assume you have a small project of 1000 units. That means 5000 units
> ( and that is assuming that 5 still holds I would expect other units
> to be more complex requiring even more unit tests, the number
> increasing exponentially with complexity. )
>
> So now you need an infrastructure to handle 1000 units + 5000 tests.
> That is what I see as the main problem with unit testing.

If your unit tests are large and complicated, you have a design problem, not
a testing problem. If you have an exponential unit testing problem, again,
you have a design problem, not a testing problem.

Mr. Beck
P.S. How does any of this make me anti-testing?