From: David B Lightstone Date: 2001-12-10T22:39:30+09:00 Subject: [ruby-talk:28081] Re: John Roth dolt ( Re: A challenge to proponents of Unit Testing. ) "Kent Beck" wrote in message news:9v1ss3$adi$1@suaar1aa.prod.compuserve.com... > > Thaddeus L Olczyk wrote in message > news:3c0fca7e.119078828@nntp.interaccess.com... > > On Fri, 07 Dec 2001 15:28:15 GMT, "David B Lightstone" > > wrote: > > > > > > > >"Keith Ray" wrote in message > > >> Kent Beck's response to that example: > > >> > > >> http://groups.yahoo.com/group/extremeprogramming/message/37242 > > >> > > >> "The biggest problem is that it doesn't balance the cost and benefits > > >> of tests. He has a triangle example which he writes (I think) 29 > > >> tests for. The test-first version can be done confidently with 5 > > >> tests." > > ...doesn't balance the cost and benefits... > > That's exactly what the anti-testing people say. > > So what does it appear he does in typical managerial fashion he > > tries to balance the equation by lowering the costs but doesn't > > consider lowering the benifits. One might be tempted to say > > that he is now getting caught up in the general XP cult like > > behaviour; but I say: Everyone has an off day, and this might have > > been one for him. Let us give him a chance to clarify his statement. > > > > Oh yeah. And describe the five tests. > > I didn't want to respond until I'd done the experiment myself. I apologize > for the length of what follows, but simple sequences of words appear to be > failing me, communication-wise. > > I assume the interface is a function that returns 1 if equilateral, 2 if > isoceles, and 3 if scalene. Here is the first test. I don't know where to > put the function yet, so I'll just implement it in the test class and > refactor later. > > TriangleTest>>testScalene > self assert: (self evaluate: 1 side: 2 side: 3) = 3 > > Using Uncle Bob's excellent trick (immortalized as Duplicate Removing Set in > the Smalltalk Best Practice Patterns), this can be implemented as: > > TriangleTest>>evaluate: aNumber1 side: aNumber2 side: aNumber3 > | sides | > sides := Set > with: aNumber1 > with: aNumber2 > with: aNumber3. > ^sides size > > Since I know Sets are insensitive to the order in which elements are added, > I conclude I don't have to test alternative orders for the arguments. > > The implementation should suffice for isoceles and equilateral triangles > without modification. However, I want to communicate that to posterity, so I > write: > > TriangleTest>>testEquilateral > self assert: (self evaluate: 2 side: 2 side: 2) = 1 > > TriangleTest>>testIsoceles > self assert: (self evaluate: 1 side: 2 side: 2) = 2 > > They both run first time. > > Now I suppose we want to "idiot-proof" our code, so an exception is thrown > if bad arguments are passed. My code does not seem to suffer from the lack > of such checking code, but I have two tests to burn. > > Now the implementation is: > > TriangleTest>>evaluate: aNumber1 side: aNumber2 side: aNumber3 > | sides | > sides := Set > with: aNumber1 > with: aNumber2 > with: aNumber3. > sides asSortedCollection first <= 0 ifTrue: [self fail]. > ^sides size > > This implementation will also work if a nil is passed, because the > asSortedCollection will throw an exception trying to compare a number and a > nil. The truly pathological case is if someone were to pass all strings, > which can compare to each other. I'll implement this one, but we're already > wandering in Silly-land. > > TriangleTest>>testStrings > [self evaluate: 'a' side: 'b' side: 'c'] > on: Exception > do: [:ex | ^self]. > self fail > > That one already works, too, because of the comparison with 0. > > There you have it--five tests that give me great confidence in the MTBF of > my function. Given the infrastructure available, 5 tests it is. Unfortunately, the production implementaion does not have the infrastructure, so we will have to port it to the production version. How much additional testing will be needed to determine such things as (1) The exception system functions correctly (maybe there is an interaction problem to be exposed) (2) That set order incertion independence From my perspective a Unit test has been transformed into an Integration test by means of assuming pretested component availability. Complexity can be shifted around, but not eliminated > > > > > > > > > It is rather hard to access either your response or Mr Beck's intent. > > >What to you believe? > > > > > >(1) Mr Beck may have read the 29 test cases and concluded that > > >the 5 tests he favors are adequate. (the original 29 hasd redundancy) > > > > > >(2) Mr Beck may have read the 29 test cases and concludes that > > >pruning a number of them was appropriate (impossible situations > > >by virtue of a priori knowledge about other aspects of the system). > > >(Hence he is doing integration testing, rather than unit testing) > > > > > >(3) Mr Beck may have determined that the customer can afford > > >to take the risk because the expected loss associated with > > >the pruned test cases is acceptable. > > > > > 4) He merges several test cases into one test case. Thus requiring the > > same amount of work. > > > > The one thing I notice about each case is that they require you to > > think of and dispose of cases. In others even though you don't write > > them, you do have to expend considerable effort on them. > > > > The things is though ( and a major part of my original post ) is that > > "only 5" sounds good until you think about it. > > What defects am I missing with the above 5 tests? Put another way, what > further tests could improve the MTBF of my function? > > > Assume you have a small project of 1000 units. That means 5000 units > > ( and that is assuming that 5 still holds I would expect other units > > to be more complex requiring even more unit tests, the number > > increasing exponentially with complexity. ) > > > > So now you need an infrastructure to handle 1000 units + 5000 tests. > > That is what I see as the main problem with unit testing. > > If your unit tests are large and complicated, you have a design problem, not > a testing problem. If you have an exponential unit testing problem, again, > you have a design problem, not a testing problem. > > Mr. Beck > P.S. How does any of this make me anti-testing? > >