From: "Joe Pallas (pallas at cs dot stanford dot edu)" <pallas@..._dot_stanford.edu>
Date: 2001-12-11T03:40:03+09:00
Subject: [ruby-talk:28122] Re: John Roth dolt (  Re: A challenge to proponents of Unit Testing. )

Kent Beck wrote:

An astounding indictment of the way XP approaches Unit Testing.

> Since I know Sets are insensitive to the order in which elements are added,
> I conclude I don't have to test alternative orders for the arguments.

So, you have made decisions about what cases to test based on the
implementation.  That implies two things:

1) You have violated the XP rule that tests should be written before the
code that they test.

2) Your test cases are specific to the implementation, and will not
adequately test a different implementation.

This means that some future refactoring may make your tests inadequate,
but the coder doing the refactoring would rely on your tests and check
in broken code.

> Now I suppose we want to "idiot-proof" our code, so an exception is thrown
> if bad arguments are passed. My code does not seem to suffer from the lack
> of such checking code, but I have two tests to burn.
....
> There you have it--five tests that give me great confidence in the MTBF of
> my function.

> What defects am I missing with the above 5 tests? Put another way, what
> further tests could improve the MTBF of my function?

What a bizarre notion.  It is not meaningful to talk about the MTBF of
software, because software does not fail.  Software is either correct or
incorrect, and that does not change over time.  Requirements change, and
execution environments change, but the program itself is not subject to
"bit rot."

Since you've implied that your function will always throw an exception
for bad arguments, let me point out that the MTFB of your function is
zero if given either of the inputs (5, 5, 11), which it will incorrectly
report to be an isosceles triangle, or (1, 3, 7), which it will
incorrectly report to be a scalene triangle.

> If your unit tests are large and complicated, you have a design problem, not
> a testing problem.

Again, this can only be true if your testing is dictated by the design,
*rather than by the requirements.*  That's not just backwards, it is
doubly dangerous in the XP milieu, where frequent changes to the design
are encouraged, and the only validation of those changes comes from the
pre-existing tests.

joe