From: dblack@...
Date: 2002-08-10T14:07:53+09:00
Subject: Re: Unit testing is considered harmful

Hi --

On Sat, 10 Aug 2002, Bill Kelly wrote:

>
> From: "MikkelFJ" <mikkelfj-anti-spam@bigfoot.com>
> >
> > I'm sorry, I could help it. I just find this considered harmful thing
> > completely overused.
> > Anyway todays topic is about that narrowminded focus on Unit Tests.
> >
> > We just saw the programming contest having the test program tested, rather
> > than the applications. Of course UnitTest never claimed to be complete.

I can't suppress a smile at the idea of my being used as an example of
excessive fanaticism about unit testing :-)

> Coincidentally, I submitted a question about this--CodingChallenge0002
> vs. unit tests--to the extreme-programming mailing list this morning.
> I was commenting on my surprise that we kept adding unit tests after
> the fact, and breaking existing solutions.  I referred people to the
> wiki page.  What happened is people replied back with detailed
> questions, specifically stating that the requirements were too vague
> and asking for clarification. . . . Which is just what happened here on
> ruby-talk when the challenge was first announced.  [[Not pointing
> fingers!! I enjoyed the challenge and learned a lot from it.]]

I guess ideally this kind of thing would be either on a wiki *or* on a
mailing list, but not both.  It's probably the fault of my quirky
habits: I liked the idea of putting the problem and the solutions on
the wiki, but for whatever reason I tend to be more at home with
email-based than wiki-based discussion.  Oh, and people did ask
questions here on the list :-)

> So I was somewhat at a loss as to how new unit tests were breaking so
> many of the existing solutions...  But I think, as was brought to my
> attention on the XP list, that the essence of the problem may have
> lied in the incomplete definition of the requirements on the wiki page,
> and . . . . well.  Pretty much that. :)

Of course, *I* understood the requirements, and yet the original
non-airtight bunch of tests was written by me :-)  That's not too
suprising, though I've found it quite interesting to see how elusive a
really "complete" set of tests for this problem (if there can be such
a thing) turns out to be.

> However, I don't, myself, see how this could be construed as a failure
> of the unit tests...  Unless it was thought that the unit test cases
> were to substitute as implicit definitions of the requirements.  I
> would agree that we may have witnessed *that* approach being harmful.
> But I still don't blame the unit tests themselves........

Ever since last year's Ruby Conference, where we talked a lot about
unit testing, I've been intrigued by the idea of having a test suite
serve as an actual definition of an interface -- and even different
test suites defining different versions.  In the case of this coding
challenge, however, I was aware that the tests probably wouldn't be
enough -- which was on my mind when I wrote:

  The challenge is to implement the method so that the tests pass and
  the requirements are met.

:-)

It certainly has been an interesting little case study.


David

-- 
David Alan Black
home: dblack@candle.superlink.net
work: blackdav@shu.edu
Web:  http://pirate.shu.edu/~blackdav