From: Faisal N Jawdat <faisal@...>
Date: 2007-02-09T03:58:59+09:00
Subject: Re: Posted Jim Weirich OSCON 2006 video presentation

On Feb 6, 2007, at 6:18 PM, Trans wrote:
> The more these test/specification/dbc system evolve the more they seem
> like double-entry bookkeeping. Aren't we just writing the code twice
> but from two differnt perspectives and seeing if they match up?

Given that double-entry bookkeeping is often cited as one of the  
subtle advancements that led to global the dominance of the Western  
world, I'm not sure this is a bad thing.

But.

"Yes and no".  For a single piece of code, it's generally a lot  
simpler to test that the code did the right thing than to write the  
functional code itself.  If you add in edge cases, the problem  
multiplies:  it's a lot easier to write 10 individual test cases,  
each for one edge case, than to write one piece of functional code  
that elegantly (or even inelegantly) handles all the edge cases.  As  
your app grows, this aspect grows by leaps and bounds.

I've been asked something along the lines of "well how do we know  
that the test code doesn't have bugs in it?".  The answer is that we  
don't.  However, think of this as a probability problem.  The  
following are gross oversimplifications, but should get the gist.

- If you have one test case per chunk of code, and that test case  
also has a 10% chance of being wrong, and if they're wrong in exactly  
the same way, then you have an 81% chance that the code executes  
correctly and tests cleanly, an 18% chance that the test fails (9%  
false positive, 9% false negative), and a 1% chance that the test  
passes incorrectly.  Since a failure will likely to look through your  
code until you either find what's wrong in your code or your test,  
then you've just dramatically decreased the chance of an uncaught bug.

- As code increases the likelyhood of bug-free code decreases  
exponentially with the amount of code.  However, since tests often  
must implicitly test things other than the specific thing they're  
testing, the likelihood of bugs remaining uncaught also decreases at  
an exponential rate.

There are limits to all of this, and I do think that people are prone  
to putting too much faith in automated testing, but it still runs  
circles around a lack of it.

-faisal