From: MikkelFJ <mikkelfj-anti-spam@...>
Date: 2002-08-10T09:05:14+09:00
Subject: Unit testing is considered harmful

I'm sorry, I could help it. I just find this considered harmful thing
completely overused.
Anyway todays topic is about that narrowminded focus on Unit Tests.

We just saw the programming contest having the test program tested, rather
than the applications. Of course UnitTest never claimed to be complete.

I read an interesting article about random testing (what and where is
forgotten), but the conclusion was that just hammering random data into a
system would give you less coverage than well designed tests, but when
counting in the time / effort spent to build these tests, you would get more
coverage by brute force.

This is of course awfully general, and UnitTesting is to be taken seriously.
But adding randomness as a key feature to a test suite is a good thing.

Another good thing, where applicable, is to specify the invariants that must
hold true. This can relatively easily be specified.

The invariant test is applied to a input/output dataset, object hierarchy,
datastruture or whatever. The random test suit then generates as many
testcases as possible and use the invariant to validate the results.
If you were clever in the random generator, you know now exactly what random
seed caused the error, and you can debug through that particular case and
see what invariant failed.

An example is a tree datastructure where each node has a cached accumulated
value of the child nodes data. The invariant will say something about how
the cache relates to actual data in child nodes. If there is a bug in the
cache update logic, the invariant will find it and have an exact test case
to reproduce.

I have in fact been using this method recently, in it pays itself back 100
fold in a complex datastructure. I know because I've tried manual debugging
of a similar problem.

If we take programming contest as example, it is important to identify the
nature of the problem. I suggested a relation to the assingment problem,
which may or may not be appropriate. But that view gives a good idea of
which data sets are valid and which a not, instead of extrapolating from a
few unit test cases. This view can then be used to build the verification
function. Having this function is not sufficient to ensure that submissions
are correct. We need to feed the contenders with test data that can be
verified.

The upcoming ICFP programming contest will heavily test contenders with
similar techniques, although the probably predefine a large test data set.
Some of the most successful solutions are not the most clever ones, but
those that have good measures for testing both during development, but
certainly also during runtime. The spend good CPU cycles on rejecting
optimizations that are incorrect due to some undetected bug.

Mikkel