From: Mike Woodhouse <mikewoodhouse@...>
Date: 2006-11-10T18:25:04+09:00
Subject: Re: unit testing with uncertainty



On Nov 10, 7:39 am, "martinus" <martin.ank...@gmail.com> wrote:
> Hi all, I am writing an particle swarm optimizer that tries to find
> good floating point values for optimization problems. I have several
> test cases, and some target fitness value that the optimizer should be
> able to reach. The problem is that since such optimizers make lots of
> use of random numbers, so the optimizer can only find in about 90% of
> the cases the target fitness value. I can relax the target fitness
> value, than 95% of the runs are ok; but that is not really a solution.
>
> Any ideas how to write tests when you have such an uncertainty about
> the result?
>
> Martin

You can use a predetermined random number sequence (or sequences) for
unit testing, where I'd suggest you ought to be working in a fairly
black and white world. Establishing expected results if the number of
drawing from the RNG is large doesn't look like a fun job. Could you
use mock objects to provide predetermined results at a higher level?

I don't see that your problem really falls into the unit test category
though, it looks more like a user, performance or acceptance test
issue, since you'll pretty much have to perform multiple runs to
establish the success rate, and the program hitting or missing the 90%
criterion is still not going to tell you definitively if it's working.
It's as much a benchmarking deal as a testing one.

--Mike