From: Ben Nagy <ben@...>
Date: 2006-11-01T15:24:25+09:00
Subject: Re: Probabilistic BDD?

> -----Original Message-----
> From: Robert Feldt [mailto:robert.feldt@gmail.com] 
> Sent: Wednesday, November 01, 2006 12:51 PM
> To: ruby-talk ML
> Subject: Probabilistic BDD?
> 
> Hi,
> 
> I'm playing around with BDD � la test/spec and foudn that I need to
> specify properties probabilistically ie saying that they are
> likely/unlikely. Has there been any previous work along these lines?
> 
> Should we add something like this to test/spec (Christian are 
> you listening? :))
[...]
> so that one can write specs like
> 
> > # Example of probabilistic specifications
> >
> > context "random generation" do
> >   unlikely "that two consecutive calls to rand gives the 
> same value" do
> >     rand(1000) == rand(1000)
> >   end
> >
> >   likely "that two consecutive calls to rand gives 
> different values" do
> >     rand(1000) != rand(1000)
> >   end
> > end
> 
> ?
> 
> This should probably be generalized so that the number of repetitions
> to run depends on the probability of the event but I think you get the
> idea.
> 
> Comments?

I think that the example you give is not appropriate for testing rand(), and
pretty much any code where the result is expected to conform to a set of
statistical properties. If you take a look at randomness test suites like
Diehard there are a battery of different tests that should be applied before
data can be called 'random' with any confidence.

http://en.wikipedia.org/wiki/Diehard_tests

The tests as you have written them would be satisfied by any number of
broken PRNGs, or even NRAAGs (Not Random At All Generators) (eg alternating
'1' and '2' ;). In particular, unlikely events must occur sometimes and
likely events must fail to occur sometimes, so some form of === seems better
than <=.

If you wanted to test RNGs then you need to run a whole series of tests -
either like the Diehard tests or just basic stuff like chi square, binomial,
monte-carlo calculation of pi etc.

More generally, I think that 'likely' and 'unlikely' are going to be so
context dependant that the user would be better off writing their own test
code, surely? I can see a place for should_be_random, but likely and
unlikely strike me as a bad idea. In any case, when running test code I
expect that it will give me the same result every time, so any tests should
at least have that property.

Sorry to sound negative. :(

ben