From: Wilson Bilkovich <wilsonb@...>
Date: 2006-11-02T00:39:38+09:00
Subject: Re: Probabilistic BDD?

On 11/1/06, Robert Feldt <robert.feldt@gmail.com> wrote:
> > I think that the example you give is not appropriate for testing rand(), and
> > pretty much any code where the result is expected to conform to a set of
> > statistical properties. If you take a look at randomness test suites like
> > Diehard there are a battery of different tests that should be applied before
> > data can be called 'random' with any confidence.
> >
> > http://en.wikipedia.org/wiki/Diehard_tests
> >
> > The tests as you have written them would be satisfied by any number of
> > broken PRNGs, or even NRAAGs (Not Random At All Generators) (eg alternating
> > '1' and '2' ;). In particular, unlikely events must occur sometimes and
> > likely events must fail to occur sometimes, so some form of === seems better
> > than <=.
> >
> > If you wanted to test RNGs then you need to run a whole series of tests -
> > either like the Diehard tests or just basic stuff like chi square, binomial,
> > monte-carlo calculation of pi etc.
> >
> I don't want to test RNG's; that was just the smallest possible
> example use of the likely/unlikely methods I could think of. I'm
> fairly well versed in RNG testing, thank you.
>
> > More generally, I think that 'likely' and 'unlikely' are going to be so
> > context dependant that the user would be better off writing their own test
> > code, surely? I can see a place for should_be_random, but likely and
> > unlikely strike me as a bad idea. In any case, when running test code I
> > expect that it will give me the same result every time, so any tests should
> > at least have that property.
> >
> > Sorry to sound negative. :(
> >
> It is ok to be negative but I have run into test situations many times
> where there is an element of varying behavior involved and specifying
> exactly what is to be expected can only be done by running multiple
> tests and making claims about overall properties of the results.
>
> But it may be the case that people should write their own test code for it yes.
>
> Still I think this is an important discussion in general since for
> complex algorithms where it is costly to calc the exact expected
> results ways to write partial specs are important.
>


Personally, I would do something like:

def something_run_a_bunch_of_times
  results = []
  100_000.times {results << whatever_is_being_verified}
  results.matches_statistical_requirements_of_domain?
end

specify "should be totally awesome" do
  something_run_a_bunch_of_times.should.be true
end