From: Eric Mahurin <eric_mahurin@...>
Date: 2005-09-28T02:14:14+09:00
Subject: Re: random testing with Test::Unit

I agree that a derived class of Test::Unit::TestCase (maybe
Test::Unit::RandomTestCase) would be appropriate.  I tried this
first.  Unfortunately, I found several places where doing this
became problematic - test runners/collectors, rubygems?? (don't
remember now).  There is the assumption that all derived
classes of Test::Unit::TestCase are the actual test case
classes for testing (not the case for this
Test::Unit::RandomTestCase class).  At the time, it seemed
easier to hack into Test::Unit::TestCase and
Test::Unit::TestSuite.  Now I'm hacking into TestSuite#tests to
redefine tests.each.  It still doesn't seem like the right way.
 Some derived classes that handle random testing
(RandomTestCase and RandomTestSuite) seems like the right way.

--- Eivind Eklund <eeklund@gmail.com> wrote:

> On 9/26/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
> >
> > Does anybody else do random testing on their ruby code
> besides
> > me? I picked it up because of my background (IC design)
> where
> > random testing is one of the verification techniques of
> > hardware.
> 
> 
> I cannot remember having used random testing for Ruby code,
> though I've used
> it for code in other languages. Having a nice framework for
> it would be,
> well, nice.
> 
> For all of these strategies, I've had to hack up Test::Unit
> in
> > the same way to add various features. Here are the things
> I've
> > added:
> >
> > + instead of running the test_* methods in a fixed order
> once,
> > run them in a random order for N passes.
> 
> 
> Running N times sounds like a subclass of TestCase would be
> appropriate.
> Otherwise, how do you distinguish between what tests should
> be run once and
> which should be run several time (for statistical coverage)?
> 
> Random order sounds reasonable for everything, anyway.
> 
> 
> + abort testing when the first failure is reached.
> 
> 
> Also subclass material.
> 
> + easy way to skip the rest of the current test (when the
> > random input is invalid). Don't include this in the number
> of
> > tests or mark it as skipped.
> 
> 
> Also subclass material, I think.
> 
> + way to pass number of passes of the test_* methods on the
> > command line.
> >
> > + display random seed and way of passing it in on the
> command
> > line.
> >
> > + test suite should have access to the --verbose level or
> > another switch to control the debugging verbosity.
> 
> 
> My gut feeling is that this is inappropriate. I see a major
> part of the
> Test::Unit style testing as the fact that the programmer only
> has to check
> for pass/fail. Passing through verbiosity information seems
> to turn this on
> its head, encouraging setting up the output for inspection.
> (On the other
> hand, I have sometimes needed information for debugging -
> calling this
> "debug level" would do much to ease my concerns...)
> 
> + way to pass command-line options down to the test suite to
> > control various things - what methods to test, what
> > classes to test, various other flags.
> 
> 
> I also get an initial bad feeling about this, as it makes it
> totally
> necessary to use the console-based testrunner.

As you guessed, I use random testing for both testing and
debugging.  With hand-coded tests, there is not as big a need
for spitting out debug info, because the failure points you to
the line # and you can gather what the inputs were to easily
recreate the problem.  This is not so with random testing -
especially when there is some state carried over from one
random test to the next (not done in hand-coded tests).  When
you have failures with random testing, you need a way to track
down exactly what happened.  You need extra debug info.

Also since random testing can run many more tests and still be
useful (thousands - even millions), having the ability to
narrow the testing down to a specific area to test/debug would
be useful.

I've never used anything but a console test runner.  Where does
stdout/stderr go for the other runners.  If you put it in a log
file, it seems like it could still be useful for other runners.


> Eivind.
> --
> Hazzle free packages for Ruby?
> RPA is available from http://www.rubyarchive.org/
> 



	
		
______________________________________________________ 
Yahoo! for Good 
Donate to the Hurricane Katrina relief effort. 
http://store.yahoo.com/redcross-donate3/