From: Eivind Eklund <eeklund@...>
Date: 2005-09-28T01:24:28+09:00
Subject: Re: random testing with Test::Unit

------=_Part_4103_6481245.1127838264241
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On 9/26/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
>
> Does anybody else do random testing on their ruby code besides
> me? I picked it up because of my background (IC design) where
> random testing is one of the verification techniques of
> hardware.


I cannot remember having used random testing for Ruby code, though I've use=
d
it for code in other languages. Having a nice framework for it would be,
well, nice.

For all of these strategies, I've had to hack up Test::Unit in
> the same way to add various features. Here are the things I've
> added:
>
> + instead of running the test_* methods in a fixed order once,
> run them in a random order for N passes.


Running N times sounds like a subclass of TestCase would be appropriate.
Otherwise, how do you distinguish between what tests should be run once and
which should be run several time (for statistical coverage)?

Random order sounds reasonable for everything, anyway.


+ abort testing when the first failure is reached.


Also subclass material.

+ easy way to skip the rest of the current test (when the
> random input is invalid). Don't include this in the number of
> tests or mark it as skipped.


Also subclass material, I think.

+ way to pass number of passes of the test_* methods on the
> command line.
>
> + display random seed and way of passing it in on the command
> line.
>
> + test suite should have access to the --verbose level or
> another switch to control the debugging verbosity.


My gut feeling is that this is inappropriate. I see a major part of the
Test::Unit style testing as the fact that the programmer only has to check
for pass/fail. Passing through verbiosity information seems to turn this on
its head, encouraging setting up the output for inspection. (On the other
hand, I have sometimes needed information for debugging - calling this
"debug level" would do much to ease my concerns...)

+ way to pass command-line options down to the test suite to
> control various things - what methods to test, what
> classes to test, various other flags.


I also get an initial bad feeling about this, as it makes it totally
necessary to use the console-based testrunner.

Eivind.
--
Hazzle free packages for Ruby?
RPA is available from http://www.rubyarchive.org/

------=_Part_4103_6481245.1127838264241--