From: Jeff Wood <jeff.darklight@...>
Date: 2005-09-28T06:02:57+09:00
Subject: Re: random testing with Test::Unit

------=_Part_3883_22504389.1127854946434
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Why do you need DEBUG LEVEL or whatever when the Ruby
environment/interpreter already has $VERBOSE ... ???

I mean, it's already there ... just turn up the verbosity ...

ruby -v mytestcases.rb

... and your done ... ( am i missing something? )

j.

On 9/27/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
>
> I gave another shot at making a derived TestCase class. I only
> needed one hack to effectively make it ignore this class when
> running tests: undef_method(:default_test). Here is what I
> came up with that has many of the features I wanted:
>
> require 'test/unit'
>
> module Test
> module Unit
> class RandomTestCase < TestCase
> def self.suite
> $debug_level =3D (ENV['DEBUG_LEVEL']||0).to_i if
> $debug_level.nil?
> if $random_seed.nil?
> $random_seed =3D ENV['RAND_SEED'].to_i.nonzero? ||
> (srand;srand)
> srand($random_seed)
> puts("random_seed: #{$random_seed}") if
> $debug_level>=3D1
> end
> random_iterations =3D (ENV['RAND_ITER']||8).to_f
> methods =3D Regexp.new(ENV['TEST_METHODS']||".*")
> suite =3D super
> tests =3D suite.tests
> tests.reject! { |t|
> !methods.match(t.name.gsub(/\Atest_/,''))
> }
> (class << tests;self;end).class_eval {
> def each
> catch(:stop_suite) {
> (@iterations*size).to_i.times {
> catch(:invalid_test) {
> yield(slice(rand(size)))
> }
> }
> }
> end
> }
> tests.instance_eval { @iterations =3D random_iterations }
> suite
> end
> undef_method(:default_test)
> def teardown
> if not passed?
> puts("\nrandom_seed: #{$random_seed}")
> throw(:stop_suite)
> end
> end
> end
> end
> end
>
>
> Here is that example I gave earlier using the above for random
> testing Array#push and Array#pop using AOP:
>
>
> class ArrayAOP < Array
> include Test::Unit::Assertions
> def push(*args)
> print("#{self.inspect}.push(*#{args.inspect}) -> ") if
> $debug_level>=3D1
> n =3D size
> na =3D args.size
> ret =3D super
> assert_equal(n+na,size)
> assert_equal(slice(-na,na),args)
> assert_same(self,ret)
> p(ret) if $debug_level>=3D1
> ret
> end
> def pop
> print("#{self.inspect}.pop -> ") if $debug_level>=3D1
> n =3D size
> ret0 =3D n.nonzero? ? slice(-1) : nil
> ret =3D super
> assert_equal(ret0,ret)
> assert_equal(n.nonzero? ? n-1 : 0,size)
> p(ret) if $debug_level>=3D1
> ret
> end
> end
>
> class ArrayTest < Test::Unit::RandomTestCase
> def self.suite
> @@object =3D ArrayAOP[]
> super
> end
> def test_push
> args =3D []
> rand(4).times {
> args << [0,"",[],nil,false][rand(5)]
> }
> @@object.push(*args)
> end
> def test_pop
> rand(4).times {
> @@object.pop
> }
> end
> end
>
>
> As you can see from above, with this RandomTestCase class, you
> can do random testing very easily. But, it still would be nice
> to get options from the command line rather than the
> environment as I'm doing in RandomTestCase.
>
> --- Eivind Eklund <eeklund@gmail.com> wrote:
>
> > On 9/26/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
> > >
> > > Does anybody else do random testing on their ruby code
> > besides
> > > me? I picked it up because of my background (IC design)
> > where
> > > random testing is one of the verification techniques of
> > > hardware.
> >
> >
> > I cannot remember having used random testing for Ruby code,
> > though I've used
> > it for code in other languages. Having a nice framework for
> > it would be,
> > well, nice.
> >
> > For all of these strategies, I've had to hack up Test::Unit
> > in
> > > the same way to add various features. Here are the things
> > I've
> > > added:
> > >
> > > + instead of running the test_* methods in a fixed order
> > once,
> > > run them in a random order for N passes.
> >
> >
> > Running N times sounds like a subclass of TestCase would be
> > appropriate.
> > Otherwise, how do you distinguish between what tests should
> > be run once and
> > which should be run several time (for statistical coverage)?
> >
> > Random order sounds reasonable for everything, anyway.
> >
> >
> > + abort testing when the first failure is reached.
> >
> >
> > Also subclass material.
> >
> > + easy way to skip the rest of the current test (when the
> > > random input is invalid). Don't include this in the number
> > of
> > > tests or mark it as skipped.
> >
> >
> > Also subclass material, I think.
> >
> > + way to pass number of passes of the test_* methods on the
> > > command line.
> > >
> > > + display random seed and way of passing it in on the
> > command
> > > line.
> > >
> > > + test suite should have access to the --verbose level or
> > > another switch to control the debugging verbosity.
> >
> >
> > My gut feeling is that this is inappropriate. I see a major
> > part of the
> > Test::Unit style testing as the fact that the programmer only
> > has to check
> > for pass/fail. Passing through verbiosity information seems
> > to turn this on
> > its head, encouraging setting up the output for inspection.
> > (On the other
> > hand, I have sometimes needed information for debugging -
> > calling this
> > "debug level" would do much to ease my concerns...)
> >
> > + way to pass command-line options down to the test suite to
> > > control various things - what methods to test, what
> > > classes to test, various other flags.
> >
> >
> > I also get an initial bad feeling about this, as it makes it
> > totally
> > necessary to use the console-based testrunner.
> >
> > Eivind.
> > --
> > Hazzle free packages for Ruby?
> > RPA is available from http://www.rubyarchive.org/
> >
>
>
>
>
> __________________________________
> Yahoo! Mail - PC Magazine Editors' Choice 2005
> http://mail.yahoo.com
>
>


--
"http://ruby-lang.org -- do you ruby?"

Jeff Wood

------=_Part_3883_22504389.1127854946434--