From: Eric Mahurin <eric_mahurin@...>
Date: 2005-09-28T07:17:22+09:00
Subject: Re: random testing with Test::Unit

Yes, there are several ruby switches to control
warnings/debugging/verbosity:

  -d              set debugging flags (set $DEBUG to true)
  -v              print version number, then turn on verbose
mode
  -w              turn warnings on for your script
  -W[level]       set warning level; 0=silence, 1=medium,
2=verbose (default)

In addition to setting some global variables ($VERBOSE, $DEBUG,
$-v, $-w, $-d), these turn on warnings that some ruby code may
not like (produce excess warnings).  -d probably also turns on
some internal ruby debugging, but I don't know where.

In addition to this, the autorunner in test/unit has
--verbose[=level].  I would like to reuse this, but it is not
very easy to access (can get it through
ObjectSpace.each_object).

The TEST_METHODS below isn't needed as test/unit/autorunner
already has a --name option which works fine for this.

--- Jeff Wood <jeff.darklight@gmail.com> wrote:

> Why do you need DEBUG LEVEL or whatever when the Ruby
> environment/interpreter already has $VERBOSE ... ???
> 
> I mean, it's already there ... just turn up the verbosity ...
> 
> ruby -v mytestcases.rb
> 
> ... and your done ... ( am i missing something? )
> 
> j.
> 
> On 9/27/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
> >
> > I gave another shot at making a derived TestCase class. I
> only
> > needed one hack to effectively make it ignore this class
> when
> > running tests: undef_method(:default_test). Here is what I
> > came up with that has many of the features I wanted:
> >
> > require 'test/unit'
> >
> > module Test
> > module Unit
> > class RandomTestCase < TestCase
> > def self.suite
> > $debug_level = (ENV['DEBUG_LEVEL']||0).to_i if
> > $debug_level.nil?
> > if $random_seed.nil?
> > $random_seed = ENV['RAND_SEED'].to_i.nonzero? ||
> > (srand;srand)
> > srand($random_seed)
> > puts("random_seed: #{$random_seed}") if
> > $debug_level>=1
> > end
> > random_iterations = (ENV['RAND_ITER']||8).to_f
> > methods = Regexp.new(ENV['TEST_METHODS']||".*")
> > suite = super
> > tests = suite.tests
> > tests.reject! { |t|
> > !methods.match(t.name.gsub(/\Atest_/,''))
> > }
> > (class << tests;self;end).class_eval {
> > def each
> > catch(:stop_suite) {
> > (@iterations*size).to_i.times {
> > catch(:invalid_test) {
> > yield(slice(rand(size)))
> > }
> > }
> > }
> > end
> > }
> > tests.instance_eval { @iterations = random_iterations }
> > suite
> > end
> > undef_method(:default_test)
> > def teardown
> > if not passed?
> > puts("\nrandom_seed: #{$random_seed}")
> > throw(:stop_suite)
> > end
> > end
> > end
> > end
> > end
> >
> >
> > Here is that example I gave earlier using the above for
> random
> > testing Array#push and Array#pop using AOP:
> >
> >
> > class ArrayAOP < Array
> > include Test::Unit::Assertions
> > def push(*args)
> > print("#{self.inspect}.push(*#{args.inspect}) -> ") if
> > $debug_level>=1
> > n = size
> > na = args.size
> > ret = super
> > assert_equal(n+na,size)
> > assert_equal(slice(-na,na),args)
> > assert_same(self,ret)
> > p(ret) if $debug_level>=1
> > ret
> > end
> > def pop
> > print("#{self.inspect}.pop -> ") if $debug_level>=1
> > n = size
> > ret0 = n.nonzero? ? slice(-1) : nil
> > ret = super
> > assert_equal(ret0,ret)
> > assert_equal(n.nonzero? ? n-1 : 0,size)
> > p(ret) if $debug_level>=1
> > ret
> > end
> > end
> >
> > class ArrayTest < Test::Unit::RandomTestCase
> > def self.suite
> > @@object = ArrayAOP[]
> > super
> > end
> > def test_push
> > args = []
> > rand(4).times {
> > args << [0,"",[],nil,false][rand(5)]
> > }
> > @@object.push(*args)
> > end
> > def test_pop
> > rand(4).times {
> > @@object.pop
> > }
> > end
> > end
> >
> >
> > As you can see from above, with this RandomTestCase class,
> you
> > can do random testing very easily. But, it still would be
> nice
> > to get options from the command line rather than the
> > environment as I'm doing in RandomTestCase.
> >
> > --- Eivind Eklund <eeklund@gmail.com> wrote:
> >
> > > On 9/26/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
> > > >
> > > > Does anybody else do random testing on their ruby code
> > > besides
> > > > me? I picked it up because of my background (IC design)
> > > where
> > > > random testing is one of the verification techniques of
> > > > hardware.
> > >
> > >
> > > I cannot remember having used random testing for Ruby
> code,
> > > though I've used
> > > it for code in other languages. Having a nice framework
> for
> > > it would be,
> > > well, nice.
> > >
> > > For all of these strategies, I've had to hack up
> Test::Unit
> > > in
> > > > the same way to add various features. Here are the
> things
> > > I've
> > > > added:
> > > >
> > > > + instead of running the test_* methods in a fixed
> order
> > > once,
> > > > run them in a random order for N passes.
> > >
> > >
> > > Running N times sounds like a subclass of TestCase would
> be
> > > appropriate.
> > > Otherwise, how do you distinguish between what tests
> should
> > > be run once and
> > > which should be run several time (for statistical
> coverage)?
> > >
> > > Random order sounds reasonable for everything, anyway.
> > >
> > >
> > > + abort testing when the first failure is reached.
> > >
> > >
> > > Also subclass material.
> > >
> > > + easy way to skip the rest of the current test (when the
> > > > random input is invalid). Don't include this in the
> number
> > > of
> > > > tests or mark it as skipped.
> > >
> > >
> > > Also subclass material, I think.
> > >
> > > + way to pass number of passes of the test_* methods on
> the
> > > > command line.
> > > >
> > > > + display random seed and way of passing it in on the
> > > command
> > > > line.
> > > >
> > > > + test suite should have access to the --verbose level
> or
> > > > another switch to control the debugging verbosity.
> > >
> > >
> > > My gut feeling is that this is inappropriate. I see a
> major
> > > part of the
> > > Test::Unit style testing as the fact that the programmer
> only
> > > has to check
> > > for pass/fail. Passing through verbiosity information
> seems
> > > to turn this on
> > > its head, encouraging setting up the output for
> inspection.
> > > (On the other
> > > hand, I have sometimes needed information for debugging -
> > > calling this
> > > "debug level" would do much to ease my concerns...)
> > >
> > > + way to pass command-line options down to the test suite
> to
> > > > control various things - what methods to test, what
> > > > classes to test, various other flags.
> > >
> > >
> > > I also get an initial bad feeling about this, as it makes
> it
> > > totally
> > > necessary to use the console-based testrunner.
> > >
> > > Eivind.
> > > --
> > > Hazzle free packages for Ruby?
> > > RPA is available from http://www.rubyarchive.org/
> > >
> >
> >
> >
> >
> > __________________________________
> > Yahoo! Mail - PC Magazine Editors' Choice 2005
> > http://mail.yahoo.com
> >
> >
> 
> 
> --
> "http://ruby-lang.org -- do you ruby?"
> 
> Jeff Wood
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com