From: Eric Mahurin <eric_mahurin@...>
Date: 2005-09-28T04:48:09+09:00
Subject: Re: random testing with Test::Unit

I gave another shot at making a derived TestCase class.  I only
needed one hack to effectively make it ignore this class when
running tests: undef_method(:default_test).  Here is what I
came up with that has many of the features I wanted:

require 'test/unit'

module Test   
  module Unit
    class RandomTestCase < TestCase
      def self.suite
        $debug_level = (ENV['DEBUG_LEVEL']||0).to_i if
          $debug_level.nil?
        if $random_seed.nil?
          $random_seed = ENV['RAND_SEED'].to_i.nonzero? ||
            (srand;srand)
          srand($random_seed)
          puts("random_seed: #{$random_seed}") if
            $debug_level>=1
        end
        random_iterations = (ENV['RAND_ITER']||8).to_f
        methods = Regexp.new(ENV['TEST_METHODS']||".*")
        suite = super
        tests = suite.tests
        tests.reject! { |t|
            !methods.match(t.name.gsub(/\Atest_/,''))
        }
        (class << tests;self;end).class_eval {
          def each
            catch(:stop_suite) {
              (@iterations*size).to_i.times {
                catch(:invalid_test) {
                  yield(slice(rand(size)))
                }
              }
            }
          end
        }
        tests.instance_eval { @iterations = random_iterations }
        suite
      end
      undef_method(:default_test)
      def teardown
        if not passed?
          puts("\nrandom_seed: #{$random_seed}")
          throw(:stop_suite)
        end
      end
    end
  end
end


Here is that example I gave earlier using the above for random
testing Array#push and Array#pop using AOP:


class ArrayAOP < Array
  include Test::Unit::Assertions
  def push(*args)
    print("#{self.inspect}.push(*#{args.inspect}) -> ") if 
      $debug_level>=1
    n = size
    na = args.size
    ret = super
    assert_equal(n+na,size)
    assert_equal(slice(-na,na),args)
    assert_same(self,ret)
    p(ret) if $debug_level>=1
    ret
  end
  def pop
    print("#{self.inspect}.pop -> ") if $debug_level>=1
    n = size
    ret0 = n.nonzero? ? slice(-1) : nil
    ret = super
    assert_equal(ret0,ret)
    assert_equal(n.nonzero? ? n-1 : 0,size)
    p(ret) if $debug_level>=1
    ret
  end
end

class ArrayTest < Test::Unit::RandomTestCase
  def self.suite
    @@object = ArrayAOP[]
    super
  end
  def test_push
    args = []
    rand(4).times {
      args << [0,"",[],nil,false][rand(5)]
    }
    @@object.push(*args)
  end
  def test_pop
    rand(4).times {
      @@object.pop
    }
  end
end


As you can see from above, with this RandomTestCase class, you
can do random testing very easily.  But, it still would be nice
to get options from the command line rather than the
environment as I'm doing in RandomTestCase.

--- Eivind Eklund <eeklund@gmail.com> wrote:

> On 9/26/05, Eric Mahurin <eric_mahurin@yahoo.com> wrote:
> >
> > Does anybody else do random testing on their ruby code
> besides
> > me? I picked it up because of my background (IC design)
> where
> > random testing is one of the verification techniques of
> > hardware.
> 
> 
> I cannot remember having used random testing for Ruby code,
> though I've used
> it for code in other languages. Having a nice framework for
> it would be,
> well, nice.
> 
> For all of these strategies, I've had to hack up Test::Unit
> in
> > the same way to add various features. Here are the things
> I've
> > added:
> >
> > + instead of running the test_* methods in a fixed order
> once,
> > run them in a random order for N passes.
> 
> 
> Running N times sounds like a subclass of TestCase would be
> appropriate.
> Otherwise, how do you distinguish between what tests should
> be run once and
> which should be run several time (for statistical coverage)?
> 
> Random order sounds reasonable for everything, anyway.
> 
> 
> + abort testing when the first failure is reached.
> 
> 
> Also subclass material.
> 
> + easy way to skip the rest of the current test (when the
> > random input is invalid). Don't include this in the number
> of
> > tests or mark it as skipped.
> 
> 
> Also subclass material, I think.
> 
> + way to pass number of passes of the test_* methods on the
> > command line.
> >
> > + display random seed and way of passing it in on the
> command
> > line.
> >
> > + test suite should have access to the --verbose level or
> > another switch to control the debugging verbosity.
> 
> 
> My gut feeling is that this is inappropriate. I see a major
> part of the
> Test::Unit style testing as the fact that the programmer only
> has to check
> for pass/fail. Passing through verbiosity information seems
> to turn this on
> its head, encouraging setting up the output for inspection.
> (On the other
> hand, I have sometimes needed information for debugging -
> calling this
> "debug level" would do much to ease my concerns...)
> 
> + way to pass command-line options down to the test suite to
> > control various things - what methods to test, what
> > classes to test, various other flags.
> 
> 
> I also get an initial bad feeling about this, as it makes it
> totally
> necessary to use the console-based testrunner.
> 
> Eivind.
> --
> Hazzle free packages for Ruby?
> RPA is available from http://www.rubyarchive.org/
> 



		
__________________________________ 
Yahoo! Mail - PC Magazine Editors' Choice 2005 
http://mail.yahoo.com