RickCheck -- automated testing of general assertions

Hej hej, fellow Rubyists!

First of all, I'd like to say that my experience with Ruby stretches
no longer than to about a week ago.  I found Ruby very easy to learn,
and generally very fun to use!  Thanks for contributing to great
things.

That said, I want to talk about testing general assertions
automatically.  By ``general assertions'', I mean assertions that
assert some property of arbitrary objects.  For example, say we wanted
to test this function:

| def random_in_range(min, max)
|   entries = (min..max).entries
|   length = entries.length
|   entries[rand(length)]
| end

We could define a Test::Unit test case:

| class RandomFromTest < Test::Unit::TestCase
|   def test_random_from
|     random = random_in_range(5, 60)
|     assert(random >= 5, "#{random} >= 5")
|     assert(random <= 60, "#{random} <= 60")
|   end
| end
  
We run it and see this:

| Loaded suite RandomFromTest
| Started
| .
| Finished in 0.001889 seconds.
| 
| 1 tests, 2 assertions, 0 failures, 0 errors

And we could define a general assertion.  I'll show you how, first,
and then tell you what's going on.

| class RandomFromTest < QuickCase
|   def test_random_from
|     random = random_in_range(5, 60)
|     assert(random >= 5, "#{random} >= 5")
|     assert(random <= 60, "#{random} <= 60")
|   end
| 
|   def prop_random_from(a, b)
|     yield [Integer, Integer]
|     random = random_in_range(a, b)
|     assert(random >= a, "#{random} >= #{a}")
|     assert(random <= b, "#{random} <= #{b}")
|   end
| end

We run it, and get:

| Loaded suite RandomFromTest
| Started
| .E
| Finished in 0.003386 seconds.
| 
|   1) Error:
| prop_random_from(RandomFromTest):
| NoMethodError: undefined method `>=' for nil:NilClass
|     ./rcheck.rb:49:in `call' (given [233, 86])
|     ./rcheck.rb:80:in `test_arbitrary_cases'
|     ./rcheck.rb:77:in `times'
|     ./rcheck.rb:77:in `test_arbitrary_cases'
|     ./rcheck.rb:90:in `verify_method'
|     ./rcheck.rb:156:in `run'
| 
| 2 tests, 4 assertions, 0 failures, 1 errors

Whoops!  Why's that?  Thinking through random_from tells us that it's
because (223..66) makes a Range of no entries!  We probably want
random_from to handle backwardsy stuff, too.  So we hack away!

| def random_in_range(a, b)
|   min, max = [a, b].sort
|   entries = (min..max).entries
|   length = entries.length
|   entries[rand(length)]
| end

Running the test now produces

|   1) Failure:
| prop_random_from(RandomFromTest)
|     [./rcheck.rb:49:in `call' (given [-26, -42])
|      ./rcheck.rb:80:in `test_arbitrary_cases'
|      ./rcheck.rb:77:in `times'
|      ./rcheck.rb:77:in `test_arbitrary_cases'
|      ./rcheck.rb:90:in `verify_method'
|      ./rcheck.rb:156:in `run']:
| -26 <= -42.
| <false> is not true.

This reflects an unfortunate truth about tests: they can be buggy,
too!  (I didn't expect this unromantic twist, but hey, that's honesty
for you.)

The problem is that the test needs to know about the ordering, too.
Else it can treat the smallest number as the maximum.  Here's a better
test:

| def prop_random_from(a, b)
|   yield [Integer, Integer]
|   min, max = [a, b].sort
|   random = random_in_range(a, b)
|   assert(random >= min, "#{random} >= #{a}")
|   assert(random <= max, "#{random} <= #{b}")
| end

This produces the pleasant:

| Loaded suite RandomFromTest
| Started
| ..
| Finished in 0.663505 seconds.
| 
| 2 tests, 1002 assertions, 0 failures, 0 errors

Yay!  OK, I'll explain how it works, now.

The whole point of this library is to automatically test some
assertions for arbitrary inputs.  Hence, we need a way to get
arbitrary inputs.  But we can't really do that without knowing what
kinds of inputs the function expects.  Therefore, the function needs
to tell the library what it wants.  I chose to use yield for this.

To say what you want, you yield some `type specifiers'.  This probably
brings images of static typing, complete with whips and straps.  But
even the statically inclined have good ideas once in a while!

To be honest, I don't have anything to do with this idea in
particular.  I stole it all from QuickCheck[1].  But I digress.

A type specifier can be a class.  This is a type specifier: Integer.
Or it can be a compound of stuff.  This is also a type specifier:
[Integer, {:max => 40}].

After you've told the library what you want, it'll make some of that,
and call you a bunch of times with it.  A whole bunch of times -- 500
times, by default!  It doesn't actually know how to generate data
itself.  For that, it sends `arbitrary' to the class you requested.

Here's the definition of Integer#arbitrary:

| class Integer
|   def Integer.arbitrary(opts = {})
|     min = opts[:min] || -100
|     max = opts[:max] || 500
|     if min == max then
|       min
|     else
|       rand(max - min) + min
|     end
|   end
| end

So you see, the default minimums and maximums are -100 and 500.  Good
to know.

Maybe you also see that it's very easy to define your own arbitrary
methods.  Which is a good thing, because the ones I defined kind of
stink.  Here's some typical output from String#arbitrary:

| irb(main):001:0> String.arbitrary
| => "We have already seen that the theory of syntactic features
| developed earlier does not readily tolerate the strong generative
| capacity of the theory."
| 
| irb(main):002:0> String.arbitrary =>
| "Furthermore, an important property of these three types of EC is not
| to be considered in determining the strong generative capacity of the
| theory."
| 
| irb(main):003:0> String.arbitrary =>
| "For one thing, the fundamental error of regarding functional notions
| as categorial is unspecified with respect to a stipulation to place
| the constructions into these various categories."

If you've read Chomsky, you might recognize the style.  Anyways, I
plan to include more useful arbitraries in the future.

Enough jabbering.  The code's at
http://www.phubuh.org/rickcheck.tar.bz2, or if you wanna, you can grab
it from my Darcs[2] repository:

| darcs get --verbose http://www.phubuh.org/repos/rickcheck

I'd love any feedback!  Have fun!


Footnotes: 
[1] http://www.cs.chalmers.se/~rjmh/QuickCheck/
[2] http://abridgegame.org/darcs/


--
brought to you by Mikael `phubuh' Brockman!
DE!  MORE CODE!  MORE CODE!  MORE CODE!  MORE CODE!  MORE CODE!  MORE CO