From: Robert Klemme <shortcutter@...>
Date: 2007-03-07T07:15:07+09:00
Subject: Re: Automatic Benchmark Iterations

On 06.03.2007 22:27, Phrogz wrote:
> When performing a benchmark comparison of several techniques, this
> idiom is very common for me:
> 
>   N = 1_000_000
>   Benchmark.bmbm{ |x|
>     x.report( 'foo' ){
>       N.times{
>         # foo code
>       }
>     }
>     x.report( 'bar' ){
>       N.times{
>         # bar code
>       }
>     }
>     x.report( 'jim' ){
>       N.times{
>         # jim code
>       }
>     }
>   }
> 
> Observation #1: It's annoying to have to type "N.times{ }" for each
> report. I also want the exact same value, and I always want some
> iterations.
> 
> Consequent Desire #1: It'd be nice if I could write the above as:
>   Benchmark.bmbm( 1_000_000 ){ |x|
>     x.report( 'foo' ){
>         # foo code
>     }
>     x.report( 'bar' ){
>         # bar code
>     }
>     x.report( 'jim' ){
>         # jim code
>     }
>   }
> 
> Observation #2: I'll frequently start with a 'guess' value of
> N=1_000_000, but if it's taking too long, I'll start playing with
> values of N until I get something that takes 1-5 seconds for the first
> compared value.
> 
> Consequent Desire #2: It'd be cool if I could write the above as:
>   Benchmark.autobm{ |x|
>     x.report( 'foo' ){
>         # foo code
>     }
>     x.report( 'bar' ){
>         # bar code
>     }
>     x.report( 'jim' ){
>         # jim code
>     }
>   }
> ...and it would figure out an appropriate number of iterations for me.
> (One approach would be to time one iteration, figure out an
> appropriate multiple, try benchmarking that number of iterations, and
> adjust the number iterations if the result is outside some desired
> range. Rinse/repeat.)
> 
> 
> The Benchmark::Report#item method (aka #report) is implemented as
> this:
> 
>     def item(label = "", *fmt, &blk) # :yield:
>       print label.ljust(@width)
>       res = Benchmark::measure(&blk)
>       print res.format(@fmtstr, *fmt)
>       res
>     end
> 
> My question: What is the least-overhead way you can come up with to
> wrap the supplied block in an iteration-controlled loop? For example:
> 
>   def item( label="", *fmt, &blk )
>     ...
>     res = Benchmark::measure( &lambda{ for i in 1..@iterations;
> blk.call; end } )
>     ...
>   end

This is one option - doesn't even require changing existing methods... 
  Number guessing left as exercise for the reader. :-)

Kind regards

	robert


robert@fussel /cygdrive/c/temp
$ ruby bmext.rb
                           user     system      total        real
foo                   0.000000   0.000000   0.000000 (  0.000000)
counter=1000
Rehearsal -------------------------------------------------------
foo                   0.016000   0.000000   0.016000 (  0.015000)
---------------------------------------------- total: 0.016000sec

                           user     system      total        real
foo                   0.000000   0.000000   0.000000 (  0.000000)
counter=2000

robert@fussel /cygdrive/c/temp
$ cat bmext.rb
require 'benchmark'

module Benchmark
   BmProxy = Struct.new :parent, :iter do
     def method_missing(s,*a,&b)
       parent.send(s,*a) { iter.times { b[] } }
     end
   end

   def self.method_missing(s,*a,&b)
     n = a.shift || 1
     m = s.to_s[1..-1]
     send(m,*a) {|x| yield BmProxy.new(x,n)}
   end
end

$i = 0

Benchmark.xbm 1000, 20 do |x|
   x.report 'foo' do
     $i += 1
   end
end

print "counter=", $i, "\n"

$i = 0

Benchmark.xbmbm 1000, 20 do |x|
   x.report 'foo' do
     $i += 1
   end
end

print "counter=", $i, "\n"

robert@fussel /cygdrive/c/temp
$