From: djberg96@... (Daniel Berger)
Date: 2004-08-26T06:10:38+09:00
Subject: StringIO affecting Benchmark results

Hi all,

Ruby 1.8.x (any version will do)

I was working updating my benchmark suite to generate a nicely
formatted HTML table using a combination of StringIO and HTML::Table
for various Ruby method benchmarks.  Unfortunately, I discovered that
using StringIO seems to alter the results of the benchmark itself.  I
can repeat this consistently.

There are two things to keep in mind initially.  First, the
BMSuite::StringSplitBench#start_benchmark (in the code below) finishes
before any data munging happens, so that's not the source of the
difference.  Second, most of the time the version that uses StringIO
is *faster* than the non-stringio version.

My guess is that it has to do with the Benchmark module and STDOUT. 
Also, does this mean then that the mere act of printing to STDOUT (or
not) is included in the benchmark itself?

Here is a sample class:

module BMSuite
   class StringSplitBench
      VERSION = "0.2.0"
      DEF_STRING = "The quick brown fox jumped over the lazy dog's
back"

      attr_accessor :max, :string, :directive
      def initialize(max = 40000, string = DEF_STRING)
         @max = max
         @string = string
         @directive = "a" * @string.length
      end

      def start_benchmark
         if $DEBUG
            puts "***************************"
            puts "* STRING SPLIT BENCHMARKS *"
            puts "***************************"
         end
         bm do |x|
            x.report("each_byte:"){
               @max.times{
                  @string.each_byte{|c| c.chr}
               }
            }
            x.report("split:"){
               @max.times{
                  @string.split('')
               }
            }
            x.report("unpack:"){
               @max.times{
                  @string.unpack(@directive)
               }
            }
         end
      end

   end
end

# Without using StringIO or HTML::Table
if $0 == __FILE__
   $:.unshift Dir.pwd

   ssb = BMSuite::StringSplitBench.new
   ssb.start_benchmark
end

# Using StringIO + HTML::Table
if $0 == __FILE__
   $:.unshift Dir.pwd

   require "stringio"
   require "html/table"
   include HTML

   t1 = Table.new{ |t| t.border = 1 }

   s = StringIO.new
   $stdout = s

   STDOUT.print "\n\nStarting benchmarks...\n\n"
   ssb = BMSuite::StringSplitBench.new
   ssb.start_benchmark

   s.rewind

   s.each_with_index{ |line,n|
      STDOUT.puts line
      line.gsub!(/\(|\)/,"")
      line.strip!
      t1 << Table::Row.new{ |r|
         if n == 0
            # Bump column headers over 1
            content = [""] + line.split + ["Iterations"]
            content.map!{ |e| e = "<font color='green'>" +
e.capitalize }
            r.content = content
         else
            r.content = line.split + [ssb.max]
         end
      }
   }

   File.open("stringsplit_results.html","w+"){ |fh| fh.puts t1.html }
   s.close
end

Regards,

Dan