From: Rick DeNatale <rick.denatale@...>
Date: 2006-09-05T22:25:50+09:00
Subject: Re: Benchmarking (Was: Re: Compiling Regexp only once)

On 9/5/06, Eric Hodel <drbrain@segment7.net> wrote:
> On Sep 4, 2006, at 8:06 PM, Rick DeNatale wrote:
>
> > So, I've corrected the benchmark.  I've also added another method
> > using String#split as Eric suggested.
> >
> > Here's the code being benchmarked:
> > rick@frodo:~/rubyscripts$ cat stringsplit.rb
>
> Putting the code here involves an extra method call.  While they are
> all wrapped in a method call equally, doing less is always better.
>
> > class String
> > [methods]
> > end
> >
> > Note that I made tested three different "pre-compiled" regex's one
> > with a Regex.new('.'), one with Regex.new(/./), and one a constant
> > with the literal regex /./.
>
> You don't need three different benchmarks for this.  Its easy to
> determine that these are the same with irb.
>
> > Now here's the benchmark:
> > rick@frodo:~/rubyscripts$ cat benchstringsplit.rb
> > require 'benchmark'
> > include Benchmark
> > load 'stringsplit.rb'
> >
> > iters = 100
>
> 100 iterations is never enough.  GC behavior, other processes waking
> up, etc. will all cause fluctuations in the benchmark.
>
> Use 100_000 or 1_000_000.
>
> > [benchmark code]
>
> Here's a more-meaningful version of yours:

[ Eric's benchmark code]

then a lot of other insightful stuff.

> But ultimately, these types of microbenchmarks are not very useful.
> Yes, you will get a speedup using /./ over RE, but will your program
> really run long enough where it matters to make a difference?  I bet
> not.
>
> > So as usual the right approach is test, profile to find out what needs
> > improvement, and benchmark.
>
> Be careful with your benchmarks, they are most useful when they are
> as small and simple as possible.  Be sure to throw away all the
> irrelevant parts.
>
> Be careful with your benchmarks, they are most useless when they are
> as small and simple as possible.  Be sure to understand how little
> speedup you'll get.

Which is the point after all.  I think that we are actually in violent
agreement about the role of benchmarking.

The key thing is a useful benchmark is crafted to show the performance
at a functional level which affects the particular application.

My benchmarks came out of another thread in which I proposed some code
and someone made the observation that my use of unpack was a quick way
to split up a string into individual 1-character strings.  I did a
benchmark to see if I could find a faster way, and when the results
seemed interesting to the "Compiling Regexp only once" thread, I
mentioned there, when it was pointed out that I had a typo which
invalidated that benchmark and re-did it.

I benchmarked several different approaches to splitting a string into
single character strings, a slightly higher level function than just a
regexp match.  And the results indicate that at that level using
literal REs doesn't really make much difference.

Doing something a particular way by rote, because you heard X, and
assume that X both applies to your situation and still applies, might
get your code written, but might over time, turn out to be folk
wisdom.

I'm old enough to remember when some folks programmed in PL/I, the
early PL/I compiler did a very poor job of generating code for
subroutine calls, which led to PL/I "best practice" documents
recommending that subroutines be avoided at all costs!  Subsequent
compilers made this harmful advice unnecessary.  Of course avoiding
subroutines is a much bigger deal than exactly how best to represent a
regexp in source code.

The bottom line is that as is often said "premature optimization is
the root of all evil."  Better to first write clearly, then test, and
then, if there is a performance problem, fix it by benchmarking,
profiling and re-coding/re-factoring.

I hope that we agree on that.
-- 
Rick DeNatale

My blog on Ruby
http://talklikeaduck.denhaven2.com/