From: "M. Edward (Ed) Borasky" <znmeb@...>
Date: 2006-07-17T22:13:41+09:00
Subject: Re: How to speed up ruby and make it as fast as possible

Alex Young wrote:
> I've never heard of one, but I can't think of anyone better than a 
> professional performance consultant to drive one :-)
Well, I'm on the YARV mailing list, and I just saw a message go by on a 
similar subject. They do have a benchmark they use, which looks like a 
mix of "traditional" small benchmarks like Tower of Hanoi and some 
benchmarks designed to deliberately exercise Ruby internals. I'm going 
to see if I can get a copy and have a look at it. No sense re-inventing 
a wheel, especially if four of them make YARV go faster. :)
>
> If they were to be chosen as the focal point for performance 
> improvement, the things I'd benefit most from being benchmarked are:
>
> - REXML
> - Mechanize
> - WEBrick
> - Test::Unit
Yeah, those look like "typical Ruby tasks" to me.
>
> I've pointed out higher-level libraries rather than core because they 
> exercise more of Ruby with fewer data points.  Is that the right 
> approach?  I realise that one downfall of this perspective is that the 
> library authors may have avoided certain techniques precisely because 
> they are too slow in the current Ruby implementation, so those 
> techniques would be completely sidelined.  Continuations are one that 
> instantly spring to mind - I can only find callcc used in one file in 
> the stdlib.
>
> An alternative would be to use rubicon's full run time as the 
> benchmark - that way we know that all the language features are being 
> flexed, but some might get too much weighting.
>
> Thoughts?
A benchmark is little different from an automated unit test. The 
"assertions" are simply replaced with (hopefully accurate) timings. It's 
a little more complicate than that, but that's the general idea. Once 
you have a suite of benchmarks, the interesting thing is that some 
tunings to the "compiler" or the runtime environment will make tests 
uniformly better, some uniformly worse and some will induce "tradeoffs". 
That's where the statistics comes in -- reducing a list of relative 
performance numbers to a single "figure of merit", usually with some 
weighting based on known usage patterns.