From: "M. Edward (Ed) Borasky" <znmeb@...>
Date: 2006-09-15T14:40:53+09:00
Subject: Re: Metaruby, BFTS, Cardinal and Rubicon - State of play?

Austin Ziegler wrote:
> On 9/14/06, Ryan Davis <ryand-ruby@zenspider.com> wrote:
>> On Sep 13, 2006, at 7:59 AM, Austin Ziegler wrote:
>> > I'd be happier if Mr Tew didn't try to lend legitimacy to the alioth
>> > shootout. Microbenchmarks don't show anything useful, even if they're
>> > run correctly -- which the shootout has never been run correctly. It
>> > isn't even administered correctly. (I was similarly annoyed that Joel
>> > Spolsky used it in his latest slam on Ruby. Stupid, Joel, stupid.)
>> Thank you, once again, for derailing a thread with your personal
>> vendetta.
> 
> Look. We *know* that the shootout is crap. We've known this for three
> years now. But we *still* have people come in and use it for a variety
> of reasons, most of which are completely bogus.

Before tossing the whole shootout overboard, I'd like to get my hands on
the complete set of timings for all the benchmarks for all the
languages. I described roughly the process in another email.

> If we, as Ruby users, want to have a set of benchmarks, we should
> develop our own and hold them to a rigorous standard. That is, the
> exact *opposite* of what Mr. Guouy's shootout does.
> 
> I'd gladly support a Ruby benchmark suite that people could use in
> talking about things. But in a variety of different threads we've seen
> that not only don't the shootout people know anything about
> benchmarks, most other people don't either (see the "For
> performance..." thread).

If the BFTS has room, why shouldn't benchmarks be part of the test
suite? I've said before, if you aren't running performance tests, you
aren't doing test driven development. :)

There are some Ruby benchmarks -- the YARV project has a small suite
they use, and there's my MatrixBenchmark.
> 
> Zed Shaw has had to do a lot of teaching about statistics. I'm sure
> that Ed Borasky could teach us a lot about benchmarking with *good*
> benchmarks to be tested under controlled or controllable situations so
> we could improve the performance of Ruby in various situations.
Again, see my other email ... I think there's something to be gained
from at least a rough analytical pass at the data from the shootout.
What I proposed would at least identify those benchmarks in the set that
are reasonable indicators of comparative language performance and which
are "special cases".

And despite the objections I've heard, I think it's a perfectly
reasonable idea to compare *general-purpose* dynamic scripting languages
like Perl, Python and Ruby on benchmarks. People are going to ask the
question; we might as well see what sort of confidence one can have in
the answer the shootout gives. Right now, what we have is Ruby turning
out near the bottom on the overall score.