From: Isaac Gouy <igouy@...>
Date: 2005-08-17T23:51:14+09:00
Subject: Re: Ruby Performance

Hi Matthew

-snip-
> By all means, carry on with experimentation with all of the languages
> that you can get your hands on.  But let's not pretend that we're
> measuring anything.

On the contrary, we are measuring 'something'.

imo The issue is interpretation - what relevance (if any) do those
measurements have, what worth (if any) do those measurements have, what
(if anything) do they mean?


-snip-
> >Some perspective on how performance varies between programming language
> >implementations and tasks.
> >
> This is a part of what I take issue with.  The site offers a great deal
> of disclaimers regarding the worthlessness of the "benchmarks" there.
> But most of the space on the main page is devoted to reporting and
> comparing these benchmarks.

Flawed is not a synonym for worthless ;-)


-snip-
> >History - Doug Bagley's Shootout begat Aldo Calpini's Win32 Shootout
> >begat Brent Fulgham's Shootout.
> >
> With regard to your justification of the name, I'm not sure that history
> is a good reason to keep it.

You may be right about that.


-snip-
> When you follow the Ruby vs Perl link, you are treated to a nice picture
> and some numbers, all of which (you would assume from reading the page)
> offer some insight into the comparative merits of the languages.
>
> In this case, the site _does_ proclaim a winner in any number of categories.

Yes - under said conditions, said program X was measured to be A,B,C
and said program Y was measured to be D,E,F.

imo that's not The issue.


> The site talks out of both sides of its mouth.  Even with all of the
> language condemning the "simplistic notion" of winners and losers, the
> site still offers results.  It is these results that are dishonest and
> misleading.

Let me clarify what I meant by "simplistic notion".

I meant it's simplistic to measure one program using language
implementation A, and one program using language implementation B, and
then *generalize* to say that language A is faster than language B.

I meant it's simplistic to measure cpu time without measuring memory
use, and then claim that program A is faster than program B.

It's simply true that under said conditions, said program X was
measured to be A,B,C and said program Y was measured to be D,E,F -
there's nothing dishonest about those results.


> I think that the site would work much better _without_ the results
> page.

Which page is "the results page"?


> Why not let the curious discover their own results?  Hands on
> experimentation is the best way to explore the concepts that you are
> interested in.  Maybe you can find a way to offer free shell accounts
> that give access to the tools required to experiment with these
> "benchmarks".
>
> Or at the very least, take the focus off of those confounded numbers.