From: "M. Edward (Ed) Borasky" <znmeb@...>
Date: 2007-10-07T03:02:15+09:00
Subject: Re: Ruby vs. PHP

Chad Perrin wrote:
> On Sun, Oct 07, 2007 at 12:38:21AM +0900, M. Edward (Ed) Borasky wrote:
>> 1. The site is very clear about what it is doing, how it is doing it, 
>> why it is doing it and to a great extent how one can (not) and should 
>> (not) interpret the results. However, anyone is free to misinterpret any 
>> of that out of either ignorance or competitive marketing zeal.
> 
> I agree.  The site is useful -- just not in the immediately obvious way.
> For that purpose, it's not useful (where "useful" includes "good at
> providing strong technical evidence").  That's because the "immediately
> obvious way" is in comparing languages to decide what should be used for
> performance purposes between languages that are very close on the
> execution performance scale, and micro-benchmarks are notoriously bad at
> that, for a number of reasons.

Well ...

1. The site compares programming languages using micro-benchmarks, or, 
more accurately, "benchmarks small enough to be easily implemented using 
a variety of languages and covering a variety of commonly-used data 
types and algorithms."

2. I've asked the Shootout team for a spreadsheet of all the timings for 
all the languages, and if I get it, I'll post an analysis here (relative 
to MRI) and on the Shootout forum (relative to gcc).

3. I define "useful" the same way the Shootout defines it -- a way of 
comparing languages on a common set of benchmarks. If you have a problem 
with the Shootout, take it up with the Shootout, not with me.

2.

>> 4. I suspect the main problem the "majority of the community" has with 
>> the site is that it shows that the current (MRI) Ruby 1.8.6 
>> implementation is slower than other languages commonly used to implement 
>> web applications. And that is mostly competitive marketing zeal. Of 
>> course the Ruby community wants Ruby to be faster than Perl, Python, PHP 
>> or Java. And when it turns out that it isn't faster, there are two 
>> things that can be done:
> 
> I don't think that's entirely the case (or even close-to-entirely).  I
> don't just participate in the Ruby community -- I'm as much an on-again,
> off-again member of a couple other language communities as I am of the
> Ruby community here at ruby-talk (if not more so).  One in particular is
> the Perl community, and just to make a point I'll compare the two on this
> subject.
> 
> When people in each community look at comparative benchmarks for Ruby and
> Perl, and when they see that Perl fairly consistently kicks the crap out
> of Ruby for execution speed, I see that in *both* communities they tend
> to follow that up with something like ". . . but micro-benchmarks are
> pretty much useless.  Only real-world performance testing matters, and
> usually even that is just a case of premature optimization."  In other
> words, considering that even the usual "winner" in such a "contest"
> dismisses those benchmarks as mostly useless in that respect, I don't
> think the problem people in the "losing" camp have with the way people
> tend to want to use benchmarks is primarily one of wanting to ignore what
> shines unfavorable light in their direction.

That's an easy way to dismiss the results, but you're assuming

1. That micro-benchmarks are pretty much useless. I don't agree. 
Processors, compilers, interpreters, virtual machines and languages can 
only be measured and optimized using micro-benchmarks.

Consider Intel (or AMD, if you prefer). Could they measure *every* 
real-world application, ranging from reading email to web servers to 
scientific matrix algorithms to video editing to multi-player shootouts 
to text indexing to ... with *every* commonly-used compiler and 
interpreter? Of course not!

You *have* to reduce the benchmark set to something reasonable and make 
some simplifying assumptions. And in the end, you have to have some 
*statistical* way of saying that a chip design is "good enough", not 
that it is perfect. I don't call that "premature optimization", I call 
it PhD-level electrical engineering!

2. That competition doesn't matter. Again, I don't agree. If it doesn't 
matter, why do we do it? We long ago stopped racing horses because it 
"improved the breed" and continue to race horses only because it is a 
source of entertainment, and of tax revenue in some jurisdictions. But 
we race computer chips and languages and algorithms because it *does* 
improve the breed.

Maybe in the 60 or so years since computer racing started, we're 
starting to reach the point where it isn't improving the breed any more 
and is only a source of entertainment. But we still don't have a 
definitive answer to whether P = NP, we have "grand challenge" problems 
that call for peta-scale computing platforms and languages to make such 
problems easily solved, there appears to be a limitless market for 
devices that put the entire historical collection of recorded music in 
one's pocket, and so on.

To sum up, as a practicing performance engineer, I can say that there 
are ways to go from micro-benchmark results to "good enough but not 
perfect" predictions of real-world performance. That's why we do it and 
why it matters in a nutshell.

> 
> I see the same things being said by Perlists and Rubyists: that
> programmer productivity, flexibility, "real-world" performance,
> maintainability, and other factors are more important.  In either camp,
> people then tend to provide evidence that their chosen language is the
> best for these factors in some way -- at least for the purposes of the
> person talking.  As someone who likes and uses both Perl and Ruby, I find
> that both "sides" are right, too.  They're just right for differet use
> cases.

Well ... in the specific case of Perl vs. Ruby, I'm not by any means an 
expert on the use cases that differentiate between the two. Ruby is by 
design a *successor* to Perl, IIRC. So I would expect Ruby to provide a 
better "user experience" than Perl for a lot of use cases.

But I don't see how it's possible that language/implementation A can be 
significantly faster than language/implementation B on a varied set of 
micro-benchmarks and yet language/implementation B can be significantly 
faster than language/implementation A on more than, say, *one*, 
real-world application. Computers and mathematics just don't work that way.
>