Re: python/ruby benchmark(don't shoot the messenger)

In article <[e-mail]>,
Austin Ziegler  <[e-mail]> wrote:
>On 6/11/05, Alexandru Popescu <[e-mail]> wrote:
>> I've run through this thread and i am finally writting this
>> because i am in a way astonished.
>>=20
>> Afaik Python uses bytecodes while Ruby is completely
>> interpretative, so imo there is not possible to drive conclusions
>> from this benchmark.
>
>Which is one of the many reasons that I have a problem with the
>alioth shootout. Ultimately, it tries to treat compiled, semi-
>compiled, and interpreted languages equally. They're not. Compiled
>languages will be faster than semi-compiled (bytecode), which will
>be faster than interpreted. The level of speed difference is a
>matter of scale, and that scale also matters on development time and
>clarity.

I have to say that I tend to agree with the astonishment.

Here's what I mean:
Someone comes along and asks about python vs. Ruby performance and 
references the alioth benchmarks.  Several folks jump on the alioth 
benchmarks and call them bogus.  I see this as blaming the messanger.

The fact is that these benchmarks exist and they cover lots of different 
types of algorithms.  They're on the web and people will look at them - 
no way to stop that from happening.

Someone mentioned the Ackermann benchmark so I had a look and found that 
gawk did much better than Ruby - that's pathetic folks.  Now, I know that 
if you could do this in gawk it should be possible to do it in Ruby as well.
There aren't even mumbers for N=8 or N=9 for the Ruby version.  Maybe it's 
because the Ruby version was written poorly.  Maybe it's because of some 
issue within the Ruby interpreter itself - I don't know, I didn't really 
take time to look into it just now.  Either way, wouldn't it be valuable 
to try to improve our score on that benchmark?  If it's because it's 
poorly written, then it's gravy.  If it's an issue with the interpreter 
then maybe it would be good to know about the issue for the YARV design.

Now sure you can say that the benchmarks are bogus (and who knows, maybe 
they are) and then take all your marbles and go home, but the benchmarks 
remain and other languages communities are trying to hone their numbers.  
If we totally sit it out, well, that's not going to look too good.

Sure these benchmarks don't tell the whole story about how nice it is to 
develop in Ruby vs. gawk, but performance _is_ an issue for some people 
and they will use this set of benchmarks (or another) as a factor in 
deciding which langauge to use.  If Ruby consistently shows up down 
towards the bottom of the list... well, it can give a false impression.  

If we tell them that they really shouldn't worry about performance (and 
who knows, maybe we would be right in doing so) it just sounds like so 
much excuse-making.

I think that as a community we should be trying to compete in these 
benchmarks just like other langauge communities are doing.  Perhaps the 
results  can help us as we do move to a VM - if nothing else to show 
that we're making improvements (look at them as a set of unit tests for 
performance).  We know that Ruby needs help in the 
performance area.  If we keep telling ourselves that Ruby is 'fast 
enough' for our application (and it may well be) are we going to be 
sitting still while other languages improve performance?  Have we been 
the hare for the last few years sitting around while the tortise (TCL, 
for example, once considered a lot slower than Ruby) is passing us up 
(by creating or improving their VMs, for example).

Of course we know that we can do all sorts of things to speed things up 
by writing C extension code, for example - that's a given, and the fact 
that it's so easy to do in Ruby makes it possible to get into a lot of 
areas where you might normally need to consider a compiled langauge, 
however, a lot of people out there want to see how Ruby performs when 
you just write Ruby code (C is a compiled language, of course).  In some 
sense marrying Ruby and C might seem like cheating to them, I suppose (try 
to see it from their perspective).

If it looks like some of the benchmarks are misleading or stacked against 
us, maybe we can propose other benchmarks where Ruby shines (BTW: why 
does it say that the Object Methods benchmark is being deprecated?), 
after all, it appears to be an open source process.

Bottom line: Ruby needs help in the performance area.  Let's admit that 
and work on improving in that area instead of shooting the messenger - 
it'll make us look like whiners if we keep doing the latter.  I'd really 
rather not have the Ruby community perceived that way from the outside.

In the meantime, perhaps we could have some Ruby quizzes that focus on 
choosing a benchmark and optimizing it.  I think we would be 
better off as a community if we participate in the process rather than 
boycott it.

....just my 2cents.  

Phil