From: Chuck Remes <cremes.devlist@...>
Date: 2010-02-24T03:30:38+09:00
Subject: Re: Performance of Ruby 1.9 vs. Ruby 1.8


On Feb 23, 2010, at 10:00 AM, Jörg W Mittag wrote:

> Chuck Remes wrote:
>> On Feb 23, 2010, at 7:50 AM, Jörg W Mittag wrote:
>>> Alexander Jesner wrote:
>>>> On 02/22/2010 20:30, Benedikt Müller wrote:
>>>>> Ruby is not the fastest interpreted language out there.
>>>> If you have not already done so, switch to Ruby 1.9.
>>> I have seen this claim that Ruby 1.9 is somehow faster than Ruby 1.8
>>> repeated over and over again, but I have *never* seen any credible
>>> evidence for that, neither in my own benchmarks nor in Antonio
>>> Cangiano's (or any other, for that matter). Does anyone have any
>>> evidence that this is actually the case? I would be very interested in
>>> that.
>> Are you looking at the same benchmarks and charts that I am?
>> 
>> http://antoniocangiano.com/category/ruby-benchmark-suite/
> 
> Yes, I am.

Oh, boy, here we go. By the tone of your response, I'm not sure if you're a jerk or I'm in a bad mood. I'll assume you are *not* and that I *am*. But just in case you are this guy...

http://xkcd.com/386/


>> This clearly shows that 1.9.1 is consistently and measurably faster
>> than 1.8 in nearly every benchmark.
> 
> No, it doesn't. It shows that 1.9.1 running on one VM is faster than
> 1.8 on a *totally different* VM. That's shoddy benchmarking. If you
> want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not 1.8
> and some other things vs. 1.9 and some totally unrelated, completely
> different other things.

I don't understand your point here. How can I benchmark 1.9.1 against 1.8.7 without the VMs being different? The whole point of the move to 1.9.1 was YARV and its new interpreter. The VMs are different and unless someone undertakes the effort to move the 1.8 syntax on top of YARV, that won't change. You mentioned this in two separate replies to this thread so I know it is an important point for you. You *cannot* bench 1.8 versus 1.9 without YARV entering the equation.

I suppose you are saying that there is no valid comparison between 1.8.x and 1.9.x because of this difference? If not, explain yourself better.

If you *do* believe that there is no valid comparison because the VMs are different, then please explain how 1.8.x and 1.9.x could be rigorously compared.

I don't know about you, but I run my ruby code on MRI, YARV, Java6 and LLVM (Rubinius and MacRuby). The VM doesn't matter to me as long as the results are the same and each platform gives me the performance I require. 

I don't know how you run your code without a VM; sounds neat.

> This is statistics 101, and is usually called "controlling your
> variables", although the great Zed Shaw in his inimitable style said
> it much better: "if you want to measure shit, don't measure other
> shit."
> 
> You claim that the performance increase is due to Ruby 1.9. But how do
> you know that? How do you know it is not due to YARV? Or different
> compiler options? Or different ./configure options? After all, it
> wouldn't be the first time that Antonio had to withdraw or correct his
> benchmark results.

You're looking for a level of rigor that just doesn't exist in software benchmarking and particularly for ruby open source. I ran these benchmarks on my system. I can give you the basic details but my machine likely performs differently from the *exact* same machine with the same software load on it. There are dozens of Linux distros all with differing patch levels; same goes for BSD, Windows and OSX. Maybe my machine runs a little hotter (therefore slower) than an exact clone; I'm not about to get a temperature controlled room to bench this language. All of these things are likely to effect the bench results.

I say, "So what." Unless there is a bug in a major system component (e.g. libc, pthreads, etc), the compiler toolchain, or a hardware defect, the results across comparable systems are pretty close. That is good enough.

Results from lots of independent sources show that 1.9 is faster than 1.8. Why? It's worthy of investigation by someone other than me. :)

>> My own experience proves this out too.
> 
> Then show your results! As I wrote before: I would love to see them!

I emailed my results to you using Cogniano's benchmark suite available from github. I used the source available via the "rvm" utility and benched both 1.8.7-p248 and 1.9.1-p376 built on my system (OSX Intel, 10.6.2, dual quad-core, blah blah blah).

Ruby 1.9.1 was faster in 60 (of 100+) tests, the same in 2 or 3, and slower in about 20. These are against the "micro" benchmarks in the test suite. In my world, the usual scripts I run are all noticeably faster with 1.9.1 versus any 1.8.x MRI branch.

cr