Re: Performance of Ruby 1.9 vs. Ruby 1.8

> -----Original Message-----
> From: J旦rg W Mittag [mailto:[e-mail]]
> Sent: Tuesday, February 23, 2010 10:00 AM
> To: ruby-talk ML
> Subject: Re: Performance of Ruby 1.9 vs. Ruby 1.8
> 
> Chuck Remes wrote:
> > On Feb 23, 2010, at 7:50 AM, J旦rg W Mittag wrote:
> >> Alexander Jesner wrote:
> >>> On 02/22/2010 20:30, Benedikt M端ller wrote:
> >>>> Ruby is not the fastest interpreted language out there.
> >>> If you have not already done so, switch to Ruby 1.9.
> >> I have seen this claim that Ruby 1.9 is somehow faster than Ruby
> 1.8
> >> repeated over and over again, but I have *never* seen any
> credible
> >> evidence for that, neither in my own benchmarks nor in Antonio
> >> Cangiano's (or any other, for that matter). Does anyone have any
> >> evidence that this is actually the case? I would be very
> interested in
> >> that.
> > Are you looking at the same benchmarks and charts that I am?
> >
> > http://antoniocangiano.com/category/ruby-benchmark-suite/
> 
> Yes, I am.
> 
> > This clearly shows that 1.9.1 is consistently and measurably
> faster
> > than 1.8 in nearly every benchmark.
> 
> No, it doesn't. It shows that 1.9.1 running on one VM is faster
> than
> 1.8 on a *totally different* VM. That's shoddy benchmarking. If you
> want to benchmark 1.8 vs. 1.9, you benchmark 1.8 vs. 1.9 and not
> 1.8
> and some other things vs. 1.9 and some totally unrelated,
> completely
> different other things.
> 

The claim is that 1.9.1, in its entirety, is generally faster than 1.8.6, in its entirety.  The data backs that up.  (I'm looking at the "A faster Ruby on Windows is possible (benchmarks for 4 implementations inside)" benchmark.)  The fact that a new VM (YARV) is part of what makes 1.9.1 different from 1.8.6 is irrelevant.  If you trying to assert that the different VM is the _only_ reason for the performance, then yes, that would be a bad claim, because there are too many other variables and changes involved.  But they didn't; the comparison was between 1.9.1 and 1.8.6.

Your statement is a little akin to saying a speed comparison between a Nissan Cube and a Nissan GT-R is invalid because they have different engines.  Well, of course they have different engines; that's part of what makes them different vehicles.

(If you were referring to the benchmark including IronRuby, well that is why it was also compared with 1.8.6 and 1.9.1; that is a comparison of the IronRuby release, with its use of the .NET CLR, with the Ruby native releases, with their respective included VMs.  Again, it窶冱 the entire release that is trying to be compared.)

> This is statistics 101, and is usually called "controlling your
> variables", although the great Zed Shaw in his inimitable style
> said
> it much better: "if you want to measure shit, don't measure other
> shit."
> 

And the variables were controlled, as best they could be.  For the "A faster Ruby on Windows is possible (benchmarks for 4 implementations inside)" benchmark, they use the same system to run all tests.  They didn't run 1.9 on a faster system than 1.8, or vice versa.  They also include 2 different installer methods as a second variable in the mix.  The comparison of Windows to Linux (above the "faster Ruby on Windows..." benchmark) were on identical systems using identical Ruby releases.  They even used 32-bit versions of Windows and Linux to remove that variable as a difference.  Obviously different compilers are involved, because they are different platforms, but that gets stated as part of the setup description.

> You claim that the performance increase is due to Ruby 1.9. But how
> do
> you know that? 

Because that is what the numbers are telling us.

>    How do you know it is not due to YARV? Or different
> compiler options? Or different ./configure options? After all, it
> wouldn't be the first time that Antonio had to withdraw or correct
> his
> benchmark results.
> 

As far as the bulk numbers are concerned, who cares?  The test conditions were given, and even the code used to get those results are available.  So take advantage of another scientific principle, and _repeat_the_experiment_for_yourself_.  If you get widely different results, share them with a full explanation of your test setup and methodology.

If someone were interested in a detailed report about which differences between 1.8.6 and 1.9.1 provide what portion of the performance differences (or lack of differences, for some tests), that would be a different type of testing.  That would be profiling, with the need to identify which sections are comparable between 1.8.6 and 1.9.1 and which are different. 

(As to your point about Antonio withdrawing or correcting benchmark results, that's just a sign of responding to peer review.  If you can point out valid reasons that the results are invalid as given, or produce a large disparity from your own runs, share that so further corrections can be made.)

> > My own experience proves this out too.
> 
> Then show your results! As I wrote before: I would love to see
> them!
> 
> jwm

The flip-side is also fair.  You claim that your own benchmarks show no difference between 1.8 and 1.9.  Fine.  But you haven't gone into any detail about how you tested them (that I've seen): what was your test setup; how did you compile and install them; where is your code that you used to benchmark the two; show the data for the different runs that will provide the statistical rigor to justify a statement that "there's no significant performance difference."  Antonio's benchmarks have 51 different tests, some with different implementations, and most show that 1.9.1 is faster than 1.8.6, but not all.  Maybe your one test (or two?) happens to be similar to one of the ones which has 1.8.6 faster than 1.9.1.  Well, if that is indicative of the type of work that you always do, then yes, 1.8.6 would not be any faster than 1.9.1.  

It was mentioned by at least one other, but here is another benchmark: http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=yarv&lang2=ruby
It's data shows that for the majority of the benchmarks, Ruby 1.9 (yarv) takes less than half the time as 1.8 (MRI).  But out of the 11 tests, there were 3 where they were closer, and 1 where 1.9 took longer than 1.8 (pidigits).  But one test out of 11 does not, IMO, invalidate the claim that Ruby 1.9 is generally faster than Ruby 1.8.

Coey