Re: Ruby 1.9.1-p0, llvm: all tests pass!

Luc Heinrich wrote:
> On 30 mars 09, at 00:11, Ollivier Robert wrote:
>> One pet peeves of mine regarding 1.9 seems to have very recently been
>> squashed by progress on llvm/clang front.  Using r68016 of llvm/clang
>> (pre-2.6), I managed to configure, make and make test of 1.9.1-p0 on  
>> my FreeBSD 7.1 box.
> Version 0.5 of MacRuby apparently uses LLVM already: 

That's not what this is about. Ollivier's post is about compiling the
C source code of YARV with the CLang/LLVM C compiler instead of the
GNU C compiler. MacRuby 0.5 is about compiling Ruby source code to
LLVM bitcode.

> "This future 
> version of MacRuby is freakishly fast and uses the LLVM to generate  
> code for the Objective-C runtime."
> 
> <http://antoniocangiano.com/2009/03/29/why-macruby-matters/>

This article is rather an example of why premature benchmarking
doesn't matter.

We have seen several times now, that Ruby implementations tend to get
orders of magnitude slower, as they become more and more Ruby
compliant. Early benchmarks of YARV showed speedups of 3x, 5x, 8x,
even up to 50x and more over MRI. According to a recent presentation
by matz, the actual speedup for real applications instead of
microbenchmarks and a real version of YARV instead of an early
prototype is more like 1.5x.

The same happened with MagLev: in the beginning there was much hype
about how they built a prototype that can run WEBRick in just 6 weeks,
that was up to 110x faster than MRI. In December of last year, they
were much closer to actually passing the RubySpec suite. They also
went from 100x faster to about the same speed as MRI.

We should well expect that something similar will happen to MacRuby.

I feel compelled to point out that both in the case of YARV and in the
case of MacRuby, those hyper-inflated benchmark numbers were *not*
actually posted by the project developers, but rather by over-excited
fanbois. In fact, if you read the announcement of MacRuby 0.5 by
project lead Laurent Sansonetti, you will find no mention of
performance numbers, and rather a focus on compatibility and RubySpec
compliance. And in the case of YARV, project lead Koichi Sasada has
always made it very clear that the microbenchmarks shipped in the
source distribution are by no means comprehensive or representative
and were specifically written to highlight only those few very
specific areas where YARV has dramatic performance improvements over
other Ruby implementations.

IronRuby project lead John Lam has similarly reported that IronRuby
gets slower, the more Ruby features are supported.

In the article, Antonio Cangiano declares MacRuby to be the fastest
Ruby implementation. But that's simply not true. Yes, MacRuby is
freakishly fast, but it's not a Ruby implementation. Not until it runs
RubySpec, Rails, Merb, RubyGems, RSpec and all the other Ruby code.
When Charles Oliver Nutter tried to benchmark MacRuby against JRuby,
it crashed on half of the benchmarks. That's not freakishly fast,
that's freakishly buggy.

It's pretty easy to make Fixnum#+ run fast, if Fixnum#+ is all you can
run.

Just dropping in LLVM is not magically going to fix all performance
problems. When the phc project (an LLVM-based compiler for PHP) first
fired up their compiler, they discovered that their code ran 10x
slower than the standard PHP interpreter. It took them a year and a
PhD thesis worth of bleeding-edge original research to become faster
than the interpreter. Granted, that was a static AOT compiler, not a
dynamic JIT, but still, it's a cautionary tale.

jwm