From: "Jörg W Mittag" Date: 2009-03-31T05:10:02+09:00 Subject: Re: Ruby 1.9.1-p0, llvm: all tests pass! Luc Heinrich wrote: > On 30 mars 09, at 00:11, Ollivier Robert wrote: >> One pet peeves of mine regarding 1.9 seems to have very recently been >> squashed by progress on llvm/clang front. Using r68016 of llvm/clang >> (pre-2.6), I managed to configure, make and make test of 1.9.1-p0 on >> my FreeBSD 7.1 box. > Version 0.5 of MacRuby apparently uses LLVM already: That's not what this is about. Ollivier's post is about compiling the C source code of YARV with the CLang/LLVM C compiler instead of the GNU C compiler. MacRuby 0.5 is about compiling Ruby source code to LLVM bitcode. > "This future > version of MacRuby is freakishly fast and uses the LLVM to generate > code for the Objective-C runtime." > > This article is rather an example of why premature benchmarking doesn't matter. We have seen several times now, that Ruby implementations tend to get orders of magnitude slower, as they become more and more Ruby compliant. Early benchmarks of YARV showed speedups of 3x, 5x, 8x, even up to 50x and more over MRI. According to a recent presentation by matz, the actual speedup for real applications instead of microbenchmarks and a real version of YARV instead of an early prototype is more like 1.5x. The same happened with MagLev: in the beginning there was much hype about how they built a prototype that can run WEBRick in just 6 weeks, that was up to 110x faster than MRI. In December of last year, they were much closer to actually passing the RubySpec suite. They also went from 100x faster to about the same speed as MRI. We should well expect that something similar will happen to MacRuby. I feel compelled to point out that both in the case of YARV and in the case of MacRuby, those hyper-inflated benchmark numbers were *not* actually posted by the project developers, but rather by over-excited fanbois. In fact, if you read the announcement of MacRuby 0.5 by project lead Laurent Sansonetti, you will find no mention of performance numbers, and rather a focus on compatibility and RubySpec compliance. And in the case of YARV, project lead Koichi Sasada has always made it very clear that the microbenchmarks shipped in the source distribution are by no means comprehensive or representative and were specifically written to highlight only those few very specific areas where YARV has dramatic performance improvements over other Ruby implementations. IronRuby project lead John Lam has similarly reported that IronRuby gets slower, the more Ruby features are supported. In the article, Antonio Cangiano declares MacRuby to be the fastest Ruby implementation. But that's simply not true. Yes, MacRuby is freakishly fast, but it's not a Ruby implementation. Not until it runs RubySpec, Rails, Merb, RubyGems, RSpec and all the other Ruby code. When Charles Oliver Nutter tried to benchmark MacRuby against JRuby, it crashed on half of the benchmarks. That's not freakishly fast, that's freakishly buggy. It's pretty easy to make Fixnum#+ run fast, if Fixnum#+ is all you can run. Just dropping in LLVM is not magically going to fix all performance problems. When the phc project (an LLVM-based compiler for PHP) first fired up their compiler, they discovered that their code ran 10x slower than the standard PHP interpreter. It took them a year and a PhD thesis worth of bleeding-edge original research to become faster than the interpreter. Granted, that was a static AOT compiler, not a dynamic JIT, but still, it's a cautionary tale. jwm