From: Chad Perrin Date: 2007-10-07T12:28:09+09:00 Subject: Re: Ruby vs. PHP On Sun, Oct 07, 2007 at 09:58:02AM +0900, M. Edward (Ed) Borasky wrote: > M. Edward (Ed) Borasky wrote: > > OK ... here's the analysis. The attached PDF is what's known as a box > and whisker plot, usually shortened to "boxplot". The raw numbers that > went into this are from the Alioth shootout page, and what I *haven't* > done is checked out which versions of Perl, Python, YARV, Ruby, jRuby > and PHP these tests used. They could be years old or they could have > been run yesterday morning. :) I discarded all the tests for which there > were any missing values. Nice. I wasn't expecting anything as effective for a quick, by-the-pants analysis as a box plot. What did you use to develop the graphic? I'm curious. . . . > > The value plotted is the ratio of seconds for the benchmark on the > dynamic language to the seconds for the benchmark with gcc. Thus, gcc > equals 1.0 across the board and lower is better/faster. How do you > interpret these plots? Correct me if I'm wrong, but . . . won't doing this as a gcc-comparative ratio bias slower languages toward an exaggerated upper bound on the box? I admit I haven't done this sort of thing in a while, so my sorta heuristic analysis of what I see may be prone to minor glitches. > > The whisker on the bottom is approximately the 5th percentile. In other > words, only five percent of the time is your performance going to be > that good or better. And the whisker on the top is approximately the > 95th percentile -- only five percent of the time is it going to be that > bad or worse. Both yarv and ruby appear to be prone to low-end statistical outliers, or perhaps to a "long tail" on the slow side of things. > > The bottom of the box is the 25th percentile. 25 percent of the time, > the performance will be that good or better. The top of the box is the > 75th percentile. 25 percent of the time it will be that bad or worse. > > So now we can make more precise Chad Perrin's notion of "general > neighborhoods of performance". We do this by looking at the height of > the median and the width of the box. And we see that relative to gcc, > YARV, Python, Perl and PHP have fairly close medians and the boxes are > about the same width. So they are all "in the same neighborhood" and gcc > is faster. Ruby and jRuby are *not* in the same neighborhood. Actually, the ruby and php medians (using lower-case to denote implementations) are almost identical, and perl's isn't far off, according to this. It appears that only in instances approaching worst-case does ruby really begin to suffer. Also, if I'm not entirely missing the implications of a gcc-ratio comparison, I'm not entirely sure that we can trust the huge visual differences in the height of the boxes to indicate significant performance problem areas. I'm also not sure the jruby implementation's numbers here are representative of JRuby's strengths. Does JRuby benefit from the benefits of Java's optimizing VM for long-running processes? I know it's probably suffering, in micro-benchmarks, from increased start-up times as it loads the VM. I'm surprised to see python's median so low. Are these benchmarks heavy on bytecode-optimized Python, or do they tend to use standard interpreted Python? > > Now suppose your boss comes to you, as bosses do, and says, "well, all > them high-falutin' box plots are dandy, but the board of directors wants > one number for each language!" It turns out (and I'll let Google and > Wikipedia fill in the blanks for you) that the one number you want to > give your boss, aside from your cell phone number, is the *geometric > mean* of all the benchmark ratios. Again, smaller is better. So here's > how the languages stack up: > > gcc 1.0 > yarv 13.2 > perl 14.0 > python 14.2 > php 15.5 > ruby 29.9 > jruby 55.0 That's a little clearer (barring the problem of whether jruby benefits from the optimizations for long-running processes), but of course suffers from the lack of attention to statistical outliers and low-end performance curves. That's one of the reasons I like box plots. > > So yes, you can pretty much expect the same performance from YARV, Perl, > Python and PHP. And you can pretty much expect something like a 13 - 16 > to one speed improvement if you decide to rewrite your application in C. > > It's pretty clear to me from these numbers is that the only reason that > deploying web applications on the LAMP stack and its cousins using > PostgreSQL, Perl, Python and Ruby is economically viable is that they > spend most of their time either in the database or on the network. The > bad news is that for an application with 100 percent dynamic language > code -- no C libraries for the intensive calculations, no highly-tuned > web servers, databases or other components -- you're going to end up > throwing twice as much hardware at scaling problems in Ruby as you will > in PHP, Perl or Python. The good news is that YARV will level the > playing field. Unless you're implying a different set of working conditions, you're not going to throw twice as much hardware at the Ruby implementation, because the bottlenecks for dynamic language web development (assuming good design) still haven't changed. They're still I/O and network traffic of various sorts, not the languages. In my experience, that's pretty much the sort of thing that happens with *everything* that is lax enough on performance needs to "settle" for something like Perl, et al. That's the neighborhood I'm talking about for performance: not so slow it can't be used for a common desktop app, not so fast that it should be used for hard-core number-crunching or graphics-intensive game development. Neighborhoods are relative, after all. There's the neighborhood with C/C++, OCaml, and so on, way up at the top; there's the neighborhood with Perl, Python, Ruby, and so on, somewhere in the middle; there's the neighborhood with the languages so slow nobody really uses them, way down at the bottom. You're more likely to find you need to make finer distinctions in execution speed up near the top, where performance *really* matters. > > One final note to implementers and language designers ... Python gets an > extra little pat on the back from me for having such a low spread. YARV > and other Ruby implementations need to pay attention to the fact that > their boxes are wider than Perl's and PHP's and a *lot* wider than > Python's. In other words, look at the benchmarks where you really suck > first. :) No kidding. Those of us picking a language for a project might want to check the specifics of where a language really sucks, too, to determine whether that's going to be a problem -- but other than that, as long as your performance needs are simple enough to allow for a dynamic language like Perl et alii, you might as well just pick the one you like the best. At least, that's my take on the matter -- look for problem areas that are deal-breakers, and otherwise don't worry about it too much unless you're writing code that has to fit in eight bits and run like the dickens (for example). Anything else strikes me as a case of "premature optimization", especially considering the difference a good algorithm can make. > > In case you want the numbers that go with the boxplots, here they are: > > gcc yarv python perl php ruby jruby > Low 1 1.2 1.4 0.93 1.4 1.5 3.4 > Q1 1 4.8 4.6 2.90 3.1 6.3 12.0 > Median 1 8.7 14.0 26.00 31.0 34.0 50.0 > Q3 1 68.0 45.0 55.00 55.0 170.0 340.0 > High 1 150.0 98.0 67.00 110.0 380.0 410.0 Thanks muchly for the statistics-wrangling. It's instructive. -- CCD CopyWrite Chad Perrin [ http://ccd.apotheon.org ] Kent Beck: "I always knew that one day Smalltalk would replace Java. I just didn't know it would be called Ruby."