From: "M. Edward (Ed) Borasky" Date: 2007-10-07T07:50:50+09:00 Subject: Re: Ruby vs. PHP Chad Perrin wrote: > No, I'm not assuming that. I'm pointing out that objections to using > those micro-benchmarks to choose a language to use for general purpose > programming are common, and that I find them agreeable. Micro-benchmarks > are *very* useful for determining problem areas in execution performance, > if you happen to be on the core implementation maitenance team, but > that's not what I was talking about when I said that as an overall > performance measure for purposes of choosing a language they aren't so > useful. Well ... OK ... you agree with me that they are useful for core implementation. And I *still* think they're useful when comparing performance of languages for general purpose programming, provided the benchmark suite is comprehensive and the statistical comparison methodology is correct. I further think that the Alioth Shootout Suite is comprehensive enough to distinguish between the performance of those languages for which *most* of the benchmarks actually ran. The trick is the statistical methodology you use. Ideally, of course, you'd have a benchmark suite that matches the application set. *That's* where people get into trouble with benchmark suites -- they assume that a given suite, for example, the SPEC integer suite, is representative of their application. Now I could look at the benchmarks themselves and throw out those that were obviously orthogonal to web servers when comparing languages for implementing web servers. But I don't think I need to go that far -- it's easier to just throw out the statistical outliers and come up with a set of numbers that's a general characterization of the relative performance of the languages. That's why I asked for the raw data (which I received, by the way). > What I suggested is that "my interpreter is faster than > yours" competition based on micro-benchmarks is pointless for languages > in the same general neighborhood of execution performance, and that > usually other concerns are *far* more important within a given > neighborhood of performance than execution speed. If you want faster > execution speed, you should be in a completely different performance > neighborhood, not quibbling over whether Ruby or Python is faster for an > arbitrary set of micro-benchmarks, in other words. Of course. I'm still in the process of crunching the numbers, but what I've seen so far is that MRI Ruby is *not* in the general neighborhood of Perl, Python or PHP, but YARV *is* in that neighborhood. I do need to tighten up the calculations slightly and get rid of some bogus points, but I'm not expecting a major change in that. As for things in the "fast lane", I started with gcc, and there are only a few things faster than gcc. The one that jumped out and bit me on the nose was a *functional* language called "clean". I'm just about to download it and see what makes it tick (or to stretch the analogy, how tightly it's wound). And the slow lane has such snails as "groovy" and "rebol" lumbering away. > Yes, we can -- but the margin for error is so great that, again, petty > microsecond disputes between Perl, Python, and Ruby are just that: petty. > I say this despite the fact that, overall, Perl kicks the tuckus of both > these other close competitors -- and Perl is one of my favorite > languages. I would have said roughly the same thing even before I > started using Ruby. Well ... let me finish with the numbers, but I think you'll be surprised at how close Perl and Python are to each other and how far away MRI is from the two of them. It's not a "petty microsecond dispute" for MRI. > Who said anything about "significantly faster"? I'm just talking about > the fact that Perl consistently kicking Ruby's tail in performance > benchmarks doesn't mean a whole lot when choosing languages for projects, > generally speaking, since if performance is enough of a concern to > override the other factors involved you should be using something like C, > Objective Caml, or at least compiled Lisp instead. Of course. But if the choice is between "dynamic languages with a sufficient supply of competent developers available", for example, Perl, Python, PHP and Ruby, and the assumption is that they are equally productive -- you're just trying to assess how much hardware you're going to have to throw at it if your customer growth takes off -- you're going to want the fastest one. Or, more to the point, you're probably going to dismiss the slowest one and move on to the other decision factors.