From: Alexandru Popescu Date: 2005-06-12T18:34:07+09:00 Subject: Re: python/ruby benchmark(don't shoot the messenger) #: the mind was *winged* after Phil Tomson said on 6/12/2005 9:20 AM :# > In article <9e7db91105061106485b68d629@mail.gmail.com>, > Austin Ziegler wrote: >>On 6/11/05, Alexandru Popescu wrote: >>> I've run through this thread and i am finally writting this >>> because i am in a way astonished. >>>=20 >>> Afaik Python uses bytecodes while Ruby is completely >>> interpretative, so imo there is not possible to drive conclusions >>> from this benchmark. >> >>Which is one of the many reasons that I have a problem with the >>alioth shootout. Ultimately, it tries to treat compiled, semi- >>compiled, and interpreted languages equally. They're not. Compiled >>languages will be faster than semi-compiled (bytecode), which will >>be faster than interpreted. The level of speed difference is a >>matter of scale, and that scale also matters on development time and >>clarity. > > I have to say that I tend to agree with the astonishment. > > Here's what I mean: > Someone comes along and asks about python vs. Ruby performance and > references the alioth benchmarks. Several folks jump on the alioth > benchmarks and call them bogus. I see this as blaming the messanger. > > The fact is that these benchmarks exist and they cover lots of different > types of algorithms. They're on the web and people will look at them - > no way to stop that from happening. > > Someone mentioned the Ackermann benchmark so I had a look and found that > gawk did much better than Ruby - that's pathetic folks. Now, I know that > if you could do this in gawk it should be possible to do it in Ruby as well. > There aren't even mumbers for N=8 or N=9 for the Ruby version. Maybe it's > because the Ruby version was written poorly. Maybe it's because of some > issue within the Ruby interpreter itself - I don't know, I didn't really > take time to look into it just now. Either way, wouldn't it be valuable > to try to improve our score on that benchmark? If it's because it's > poorly written, then it's gravy. If it's an issue with the interpreter > then maybe it would be good to know about the issue for the YARV design. > > Now sure you can say that the benchmarks are bogus (and who knows, maybe > they are) and then take all your marbles and go home, but the benchmarks > remain and other languages communities are trying to hone their numbers. > If we totally sit it out, well, that's not going to look too good. > > Sure these benchmarks don't tell the whole story about how nice it is to > develop in Ruby vs. gawk, but performance _is_ an issue for some people > and they will use this set of benchmarks (or another) as a factor in > deciding which langauge to use. If Ruby consistently shows up down > towards the bottom of the list... well, it can give a false impression. > > If we tell them that they really shouldn't worry about performance (and > who knows, maybe we would be right in doing so) it just sounds like so > much excuse-making. > > I think that as a community we should be trying to compete in these > benchmarks just like other langauge communities are doing. Perhaps the > results can help us as we do move to a VM - if nothing else to show > that we're making improvements (look at them as a set of unit tests for > performance). We know that Ruby needs help in the > performance area. If we keep telling ourselves that Ruby is 'fast > enough' for our application (and it may well be) are we going to be > sitting still while other languages improve performance? Have we been > the hare for the last few years sitting around while the tortise (TCL, > for example, once considered a lot slower than Ruby) is passing us up > (by creating or improving their VMs, for example). > > Of course we know that we can do all sorts of things to speed things up > by writing C extension code, for example - that's a given, and the fact > that it's so easy to do in Ruby makes it possible to get into a lot of > areas where you might normally need to consider a compiled langauge, > however, a lot of people out there want to see how Ruby performs when > you just write Ruby code (C is a compiled language, of course). In some > sense marrying Ruby and C might seem like cheating to them, I suppose (try > to see it from their perspective). > > If it looks like some of the benchmarks are misleading or stacked against > us, maybe we can propose other benchmarks where Ruby shines (BTW: why > does it say that the Object Methods benchmark is being deprecated?), > after all, it appears to be an open source process. > > Bottom line: Ruby needs help in the performance area. Let's admit that > and work on improving in that area instead of shooting the messenger - > it'll make us look like whiners if we keep doing the latter. I'd really > rather not have the Ruby community perceived that way from the outside. > > In the meantime, perhaps we could have some Ruby quizzes that focus on > choosing a benchmark and optimizing it. I think we would be > better off as a community if we participate in the process rather than > boycott it. > > ....just my 2cents. > > Phil > > Lots of thanks Phil for better expressing my thoughts and feelings. That is exactly what I wanted to say - unfortunately the feelings just cut off my vocabulary. And just as a confirmation of my last paragraph: [stupid-quoting-myself] I know, you may say: `maybe Ruby is not for you� (and maybe, I can agree with this). I must sadly confess that I haven't got the same impression while looking at Python (and from the sizes of communities I can deduce many others haven't felt the same). [/stupid-quoting-myself] I have received offline messages suggesting to go with other solutions and some solid arguments why I should do it. I have to agree that it is no pleasure to find your invention `hammered� by a (possible wrong) benchmark; but I think when this happens I would go out shouting my benchmarks where i am kicking ass. I remember that Java community did this a lot while the first c++ vs java benchmarks have been published. I remember that a few years ago when such a benchmark (driven by `paid� money by MS), was proving that .NET was surclassing J2EE, finally even IBM joined the war and proved it completely wrong. :alex |.::the_mindstorm::.|