From: ptkwt@... (Phil Tomson) Date: 2005-06-13T15:55:31+09:00 Subject: Re: python/ruby benchmark(don't shoot the messenger) In article <9e7db911050612113833d7fd7a@mail.gmail.com>, Austin Ziegler wrote: >On 6/12/05, Steven Jenkins wrote: >> Austin Ziegler wrote: >> What's more amazing is that these fools why buy into this "crap" have >> made fundamental contributions to the theory and practice of digital >> communications, image processing, and numerical analysis, practically >> invented systems engineering, found volcanoes on Io and landslides on >> Venus, communicated with spacecraft beyond the edge of the solar system, >> put three rovers on the surface of Mars, and a whole bunch of other >> stuff that will go into history books. > >Huh. They didn't do it with benchmarks. (And I could very easily point >out that those same people have screwed up massively -- forgetting to >convert between English units and Metric units?) Look closely at the >people who espouse benchmarks. They're mostly marketers or fools who >can't tell the difference. > >There are *real* measures to deal with; they're not benchmarks. They >aren't and never have been. > >For a first person shooter, the real measure is "is the game fun?" The >answer will be different for everyone, but there are some objective >things that will break the "fun" factor for just about everyone. >Frames Per Second. Load Time. These things should be as fast as they >possibly can. Gigaflops never enters the question here. Nor does >specmark or anything else like that. > >For image manipulations, they need to be quick. But not once does the >Ackermann function ever enter the question. No, but as you've said, you need quick response from a game in order for it to be engaging. It comes down to frames per second (as you note). > >Never *once* have I needed to implement an Ackermann function. Not >once. In my entire career. No one is saying that you would need to. The Ackermann benchmark can be a measure of how well your language handles recursion (of course it can be written iteratively as well). Sure it may not be something that you'd ever use, but if someone is perusing the benchmark results and sees that Ruby falls way behind gawk in this particular benchmark, they're likely to draw some conclusions. Now you and I might think that the conclusions they draw are ot fair or accurate, but that doesn't matter. If I've never encountered Ruby before and all I know of it is from the alioth benchmarks, the conclusions I come to will not be positive. >I look at the crap that is on alioth and >there's very little that represents common use. There's some neat >things -- the new DNA transformation ones -- but exactly how many >people will actually be using that in their work? >I won't. Wouldn't have my entire career. Not everyone is doing web development. Some people work in interesting areas like Bioinformatics. It's great that Ruby is finding a niche in web development with Rails, but I really hope that doesn't translate into Ruby only aspiring to be another PHP (ie. being perceived as a language that is only useful for web development - I'm afraid that's already starting to happen). The fact is that it's in various scientific areas where you'll find the most compute-intensive applications (the other area would be gaming), so including DNA transformation benchmarks can be very informative. >Measures that mattered to me >at my last job were "how many bills can I generate in an hour?" At my >current job "what's the average backup speed throughput for this?" > >If we're not getting the performance we need, we fix the damn problem. Good; we can all agree on this ;-) >We don't rely on "benchmarks" -- we rely on real world measurements of >our real problems. Not on pseudo-crap like gigaflops or specmark or >the speed of an airborne swallow. Actually, strike that. The last is >useful. > Again, benchmarks don't tell the whole story, but unfortuneately many will judge Ruby based on these benchmarks - that's just reality. What are you going to do, buy banner ads on the alioth site which read "Don't trust these benchmarks!". I don't think that'll work. Since benchmarks tend to be a uniform measure (we can't all trot out our 'real world' code to be translated into all the popular languages and then tested and timed in each one - and if we did that would just become the next benchmark, wouldn't it?). A good set of benchmarks needs to be easily implemented in all of the languages being compared and they also need to exercise a lot of different features that would show up in real world programs. I'm going to give the alioth folks the benefit of the doubt here and assume that they want to develop a 'good' set of benchmarks that can be used to measure relative performance of various languages. We can either dismiss their efforts with various pejoratives (which will make people outside of our community wonder what we're trying to hide) or help out. Phil