Re: python/ruby benchmark(don't shoot the messenger)

In article <[e-mail]>,
Austin Ziegler  <[e-mail]> wrote:
>On 6/12/05, Steven Jenkins <[e-mail]> wrote:
>> Austin Ziegler wrote:
>> What's more amazing is that these fools why buy into this "crap" have
>> made fundamental contributions to the theory and practice of digital
>> communications, image processing, and numerical analysis, practically
>> invented systems engineering, found volcanoes on Io and landslides on
>> Venus, communicated with spacecraft beyond the edge of the solar system,
>> put three rovers on the surface of Mars, and a whole bunch of other
>> stuff that will go into history books.
>
>Huh. They didn't do it with benchmarks. (And I could very easily point
>out that those same people have screwed up massively -- forgetting to
>convert between English units and Metric units?) Look closely at the
>people who espouse benchmarks. They're mostly marketers or fools who
>can't tell the difference.
>
>There are *real* measures to deal with; they're not benchmarks. They
>aren't and never have been.
>
>For a first person shooter, the real measure is "is the game fun?" The
>answer will be different for everyone, but there are some objective
>things that will break the "fun" factor for just about everyone.
>Frames Per Second. Load Time. These things should be as fast as they
>possibly can. Gigaflops never enters the question here. Nor does
>specmark or anything else like that.
>
>For image manipulations, they need to be quick. But not once does the
>Ackermann function ever enter the question.

No, but as you've said, you need quick response from a game in order for 
it to be engaging.  It comes down to frames per second (as you note).

>
>Never *once* have I needed to implement an Ackermann function. Not
>once. In my entire career. 

No one is saying that you would need to.  The Ackermann benchmark can be 
a measure of how well your language handles recursion (of course it can 
be written iteratively as well).  Sure it may not be something that you'd 
ever use, but if someone is perusing the benchmark results and sees that 
Ruby falls way behind gawk in this particular benchmark, they're likely 
to draw some conclusions.  Now you and I might think that the conclusions 
they draw are ot fair or accurate, but that doesn't matter.  If I've 
never encountered Ruby before and all I know of it is from the alioth 
benchmarks, the conclusions I come to will not be positive.  

>I look at the crap that is on alioth and
>there's very little that represents common use. There's some neat
>things -- the new DNA transformation ones -- but exactly how many
>people will actually be using that in their work?
>I won't. Wouldn't have my entire career. 

Not everyone is doing web development.  Some people work in interesting 
areas like Bioinformatics.  It's great that Ruby is finding  a niche in web 
development with Rails, but I really hope that doesn't translate into 
Ruby only aspiring to be another PHP (ie. being perceived as a language 
that is only useful for web development - I'm afraid that's already 
starting to happen).

The fact is that it's in various scientific areas where you'll find the 
most compute-intensive applications (the other area would be gaming), so 
including DNA transformation benchmarks can be very informative.

>Measures that mattered to me
>at my last job were "how many bills can I generate in an hour?" At my
>current job "what's the average backup speed throughput for this?"
>
>If we're not getting the performance we need, we fix the damn problem.

Good; we can all agree on this ;-)

>We don't rely on "benchmarks" -- we rely on real world measurements of
>our real problems. Not on pseudo-crap like gigaflops or specmark or
>the speed of an airborne swallow. Actually, strike that. The last is
>useful.
>

Again, benchmarks don't tell the whole story, but unfortuneately many 
will judge Ruby based on these benchmarks - that's just reality.  What 
are you going to do, buy banner ads on the alioth site which read "Don't 
trust these benchmarks!".  I don't think that'll work.

Since benchmarks tend to be a uniform measure (we can't all trot out 
our 'real world' code to be translated into all the popular languages and 
then tested and timed in each one - and if we did that would just become 
the next benchmark, wouldn't it?).  A good set of benchmarks needs to be 
easily implemented in all of the languages being compared and they also 
need to exercise a lot of different features that would show up in real 
world programs.  I'm going to give the alioth folks the benefit of the 
doubt here and assume that they want to develop a 'good' set of 
benchmarks that can be used to measure relative performance of various 
languages.  We can either dismiss their efforts with various 
pejoratives (which will make people outside of our community wonder 
what we're trying to hide) or help out.



Phil