From: Isaac Gouy <igouy@...>
Date: 2006-08-11T06:20:15+09:00
Subject: Re: Been there, done that - language shootout


Caleb Clausen wrote:
> On 8/10/06, Isaac Gouy <igouy@yahoo.com> wrote:
> > Caleb Clausen wrote:
> > > (Provided we undestand that this benchmark is testing the performance
> > > of an arbitrary algorithm, not the performance that a real program
> > > using random numbers in a given language would experience.)
> >
> > Maybe you could explain a little more - if someone was using one of
> > those programs to generate Fasta files for program testing and
> > algorithm development in their lab, would that make it a real program
> > rather than an arbitrary algorithm? What do you say the difference is
> > between a real program and an arbitrary algorithm?
>
> My understanding of the goal of the fasta program(s) (aside from being
> a benchmark) is to generate a random dna sequence, possibly with some
> biases about which symbols are more likely. If I were really using
> such a program in my work, and speed in that program was important to
> me, (and I was writing it in Ruby,) I would not use the version you
> give on your website. I would make use of Ruby's already existing
> random number generator. (Among other things, ruby uses (AFAIK) the
> well-regarded Mersenne twister algorithm, whereas you have specified a
> linear congruential generator, which is considered inadequate for
> hard-core random number applications.)

I don't think you gave a direct answer to - What do you say the
difference is between a real program and an arbitrary algorithm? - so
please correct what I get wrong.

You're saying if this was a "real program" it would use whichever
library function the language provides for generating random numbers -
if that's all there is to the distinction, when there are no
appropriate library functions are "real programs" and "arbitrary
algorithms" the same thing?


> For benchmarking purposes, what you have already is just fine,
> provided you understand the limitations. I understand the need to have
> all the languages implement the same algorithm so that you can have an
> apple-to-apples comparison. I can also sympathise with your desire to
> keep the testing simpler by requiring a specific output. It's just
> that this isn't the way a serious programmer would actually use random
> numbers in Ruby (and probably most other languages).
>
> What you are really testing is not the speed at which ruby generates
> random numbers, as they would be used in the real world, but the speed
> at which it can execute the linear congruential algorithm, which
> ultimately depends on the speed at which it can execute integer math
> operations.

We're not testing the speed at which Ruby generates random numbers,
period. (Are those 6 word summary descriptions that confusing?)

We're testing the speed at which it can execute a  linear congruential
algorithm, do binary-chop on a tiny array, slice and build strings, and
print strings in FASTA format. (Replace the linear congruential
algorithm by calls to Ruby rand and the time will be reduced to ~400s)