From: Stephen Kellett <snail@...>
Date: 2005-05-17T18:05:31+09:00
Subject: Re: Ruby/Odeum vs. Lucene Performance

In message <1116294726.4747.30.camel@localhost.localdomain>, Zed A. Shaw 
<zedshaw@zedshaw.com> writes
>The first thing is that there's not statistical basis for "1000 times".

There is. The error is smaller. If you don't believe me you need to 
examine why pollsters always ask at least 1000 potential voters their 
opinion. The error rate is +/- 3% with a sample size of approx 1000 
voters. Ask 10 people and predict the election result and your error 
will be much greater than 3%. The pollsters are in it to make money 
predicting outcomes. If they could get away with 5 or 10 samples, they 
would. It would be more profitable. They don't do it that way.

Also, having written timing analysis programs and deliberately written 
in options to allow me to run the test once, 10 times, a million times 
whatever, I notice the more tests you run, the errors from the fast one 
and the slow one get averaged out and you get closer to what the real 
result it.

I disagree with you.

>You actually want to run the test several times in a series of sample
>runs and then determine the common ramp-up time from a cold start.

Well if you want to do that, I hope you cold start includes a reboot of 
the machine. You don't want anything in the cache.

>Otherwise you'll never know if the few times you ran your "1000 times"
>test were just flukes or not.

I think you misunderstand me. I mean you need to run your test 1000 
times, not put you test in a loop for a 1000 times and run it a few 
times. If you are doing that from a cold-start (after boot up) I can see 
why you wouldn't want to do that :-). If you are just doing it from the 
command line, wrap it in a shell script to execute ruby/jvm 1000 times.

The number doesn't have to be 1000, it needs to be something large 
enough to make the error small enough to be discountable. You decided 
what is discountable for your purposes.

>Also, there's solid statistics behind only doing a few samples, but I
>didn't use any of those techniques.  I believe entire industries have
>been founded on papers with only 3 samples. :-)

I think you are mistaken. Back to the pollsters again...
You are voting liberal, he is voting conservative and she is voting 
Labour. So whats the result of the election? :-)

Stephen
(For American readers, replace with Ralf Nader, Republican and 
Democrat).
-- 
Stephen Kellett
Object Media Limited
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting