From: Daniel Schierbeck <daniel.schierbeck@...>
Date: 2007-03-09T11:26:13+09:00
Subject: Re: A better benchmarking syntax (was: Automatic Benchmark	Iterations)

On Fri, 2007-03-09 at 03:30 +0900, pat eyler wrote:
> > Unfortunately, I do not have great experience with benchmarking, neither
> > the underlying math (well, I have some), nor the actual need of the
> > people who would use an extended framework.
> 
> The math isn't my strongpoint either (on the statistical side), but I'm happy
> to kibbitz (and to try to engage other folks who can help) if you'd like.

I would surely need help of statistics gurus to get it right, so please
do. The project itself might not begin immediately though (I've got
exams three weeks from now :x)

> > The only thing I have ever used benchmarking for was to compare the
> > speed of two methods with the same functionality (e.g. sorting).
> >
> > So what I initially want to ask you guys is this: what else do you use
> > benchmarking for? Do you benchmark your entire application, and compare
> > results from different revisions? I'd really like to know.
> 
> well, the JRuby/rubinius/etc folks could sure use a common benchmarking
> suite to look at sets of results from different builds (and different
> platforms).
> I do some revision v revision benchmarking of apps as well.

I thought that was what had the most potential. I might take a look
around to see if there are any existing solutions (there probably are
for Rails) and if I could extract some code into a library. I'll
probably have to write a lot of things from the ground up, though.

> > Second, if there are others who would be interested in contributing to
> > any project, given that the need is real, I would be delighted to hear
> > from them!
> >
> 
> I'll weigh in some more as scope and plans become more clear (and after
> I get the other eleventy-six things off my plate).

I know the feeling ;)


Cheers,
Daniel