From: Paul Brannan <pbrannan@...>
Date: 2007-11-06T02:02:06+09:00
Subject: Re: hacking on the "heap" implementation in gc.c

On Fri, Nov 02, 2007 at 04:09:53AM +0900, Lloyd Hilaiel wrote:
> What I would love is criticism on the approach, and some performance  
> testing of this patch to see if the gains I'm seeing are artificial.   
> This patch is just a proof of concept and play, many things would  
> need to be cleaned up before it would be something that could be  
> useful (RUBY_CRITICAL being one thing of many).

Interesting work.  Reducing the memory footprint is usually a good thing
and can result in better performance than measured by benchmarks.  I
think your performance tests need a little work, though:

1. Some of these tests don't run long enough to make me comfortable that
   the difference in time isn't due to limitations in timing.  My rule
   of thumb is never to run a timing test for less than 10 seconds
   unless I have good reason.

2. I would like to see user and system time reported in addition to
   wallclock time.

3. I would recommend running each test case multiple times, alternating
   between each GC implementation on each iteration.  This will give
   more data points so a proper statistical comparison can be done
   between the two data sets.

4. Often, interesting performance difference isn't the the mean but the
   worst case.  For example, I'm curious to know why you use more memory
   in ostruct_read_yaml.rb and, if it is a common use case, whether that
   number can be reduced through more tweaking.  I'm also curious why
   classes_read.rb and shrinkarray.rb are slower, and whether that's
   important.

I'm also not able to reproduce the 102% time difference that you've
calculated:

irb(main):059:0> bf = [0.459784, 25.913408, 0.134596, 0.76034, 2.64286, 13.673743,
irb(main):060:1*     0.825455, 0.497352, 9.002692, 0.170999, 1.933119, 0.089326, 1.982008,
irb(main):061:1*       4.926278, 3.076644, 1.205506, 0.056913, 1.932001, 1.142735, 0.0522]
=> [0.459784, 25.913408, 0.134596, 0.76034, 2.64286, 13.673743, 0.825455, 0.497352, 9.002692, 0.170999, 1.933119, 0.089326, 1.982008, 4.926278, 3.076644, 1.205506, 0.056913, 1.932001, 1.142735, 0.0522]
irb(main):062:0> af = af = [0.448108, 26.433873, 0.142204, 0.752775, 2.865675, 14.471611,
irb(main):063:1*     0.84994, 0.512148, 9.168935, 0.17948, 2.116156, 0.092244, 1.908458,
irb(main):064:1*       4.773025, 3.284315, 1.131734, 0.062934, 2.122906, 1.114133, 0.052393]
=> [0.448108, 26.433873, 0.142204, 0.752775, 2.865675, 14.471611, 0.84994, 0.512148, 9.168935, 0.17948, 2.116156, 0.092244, 1.908458, 4.773025, 3.284315, 1.131734, 0.062934, 2.122906, 1.114133, 0.052393]
irb(main):065:0> d = []; af.each_with_index { |x,i| d.push(x / bf[i]) }
=> [0.448108, 26.433873, 0.142204, 0.752775, 2.865675, 14.471611, 0.84994, 0.512148, 9.168935, 0.17948, 2.116156, 0.092244, 1.908458, 4.773025, 3.284315, 1.131734, 0.062934, 2.122906, 1.114133, 0.052393]
irb(main):066:0> d
=> [0.974605466914899, 1.02008477618999, 1.05652471098695, 0.990050503722019, 1.08430828723428, 1.05835037268142, 1.02966242859998, 1.02974955363606, 1.01846592108227, 1.04959678126773, 1.09468480729846, 1.03266686071245, 0.962891168955928, 0.968890712217215, 1.06749919717718, 0.938804120427439, 1.1057930525539, 1.09881206065628, 0.974970574980201, 1.00369731800766]
irb(main):067:0> mean = d.inject { |i,j| j + i } / d.size
=> 1.02800543376512

I think maybe you have rounding error?

Paul