From: Lloyd Hilaiel <lloyd@...>
Date: 2007-11-06T03:15:52+09:00
Subject: Re: hacking on the "heap" implementation in gc.c


On Nov 5, 2007, at 10:02 AM, Paul Brannan wrote:

> On Fri, Nov 02, 2007 at 04:09:53AM +0900, Lloyd Hilaiel wrote:
>> What I would love is criticism on the approach, and some performance
>> testing of this patch to see if the gains I'm seeing are artificial.
>> This patch is just a proof of concept and play, many things would
>> need to be cleaned up before it would be something that could be
>> useful (RUBY_CRITICAL being one thing of many).
>
> Interesting work.  Reducing the memory footprint is usually a good  
> thing
> and can result in better performance than measured by benchmarks.  I
> think your performance tests need a little work, though:
>
> 1. Some of these tests don't run long enough to make me comfortable  
> that
>    the difference in time isn't due to limitations in timing.  My rule
>    of thumb is never to run a timing test for less than 10 seconds
>    unless I have good reason.
>
> 2. I would like to see user and system time reported in addition to
>    wallclock time.
>
> 3. I would recommend running each test case multiple times,  
> alternating
>    between each GC implementation on each iteration.  This will give
>    more data points so a proper statistical comparison can be done
>    between the two data sets.
>
> 4. Often, interesting performance difference isn't the the mean but  
> the
>    worst case.  For example, I'm curious to know why you use more  
> memory
>    in ostruct_read_yaml.rb and, if it is a common use case, whether  
> that
>    number can be reduced through more tweaking.  I'm also curious why
>    classes_read.rb and shrinkarray.rb are slower, and whether that's
>    important.
>
> I'm also not able to reproduce the 102% time difference that you've
> calculated:
>
> irb(main):059:0> bf = [0.459784, 25.913408, 0.134596, 0.76034,  
> 2.64286, 13.673743,
> irb(main):060:1*     0.825455, 0.497352, 9.002692, 0.170999,  
> 1.933119, 0.089326, 1.982008,
> irb(main):061:1*       4.926278, 3.076644, 1.205506, 0.056913,  
> 1.932001, 1.142735, 0.0522]
> => [0.459784, 25.913408, 0.134596, 0.76034, 2.64286, 13.673743,  
> 0.825455, 0.497352, 9.002692, 0.170999, 1.933119, 0.089326,  
> 1.982008, 4.926278, 3.076644, 1.205506, 0.056913, 1.932001,  
> 1.142735, 0.0522]
> irb(main):062:0> af = af = [0.448108, 26.433873, 0.142204,  
> 0.752775, 2.865675, 14.471611,
> irb(main):063:1*     0.84994, 0.512148, 9.168935, 0.17948,  
> 2.116156, 0.092244, 1.908458,
> irb(main):064:1*       4.773025, 3.284315, 1.131734, 0.062934,  
> 2.122906, 1.114133, 0.052393]
> => [0.448108, 26.433873, 0.142204, 0.752775, 2.865675, 14.471611,  
> 0.84994, 0.512148, 9.168935, 0.17948, 2.116156, 0.092244, 1.908458,  
> 4.773025, 3.284315, 1.131734, 0.062934, 2.122906, 1.114133, 0.052393]
> irb(main):065:0> d = []; af.each_with_index { |x,i| d.push(x / bf 
> [i]) }
> => [0.448108, 26.433873, 0.142204, 0.752775, 2.865675, 14.471611,  
> 0.84994, 0.512148, 9.168935, 0.17948, 2.116156, 0.092244, 1.908458,  
> 4.773025, 3.284315, 1.131734, 0.062934, 2.122906, 1.114133, 0.052393]
> irb(main):066:0> d
> => [0.974605466914899, 1.02008477618999, 1.05652471098695,  
> 0.990050503722019, 1.08430828723428, 1.05835037268142,  
> 1.02966242859998, 1.02974955363606, 1.01846592108227,  
> 1.04959678126773, 1.09468480729846, 1.03266686071245,  
> 0.962891168955928, 0.968890712217215, 1.06749919717718,  
> 0.938804120427439, 1.1057930525539, 1.09881206065628,  
> 0.974970574980201, 1.00369731800766]
> irb(main):067:0> mean = d.inject { |i,j| j + i } / d.size
> => 1.02800543376512
>
> I think maybe you have rounding error?
>
> Paul
>

I agree with all of your points.  The testing does indeed lack rigor.
So these stats should be considered preliminary, and all they really  
say is that
  this is a tactic worth further exploration.

I'll look at each one of your points in the coming week.  In addition  
to user/system time,
I'm trying to figure out how to integrate actual heap memory usage in  
addition to
memory allocated to the ruby heap.  A consideration here is something  
portable that
changes ruby as little as possible.

Most of the time on this thing has been figuring out how quantify the  
change,
rather than making it.  It seems like having a robust, built in "ruby  
performance suite"
could perhaps encourage future exploration.

./configure && make && make benchmark

best,
lloyd