From: Robert Klemme <shortcutter@...>
Date: 2007-10-17T18:47:44+09:00
Subject: Re: Struct is slow

2007/10/17, Joel VanderWerf <vjoel@path.berkeley.edu>:
> Sylvain Joyeux wrote:
> >> I'm not sure whether you are familiar with Benchmark#bmbm which does a
> >> rehearsal - personally I rather not switch off GC since in realistic
> >> situations GC time belongs into the mix.  But results are rather
> >> similar:
> > Sure. GC does belong to the mix. Now, if GC is enabled you are not able to
> > compare anything. Let's assume that GC runs during 'array init', you will
> > say 'hey, struct init is faster'. Now, if GC runs during 'struct init' the
> > result may change ...

But you have a *lot* invocations and it's highly unlikely that GC runs
during every init. On the other hand, if you allocate a lot of memory
from OS that may slow things down. And this will happen without GC.

Also, IMHO cost of memory GC overhead is also part of the runtime
performance.  If you compare two approaches and one allocates a lot
more memory than the other, then GC times belong in the measurement
because in a real application that approach will also lead to
increased GC overhead.

> > Keeping GC is meaningful when benchmarking a whole application. In
> > microbenchmarks like these, it is simply noise.
>
> For fairness, you could do it this way (assuming REP is large enough):
>
> ...
> Benchmark.bmbm 15 do |x|
>    GC.start
>    x.report 'struct init' do
>      REP.times { Foo.new data, data }
>      GC.start
>    end
>
>    GC.start
>    x.report 'array init' do
>      REP.times { [data, data] }
>      GC.start
>    end
> ...

GC.start is not guaranteed to actually run GC AFAIK.  I'd probably rather do

Benchmark.bmbm do
  GC.stop
  # tests
  GC.start
end

There is a chance that GC.start will clean up all the memory allocated
during the first run.

Kind regards

robert