From: Eric Wong <normalperson@...>
Date: 2014-01-31T08:34:43+00:00
Subject: [ruby-core:60382] Re: [ruby-trunk - Bug #5985] miniruby skews "make benchmark" results

ko1@atdot.net wrote:
> 
> > +BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPT

> We need to discuss which version we want to compare. static_ruby and
> dynamic (using libruby) ruby show different performance (basically,
> dynamic ruby is slower).
> 
> To avoid such difference, recently I use benchmark/driver.rb directly
> to compare installed rubies.  To do so, I install non-modified trunk
> and modified trunk before benchmark.
>
> What should happen on "make benchmark"?  This rule I wrote is to
> compare installed "ruby 1.8" and built miniruby. it is not fair
> comparison but it is enough to check the rough performance.

I can count 3 options right now for "make benchmark":

a) "make benchmark" becomes a help message to tell people
    to only benchmark with installed Rubies.

b) use LD_PRELOAD for shared Rubies (not portable?)

c) ignore dynamic for benchmark, it will always
   be slower than static (neither is nearly as fast as miniruby)

I think c) is the easiest path right now, static-ruby is closer to
dynamic ruby.  miniruby is too far different and unrealistic.

In all cases, we should probably add a warning if any benchmarked
ruby is dynamic.