[ruby-core:59889] [ruby-trunk - Bug #5985] miniruby skews "make benchmark" results

Issue #5985 has been updated by Eric Wong.


 Btw, I'm still getting more consistent results with normal ruby instead
 of miniruby, probably because the encodings still affect heap size and
 memory layout.
 
 I suggest defining BENCHRUBY in my patch:
 http://bogomips.org/ruby.git/patch?id=1ec0da52b7ce2e0e16f17
 
 The following changes since commit 0558b8a5c40b93c01f5724fe8a3ab409d4374490:
 
   *  ext/psych/lib/psych.rb: fix typo by @jwworth [fix GH-500] *  lib/rake/file_list.rb: ditto (2014-01-19 05:48:33 +0000)
 
 are available in the git repository at:
 
   git://80x24.org/ruby.git benchmark
 
 for you to fetch changes up to 1ec0da52b7ce2e0e16f1729933ea795b12fea7ae:
 
   common.mk: define BENCHRUBY to avoid miniruby (2014-01-19 07:02:16 +0000)
 
 ----------------------------------------------------------------
 Eric Wong (1):
       common.mk: define BENCHRUBY to avoid miniruby
 
  common.mk | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)

----------------------------------------
Bug #5985: miniruby skews "make benchmark" results
https://bugs.ruby-lang.org/issues/5985#change-44442

* Author: Eric Wong
* Status: Closed
* Priority: Low
* Assignee: Koichi Sasada
* Category: build
* Target version: 2.0.0
* ruby -v: -
* Backport: 
----------------------------------------
miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v \
	            --executables="ruby; ./miniruby -I../lib -I. -I.ext/common  ../tool/runruby.rb --extout=.ext  -- --disable-gems" \
	            --pattern=vm3_gc --directory=../benchmark 
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]

-----------------------------------------------------------
vm3_gc

#! /usr/bin/ruby
5000.times do
  100.times do
    {"xxxx"=>"yyyy"}
  end
  GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	.2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	.1.857623815536499

-----------------------------------------------------------
raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)
-----------------------------------------------------------
benchmark results:
name	ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]	average difference
vm3_gc	2.752	1.858	-0.895
-----------------------------------------------------------
average total difference is -0.894646406173706




-- 
http://bugs.ruby-lang.org/