From: Eric Wong <normalperson@...>
Date: 2014-01-14T22:37:45+00:00
Subject: [ruby-core:59762] Re: [ruby-trunk - Bug #5985] miniruby skews "make benchmark" results

Btw, I'm still having this problem and it's very noticeable with:
    make benchmark-each ITEM=bm_so_reverse_complement

This is running the same COMPARE_RUBY as what was just installed via
"make install"   Strangely, miniruby is the one that is slow, here.

    $ make benchmark-each ITEM=bm_so_reverse_complement
    ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]   1.530159944
    built-ruby      2.93252692

    -----------------------------------------------------------
    raw data:

    [["so_reverse_complement", [[1.530159944], [2.93252692]]]]

    Elapsed time: 4.464871542 (sec)
    -----------------------------------------------------------
    benchmark results:
    Execution time (sec)
    name    ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]
    built-ruby
    so_reverse_complement   1.530   2.933

    Speedup ratio: compare with the result of `ruby 2.2.0dev (2014-01-14
    trunk 44595) [x86_64-linux]' (greater is better)
    name    built-ruby
    so_reverse_complement   0.522

Compare with the following, the numbers only differ in noise:

    $ ruby --disable=gems ./benchmark/driver.rb -v  --executables="a::ruby \
    --disable=gems; b::ruby --disable=gems" \
    --pattern=bm_so_reverse_complement --directory=./benchmark

    a       1.547410982
    b       1.537930862

    -----------------------------------------------------------
    raw data:

    [["so_reverse_complement", [[1.547410982], [1.537930862]]]]

    Elapsed time: 3.087557723 (sec)
    -----------------------------------------------------------
    benchmark results:
    Execution time (sec)
    name    a       b
    so_reverse_complement   1.547   1.538

    Speedup ratio: compare with the result of `a' (greater is better)
    name    b
    so_reverse_complement   1.006