RE: Ruby performance on the Language Shootout

Hello,

> -----Original Message-----
> From: Mike Hall [mailto:[e-mail]]
> Sent: Wednesday, February 13, 2002 8:35 PM
> To: ruby-talk ML; undisclosed-recipients:
> Subject: Re: Ruby performance on the Language Shootout
...
> Also, there's a semi-shoot-out going on over in the Tcl newsgroup.
> Actually, they're seeing how Tcl fares in a Python-Java shootout.
>
> > Jean-Luc Fontaine wrote:
> > > Look at:
> > >    http://www.twistedmatrix.com/users/glyph/rant/python-vs-java.html
> > > Maybe we could try those with Tcl?
>
> Perhaps some seasoned and adventurous Rubyists could see how Ruby
> compares?
>

I don't know if I qualify as a seasoned Rubyist, but given the simplicity
of the code involved in this comparison, I took the chance to write the
ruby version plus  a 'gbench.rb' script that performs the benchmarks
in a flexible way, so that one can add scripts or languages easily.

Some notes about the test runner:
  - benchmarks are given on the command line using the test's name
    (i.e. "gbench.rb speed list object"); if none are given,
    all those found in the current directory are used

  - To make echo to another thread, I've used the 'respond_to?' in
    a way that I found quite appropriate.
    So for those who wanted an example...

  - I used some Unixisms (" > /dev/null", 'Open3.popen3'), so the
    script is not portable on Windows (ruby-cygwin *might* work)

  - well, enough said about this little benchmark, back to work :)
    However, as you can already see on the original page linked
    in the quote, this benchmark is quite sparse, and there could
    be many more things tested in this way: regexp, maths,
    networking, exception, eval, introspection, etc.


Download it on my Ruby page:

 http://www.bct-portal.com/opensource/ruby/gbench.tgz


Some notes about the results I've found:

  - Clearly, java stands better now than in the original paper.
    (Ok, Glyph did use jdk1.1.7, I'm using IBM's 1.3.0).
    But some criticisms still hold:
    - java has a very long startup time (see the no_test and the speed_test)
    - java has problems handling the console output (console_test),
      but this is unrelated to streams, where it performs well
    For the rest, java is consistently the fastest language. We wouldn't
    expect less from a language with a JIT compiler...
    This remark made me think about disabling the jitc, see the results
    below and try to not laugh :)
    Ok, folks, do you really think it's worth putting a lot of
    work in getting a byte compiler for Ruby?
    I would say that the current model is quite smart and efficient.

  - Python2 seems a little faster than Python1.5 (except for I/O).
    Using '-O' (basic optimizations) consistently gives you faster
    results.
    Good!

  - Ruby 1.6.5 competes nicely with them: only slower for ONE test!
    (test_hash). Should there be some enhancements for this one?
    It is significantly faster than Python for I/O, and has
    a very fast startup.
    I would like to see the results for Ruby 1.7!


Example output (on a PIII-600 with Linux 2.2.16):
--------------------------------------------------------------------------

Benchmarking the following languages...
python:         1.5.2
python/opt:     1.5.2 (with -O)
python2:        2.2
python2/opt:    2.2 (with -O)
ruby:   1.6.5
java:   1.3.0
java (without JITC):    1.3.0
------
Setup...
... for java:
  compiling IoTest.java
  compiling ListTest.java
  compiling NativeTest.java
  compiling NoTest.java
  compiling ObjectTest.java
  compiling SpeedTest.java
... for java (without JITC):
------
Benchmarking console_test...
        python does it in       0m9.700s
        python/opt does it in   0m9.450s
        python2 does it in      0m10.620s
        python2/opt does it in  0m10.480s
        ruby does it in         0m6.830s
        java does it in         0m14.750s
        java (without JITC) does it in  1m24.580s

Benchmarking hash_test...
        python does it in       0m3.010s
        python/opt does it in   0m2.750s
        python2 does it in      0m2.900s
        python2/opt does it in  0m2.610s
        ruby does it in         0m3.440s
        java does it in         0m1.590s
        java (without JITC) does it in  0m8.650s

Benchmarking io_test...
        python does it in       0m13.830s
        python/opt does it in   0m13.460s
        python2 does it in      0m15.490s
        python2/opt does it in  0m15.570s
        ruby does it in         0m6.260s
        java does it in         0m3.850s
        java (without JITC) does it in  0m27.520s

Benchmarking list_test...
        python does it in       0m4.710s
        python/opt does it in   0m4.540s
        python2 does it in      0m4.160s
        python2/opt does it in  0m3.990s
        ruby does it in         0m2.370s
        java does it in         0m1.050s
        java (without JITC) does it in  0m3.520s

Benchmarking native_test...
        python2 does it in      0m2.700s
        python2/opt does it in  0m2.610s
        ruby does it in         0m2.020s
        java does it in         0m1.550s
        java (without JITC) does it in  0m1.870s

Benchmarking no_test...
        python does it in       0m0.010s
        python/opt does it in   0m0.020s
        python2 does it in      0m0.020s
        python2/opt does it in  0m0.020s
        ruby does it in         0m0.010s
        java does it in         0m0.350s
        java (without JITC) does it in  0m0.210s

Benchmarking object_test...
        python does it in       0m8.800s
        python/opt does it in   0m8.360s
        python2 does it in      0m7.760s
        python2/opt does it in  0m7.430s
        ruby does it in         0m6.910s
        java does it in         0m0.560s
        java (without JITC) does it in  0m1.220s

Benchmarking speed_test...
        python does it in       0m0.960s
        python/opt does it in   0m0.790s
        python2 does it in      0m0.850s
        python2/opt does it in  0m0.720s
        ruby does it in         0m0.610s
        java does it in         0m0.340s
        java (without JITC) does it in  0m0.350s

------