From: Sean Russell <ser@...>
Date: 2002-05-06T00:52:53+09:00
Subject: Re: Ruby vs. Java vs. Native trivia

Stefan Matthias Aust wrote:

> I'm really surprised by your test results.  For every benchmark I did on
> Java and native code compilation, the native code was often close to the

Well, I'm not really trying to run benchmarks here.  As I mentioned in the 
posting, I was poking around with GCJ, and purposefully chose a small app 
to reduce the number of variables until I get comfortable with it.

That given, small utility apps like this are rather important to me, so in a 
way this /is/ a useful benchmark for /me/.  I consider it a "real world" 
benchmark, rather than an empirical comparison.

> performance of the 1.4 server VM but seldom faster.  The startup
> performance wasn't that different.  

What are you running on?  Java on my 450MHz AMD Linux machine (512Mb RAM) 
always has a significant start-up time.  I've been told that OSX has a very 
intelligent VM, in which only one instance of the VM is ever running on any 
given machine, so the VM loading and initialization only happens for the 
first app you run.  This is, sadly, not the case for the Linux VM.

> sophisticated native code compiler.  The last time when I tried GCJ 2.x,
> it generated very poor code and was suffering from an all-too-simple

Hm.  I'm not sure garbage collection entered into this app much :-)

I'm using GCJ 3.0.4, and the startup times are significantly faster than the 
JVM.

>  time ruby -e 'require "cgi"; exit 0'

gives me:

ruby -e 'require "cgi"; exit 0'  0.14s user 0.05s system 79% cpu 0.240 total

> Without the require statement, which you might consider unfair, the time
> is reduced to ~0.10s.

Wow.  On my machine, this is 0.065s, averaged out of 7 runs.  It varies 
widely between 0.1s and 0.02s.

While we're on the topic of trivial, useless, and probably totally bogus 
benchmarks:

  ruby -e 't=Time.now;100.times{`ruby -v`};puts Time.now-t'

gives me a pretty steady 3.75s.

  ruby -e 't=Time.now;10.times{`java -version 2>&1`};puts Time.now-t'

gives me about 11s.  Notice that this only ran 10 times, so the JVM is about 
29 times slower than the Ruby VM (interpreter), just initializing.  Running 
the same benchmark with the -server option to the JVM makes it take even 
longer; about 17.5 seconds.

> A simple java program, which does nothing but System.exit(0) takes - I
> always measure the average between 3 calls after doing one call to warm up
> the system - 0.36s to run.

I'm /really/ interested to hear which OS you're running this on.

> What kind of program did you benchmark?  Which where the hot spots in
> code?> Did you profile your code?  What's the memory consumption?  How
> many time does it spend in garbage collection?

The program is very small.  It allocates four or five objects, does some 
floating point math, and prints out the results.  As I said, it takes the 
current time and date, converts it to base-10, does some string formatting, 
and prints it out.  I haven't done any profiling on the application at all; 
the JVM always consumes about 24Mb of memory, no matter how trivial the 
application -- the application itself uses minimal memory, and I doubt that 
any of the versions hit garbage collection at all, unless upon exit.

> It's interesting to see Ruby winning the benchmark, but I'm afraid, that's
> not a typical result.

Well, it doesn't.  If you looked at the normalized numbers (based on CPU 
usage), Ruby lags, but just barely, behind the GCJ version.

As I play with GCJ some more, I'll try some more complex applications.  I 
tend to use Java for GUI apps and Ruby for non-GUI apps these days, so the 
problem domains don't intersect.

As far as larger applications go, I have some benchmarks on my REXML page.  
These are TOTALLY unfair, since the code being compared is COMPLETELY 
different, but I have numbers for various XML processing run times for 
REXML (which is pure Ruby) and EXML (which is pure Java).  EXML is several 
orders of magnitude faster.  EXML does byte parsing of XML, whereas REXML 
does Regexp parsing, but I think the biggest difference is that the JVMs 
are just faster once they get loaded up.

--- SER