Re: Confirm my Performance Test Against Java?

First, I should say that I'm going to present arguments for Ruby here, whether 
or not I think it's the best choice. Without knowing what you need, I really 
can't say.

On Thursday 20 August 2009 12:25:58 am Ben Christensen wrote:
> Yes, "CPU is cheap", but that applies to Java as well.

But if you are in a position to be able to throw more hardware at the problem, 
it does really become a question of CPU time vs programmer time. That is, if 
Ruby really does cost 3x the CPU of Java, you can calculate in real dollars 
how much it will cost to use.

Speaking for myself, I certainly feel Ruby makes me more than three times as 
productive as Java, and programmer time is much more expensive than CPU time.

Obviously, there are cases where spending programmer time makes sense. For 
example, performance-critical desktop apps (CAD, games, etc) cannot use the 
"throw more CPU at it" argument, because you're now forcing your clients to 
upgrade their hardware to use your product. A large enough organization might 
want to optimize as much as they can -- if rewriting it in C saves a thousand 
machines and takes a developer a year, it's probably worth it, unless you can 
get a thousand machines cheaper than a developer.

On the other hand, there's a case to be made that you should "write one to 
throw away" -- if you can do it in Ruby in a few days, and rewrite it in Java 
in a few weeks, the rewrite can take lessons learned from that ruby sketch.

It would also give you time to evaluate the "fast enough" argument. If it 
turns out that you have plenty of extra capacity, and the Ruby version runs 
fast enough, it may not be worth rewriting. If it turns out that Ruby is too 
slow (even after trying Ruby 1.9 and JRuby), you've only lost a few days.

> Perhaps in a straight-forward CRUD app where all that's being done is
> retrieving/storing data in a database where IO truly is the bottleneck
> that no amount of optimization can improve - then it doesn't matter and
> "good enough" rings true where IO takes 100ms and the Ruby/Java portion
> is only 10-20ms on top of the IO.

That depends...

Response time is only part of the story. What you really want to benchmark is 
requests per second, and that's not always as simple as multiplying response 
time.

> ## C Extensions to Ruby ##
>
> This feels akin to saying in the late 90s that to make Java perform
> well, use JNI to use C. To me it defeats the whole purpose of the "Ruby
> is simple and pleasant" paradigm. If I have to optimize it with C, then
> it's no longer simple or a joy to use.

It's a bit like C -- it's going to be fast enough most of the time, but 
there's always the possibility you'll find some small part that can be 
rewritten in assembly to squeeze some extra performance out of it.

Most of what I said could be summarized as: The speed of a nonworking program 
is irrelevant, and premature optimization is the root of all evil. (I don't 
remember who I'm quoting, but it was someone important.)

My preference would be, if I can write 97% of the program in Ruby, and 3% in 
C, is that really going to be less pleasant than writing 100% of the program 
in Java?

> Nor am I convinced yet that managing a codebase over 5-10 years and 40+
> developers is any easier with Ruby - Java's static typing and now its
> generics (the polar oppostite approach of 'duck typing' in Ruby) are
> actually very nice for readability, navigation of code, refactoring and
> other such things on such large codebases when so much of it is from
> other coders, teams, or just plain old and forgotten about.

I suspect most of that is due to the tools, more than the type system itself.

The most compelling argument I've heard is: Type checks are just a special 
case of unit tests. You need unit tests anyway, and good unit tests will 
already more than cover what type checks were meant to save you from.

I suppose I'm curious -- when was the last time the type system saved you? 
That is, when was the last time you tried to pass an object of the wrong type 
to a method, and gotten a type error of some sort, and realized you needed to 
do something other than a simple typecast on that object?

> Also, no amount of "throwing hardware at it" will make Ruby faster than
> throwing the same hardware at Java - which is ultimately I think the
> biggest issue I have with the concept.

Indeed -- but again, you're paying for it with increased developer time.

And throwing the same hardware at Java that you would need for Ruby just gives 
you a bunch of unused capacity -- you'd be buying less hardware. So it is 
pretty much a straight trade.

> If I throw the fastest hardware I
> can at something, I want my user to get the biggest bang for the buck -

Which is pretty much going to give you benchmark candy. If your site is 
slowing down, that's a bug. Once the speed of the site is acceptable, and 
you're set up to handle spikes appropriately, more performance doesn't really 
buy you anything other than "because you can".

> As for "brevity" equalling "maintainability" and "less bugs" - I tend to
> disagree when the pursuit results in code such as this example given:
>
>     puts "The number of tokens is: %d." % File.open(ARGV[0]){|f|
>     f.inject(0){|a,l| a+l.split.length } } ,
>       "It took #{(Time.now - start) * 1000} ms"

Probably true for that -- though, to be fair, I wouldn't have put it that way.

However, there was a study done, at one point, which showed that the ratio of 
bugs to lines of code was constant across languages. So while I wouldn't say 
it makes sense to make things unreadably brief, Ruby is usually both more 
readable and shorter. 100 lines of code is generally easier to read and debug 
than a thousand.

> Perhaps your experiences are different - but most development teams have
> a lot of more junior and intermediate developers than senior - and the
> more verbose, easy to read, simple to understand code is far more
> preferable - even for myself when I must review, debug and profile the
> code of dozens of others - as opposed to something that looks like an
> academic puzzle to unravel.

I think that particular code sample was misleading -- certainly, you can play 
Perl Golf in any language. But you have coding conventions in Java, and you 
would in Ruby.

> What types of applications and codebases do you feel truly are served
> best by Ruby - and therefore not in need of the highest performance
> given to the end-user?

I feel the kind that is served best is any sort of web app, or small scripts 
for system administration -- particularly anything that needs to be flexible 
and constantly maintained and improved, for which the developer controls the 
hardware.

Were there better deployment tools (and Shoes seems to be an effort in that 
direction), I'd also say any sort of desktop app that doesn't need the highest 
performance possible. Frankly, that's most of them -- an instant messaging 
client, for instance, doesn't need to be blazingly fast, just fast enough.

But, it's not always about whether the end-user needs the highest performance. 
It's about whether it's possible to throw CPUs at the problem, or whether the 
CPU is the bottleneck at all.