From: Ben Christensen <benjchristensen@...>
Date: 2009-08-20T14:16:31+09:00
Subject: Re: Confirm my Performance Test Against Java?


@Matthew K. Williams

-- 1. How *often* are you going to be processing these files?  If they 
are
-- batch style jobs, then does absolute speed matter over 
maintainability?

The particular application I'm looking at in the future has a virtually 
continuous feed of incoming data from multiple concurrent sources.

Thus I'm looking at what language the processing code would be in. My 
default go to is Java - but I want to consider Ruby and not blindly just 
use what I'm accustomed to before establishing what will likely be in 
existence for the next 3-5 years.

In an existing system doing similar data processing, it is indeed a 
batch process - but one that preferably didn't exist - thus the concept 
of potentially doubling the time isn't appealing - as it's already a 
thorn in the side of operations at which hardware is thrown to 
alleviate.

In another system we horizontally cluster and shard data processing as 
much as possible to parallelize the effort - and do as much as we can to 
optimize performance. For example, daily jobs are required, but the 
volume of data progressed to where the old system was taking days to 
process a single job - hence the new system which now handles a job in 
4-6 hours - and we're looking at other ways of reducing that further but 
so far their cost exceeds business value for now.


-- 2. Are there any reasons to not keep the data in a database and then
-- perform queries, etc.?

SQL is far slower at handling this type of processing in most cases with 
large volumes of data where the incremental inefficiencies of things 
like REGEX and SQL really add up over 10s of millions of executions.

I have recently dealt with a large database (100+ GB) where to achieve 
the necessary performance thresholds we finally had to revert to the use 
of C to write UDFs in MySQL that could process the data efficiently 
without needing to pull the data out of the database, process in Java 
then re-insert, and therefore create huge IO burdens. It was an order of 
magnitude or two faster using this approach rather than straight SQL 
and/or pulling the data out to process externally.

This is a rare thing - this project was the first time I've ever had to 
do that due to very unique needs of the project.

Generally however I have Java in asynchronous processes doing the data 
processing and manipulation.

The analysis of Ruby performance doing these types of jobs was intended 
to find what cost the adoption of Ruby would incur.

It appears that Ruby is not well suited to data processing type 
applications from what I've seen and heard so far.

In another simple test I did where I was iterating over a large amount 
of data, I was shocked at how poorly the Ruby implementation did. It 
seems the looping itself was a very inefficient action in the Ruby 
interpreter.

Hopefully this helps provide some context to my questions about Ruby in 
regards to batch process of data.
-- 
Posted via http://www.ruby-forum.com/.