Re: Faster Marshaling?

Eric Hodel wrote:
> On Jul 26, 2008, at 19:58 PM, Greg Willits wrote:
>>  will be a problem (currently it can take several seconds to restore
>> restore as possible -- no need to "reconstruct" data piece by piece
>> which Ruby seems to be doing now. It takes < 250ms to load an 11MB raw
>> data file via readlines, and 2 seconds to load a 9MB sized Marshal  
>> file,

> Ruby is going to need to call allocate for each object in order to
> register with the GC and build the proper object graph.  I doubt
> there's a way around this without extensive modification to ruby.

Hmm. makes sense of course, was just hoping someone had a clever 
replacement.

I'll just have to try clever code that minimizes the frequency of 
re-loads.

If you're curious about the back story, I've explained it more below.


> readlines? not read?  readlines should be used for text, not binary
> data.  Also, supplying an IO to Marshal.load instead of a pre-read
> String adds about 30% overhead for constant calls to getc.

Wasn't using it on binary data -- was just making a note that an 11MB 
tab file (about 45,000 lines) took all of 250ms (actually 90ms on my 
server drives) to read into an array using readlines. Whereas loading a 
marshaled version of that same data (reorganized, and saved as an array 
of hashes) from a file that happened to be 9Mb took almost 2 seconds -- 
so there's clearly a lot of overhead in re-storing a marshaled object. 
That was my point.

> 9MB seems like a lot of data to load, how many objects are in the
> dump?  Do you really need to load a set of objects that large?

Yes, and that's not the largest, but it's an average. Range is 1 MB to 
30 MB of raw data per file. A few are 100+ MB, one is 360 MB on it's 
own, but it's an exception.

This is a data aggregation framework. One generic framework will run as 
multiple app-specific instances where each application has a data set of 
4-8GB of raw text data (from 200-400 files). That raw data is loaded, 
reorganized into standardized structures, and one or more indexes 
generated per original file.

One application instance per server. The server is used as a workgroup 
intranet web server by day (along with it's redundant twin), and as an 
aggregator by night.

That 9MB Marshaled file is the result of one data source of 45,000 lines 
being re-arranged, each data element cleansed and transformed, and then 
stored as an array of hashes. An index is stored as a separate Marshaled 
file so it can be loaded independently.

Those 300 or so original files, having been processed and indexed, are 
now searched and combined in a complex aggregation (sadly, not just 
simple mergers) which nets a couple dozen tab files for LOAD DATA into a 
database for the web app.

Based on a first version of this animal, spikes on faster hardware, 
accounting for new algorithms and growth in data sizes, this process 
will take several hours even on a new intel server even with everything 
loaded into RAM.  And that's before we start to add a number of new 
tasks to the application.

Of course, we're looking at ways to split the processing to take 
advantage of multiple cores, but that just adds more demand on memory 
(DRb way too slow by a couple orders of magnitude to consider using as a 
common "memory" space" for all cores).

The aggregation is complex enough that in a perfect world, I'd have the 
entire data set in RAM all at once, because any one final data table 
pulls it's data from numerous sources and alternate sources if the 
primary doesn't have it on a field by field basis. field1 comes from 
sourceX, field2 from sourceA, and sourceB if A doesn't have it. It gets 
hairy :-)

Unlike a massive web application where any one transaction can take as 
long as 1 second even 2 to complete, and you throw more machines at it 
to handle increase in requests, this is a task trying to get tens of 
millions of field tranformations, and millions of hash reads completed 
linearly as quickly as possible. So, the overhead of DRb and similar 
approaches aren't good enough.


David Masover wrote:
>> - I need to swap data sets often enough that performance
>>    will be a problem (currently it can take several seconds to restore
>>    some marshaled data -- way too long)
> 
> Why do you need to do this yourself?

As a test, I took that one 9MB sample file mentioned above, and loaded 
it as 6 unique objects to see how long that would take, and how much RAM 
would get used -- Ruby ballooned into using 500MB of RAM. In theory I 
would like to have every one of those 300 files in meory, but 
logistically I can easily get away with 50 to 100 at once. But if Ruby 
is going to balloon that massively, I won't even get close to 50 such 
data sets in RAM at once. So, I "need" to be able to swap data sets in & 
out of RAM as needed (hopefully with an algorithm that minimizes the 
swapping by processing batches which all refc the same loaded data 
sets).


>> - the scaling is such that more RAM per box is costly enough to pay for
>>    development of a more RAM efficient design
> 
> What about more swap per box? It might be slower, maybe not, but it seems
> like the easiest thing to try.

More "swap"? You mean virtual memory? I may be wrong, but I am assuming 
regardless of how effective VM is, I can easily saturate real RAM, and 
it's been my experience that systems just don't like all of their real 
RAM full.

Unless there's some Ruby commands to tell it to specifically push 
objects into the OS's VM, I think I am stuck having to manage RAM 
consumption on my own. ??


> Another possibility would be to use something like ActiveRecord -- 

Using the db especially through AR would be glacial. We have a db-based 
process now, and need something faster.

-- gw

-- 
Posted via http://www.ruby-forum.com/.