From: Eric Hodel <drbrain@...7.net>
Date: 2008-07-27T12:23:48+09:00
Subject: Re: Faster Marshaling?

On Jul 26, 2008, at 19:58 PM, Greg Willits wrote:
> Exploring options... wondering if there's anything that can replace
> marshaling that's similar in usage (dump & load to/from disk file),  
> but
> faster than the native implementation in Ruby 1.8.6
>
> I can explain some details if necessary, but in short:
>
> - I need to marshal,
>
> - I need to swap data sets often enough that performance
>   will be a problem (currently it can take several seconds to restore
>   some marshaled data -- way too long)
>
> - the scaling is such that more RAM per box is costly enough to pay  
> for
>   development of a more RAM efficient design
>
> - faster performance Marshaling is worth asking about to see how much
> it'll get me.
>
> I'm hoping there's something that's as close to a memory space dump &
> restore as possible -- no need to "reconstruct" data piece by piece
> which Ruby seems to be doing now. It takes < 250ms to load an 11MB raw
> data file via readlines, and 2 seconds to load a 9MB sized Marshal  
> file,

readlines? not read?  readlines should be used for text, not binary  
data.  Also, supplying an IO to Marshal.load instead of a pre-read  
String adds about 30% overhead for constant calls to getc.

9MB seems like a lot of data to load, how many objects are in the  
dump?  Do you really need to load a set of objects that large?

> so clearly Ruby is busy rebuilding stuff rather than just pumping a  
> RAM
> block with a binary image.

Ruby is going to need to call allocate for each object in order to  
register with the GC and build the proper object graph.  I doubt  
there's a way around this without extensive modification to ruby.