From: David Masover <ninja@...>
Date: 2008-07-27T14:17:33+09:00
Subject: Re: Faster Marshaling?

On Sunday 27 July 2008 00:07:10 Greg Willits wrote:

> >> - the scaling is such that more RAM per box is costly enough to pay for
> >>    development of a more RAM efficient design
> > 
> > What about more swap per box? It might be slower, maybe not, but it seems
> > like the easiest thing to try.
> 
> More "swap"? You mean virtual memory? I may be wrong, but I am assuming 
> regardless of how effective VM is, I can easily saturate real RAM, and 
> it's been my experience that systems just don't like all of their real 
> RAM full.

In general, yes. However, if this is all the system it's doing, I'm suggesting 
that it may be useful -- assuming there isn't something else that makes this 
impractical, like garbage collection pulling everything out of RAM to see if 
it can be collected. (I don't know enough about how Ruby garbage collection 
works to know if this is a problem.)

But then, given the sheer size problem you mentioned earlier, it probably 
wouldn't work well.

> > Another possibility would be to use something like ActiveRecord -- 
> 
> Using the db especially through AR would be glacial. We have a db-based 
> process now, and need something faster.

I specifically mean something already designed for this purpose -- not 
necessarily a traditional database. Something like berkdb, or "stone" (I 
think that's what it was called) -- or splitting it into a bunch of files, on 
a decent filesystem.