From: "Ara.T.Howard" <ahoward@...>
Date: 2003-12-12T01:57:05+09:00
Subject: Re: [Q] Fast loading of BIG data structures

On Thu, 11 Dec 2003, Steven Lumos wrote:

> Date: Thu, 11 Dec 2003 01:23:56 GMT
> From: Steven Lumos <slumos@yahoo.com>
> Newsgroups: comp.lang.ruby
> Subject: [Q] Fast loading of BIG data structures
> 
> 
> I have a (really) big data structure, which looks like:
> 
>   Features = [
>     ["feature", ["parent1", "parent2"], {:C1 => [0.0, (7 more)], :C2 => ...}],
>     ...a LOT more (like 2500) lines...
>   ]
> 
> I was initially using (shudder) XML for storage, but as the structure
> developed, we literally had to choose between dumping XML or dumping
> Ruby.  I thought about other generic methods but ended up settling on
> writing out the Ruby representation of the structure and loading it
> with require.  What I'm currently using is exactly like the above but
> with the constant wrapped in a module.

that's a cool idea: a code generation database

> My question is: Is there an even faster way to load a big structure than
> this?

if you could simplify your structure a little it might be good to put into a
bdb.  you may not see a huge performance gain for one process, but bdb uses
memory pools and so you should see big gain if > 1 process is accessing the
data in a read-only way.

-a
-- 

ATTN: please update your address books with address below!

===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| STP     :: http://www.ngdc.noaa.gov/stp/
| NGDC    :: http://www.ngdc.noaa.gov/
| NESDIS  :: http://www.nesdis.noaa.gov/
| NOAA    :: http://www.noaa.gov/
| US DOC  :: http://www.commerce.gov/
|
| The difference between art and science is that science is what we
| understand well enough to explain to a computer.  
| Art is everything else.  
|   -- Donald Knuth, "Discover"
|
| /bin/sh -c 'for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done' 
===============================================================================