From: Steven Lumos <slumos@...>
Date: 2003-12-11T10:27:02+09:00
Subject: [Q] Fast loading of BIG data structures


I have a (really) big data structure, which looks like:

  Features = [
    ["feature", ["parent1", "parent2"], {:C1 => [0.0, (7 more)], :C2 => ...}],
    ...a LOT more (like 2500) lines...
  ]

I was initially using (shudder) XML for storage, but as the structure
developed, we literally had to choose between dumping XML or dumping
Ruby.  I thought about other generic methods but ended up settling on
writing out the Ruby representation of the structure and loading it
with require.  What I'm currently using is exactly like the above but
with the constant wrapped in a module.

My question is: Is there an even faster way to load a big structure
than this?

My first thought was to use Marshal, but I was surprised to find that
Marshal.load takes about twice as long as require:

  $ ruby -v
  ruby 1.8.0 (2003-08-04) [sparc-solaris2.8]
  $ time ruby -e "require 'network'"
  real    0m3.431s
  user    0m3.080s
  sys     0m0.200s
  $ time ruby -e "File.open('network.dump') {|f| Marshal.load(f)}"
  real    0m7.321s
  user    0m6.880s
  sys     0m0.170s

Steve