From: Philip Mak <pmak@...>
Date: 2003-03-12T21:28:04+09:00
Subject: Marshal.dump() taking a long time

I have a program that parses a text file into a list of objects, then
tries to write it out using Marshal.dump(). But, it's taking a
ridiculous amount of time.

This is how I'm using Marshal.dump:

STDERR.puts "Successfully parsed #{line_num} entries. Saving..."
print Marshal.dump(dict)
STDERR.puts "Saved."

And this is what happens:

$ ./edict.rb < edict > edict.dat
Parsing dictionary entries...
Successfully parsed 103527 entries. Saving...
(it just sits here forever)

The text file itself consists of 103527 lines, and it took less than
10 seconds before it said "Successfully parsed". But I waited several
minutes, and the Marshal.dump() still didn't complete.

The object being dumped is a list of 103527 objects, each containing
two strings, and a short list of strings. This object is generated
from a 5 MB data file.

It runs fine if I feed it only a 1000-line sample of the actual data
file.

Does anyone have suggestions on how to make Marshal.dump() work
faster? I wanted to be able to load the parsed representation of the
file quickly whenever the program starts, rather than having to parse
a text file every time...