From: Michael Fellinger <m.fellinger@...>
Date: 2010-08-22T07:56:43+09:00
Subject: Re: YAML::dump slow

On Sun, Aug 22, 2010 at 1:45 AM, Ralph Shnelvar <ralphs@dos32.com> wrote:
> This is interesting and counterintuitive ...
>
> I am creating a YAML file with about 200,000 entries.   The YAML file "reflects" the data structure of a directory tree of a disk.   All I have is directory names and subdirectories and filenames in the (sub) directories.  I have 100,000 simulated directories and 100,000 simulated files.
>
> It takes about
>   10 seconds to create this data structure in memory
>   10 _minutes_ to create the YAML string (i.e. YAML::dump)
>    1 second to write out the YAML string (25 megabytes)
>    4 seconds to read the YAML string (why so long?)
>    8 seconds to do the YAML::load
>
> Anyone have any ideas how to speed up the creation of the YAML file or why there is such an asymmetric amount of time for YAML::dump and YAML::load?

If you're on 1.9.2, you could try the new YAML library psych:
iota ~ % time ruby -rpsych -e 'Psych.load(Psych.dump(Dir["/sys/**/*"]))'
1.20s user 0.07s system 99% cpu 1.283 total

Of course, that's just a simple Array, but would be interesting to see
how your data behaves.

-- 
Michael Fellinger
CTO, The Rubyists, LLC