From: Eric Wong <normalperson@...>
Date: 2010-07-09T08:49:25+09:00
Subject: [ruby-core:31140] Re: [Feature #1478] Ruby archive

Hongli Lai <redmine@ruby-lang.org> wrote:
> However I think there's another legit use case for a Ruby Archive
> format that might have been overlooked: improving cold startup
> performance. A typical Rails app needs to load tons of rb files from
> disk. This leads to a lot of disk seeking. On NFS this is even more
> painful. We have a client that stores their Rails app files on NFS
> file servers and starting up a single Rails app worker can take half a
> minute thanks to all the I/O. If you can bundle all rb files in the
> same file, which is compressed but has an uncompressed index, then you
> can improve cold startup performance of the Rails app by many times.
> I've experimented with this about 2 years ago with zip files and and
> it sped up cold startup performance by a factor of 5, though warm
> startup performance suffered a little. I found that zip files were not
> adequate because their table of contents are not sorted so to lookup a
> single file you need to linearly scan through the entire table of
> contents. A new format will probab!  ly have to be introduced for
> maximum performance.

I completely agree compression helps performance on modern systems given
huge disparity between disk performance and memory/CPUs.

However, keep in mind there are also filesystems/kernel folks are also
working on improving these situations with transparent FS compression,
better caching (especially for remote filesystems), and even in-memory
compression of caches.  On the hardware side, SSDs are becoming more
common, too.

If Ruby does end up going with an archive format, I favor using
TokyoCabinet even if it is an extra dependency.  It is concurrency-safe,
fast and does transparent compression.

-- 
Eric Wong