From: "David G. Andersen" <dga@...>
Date: 2004-10-26T10:06:55+09:00
Subject: Re: Speed gap between zcat and zlib's GzipReader

On Fri, Oct 22, 2004 at 11:30:34AM +0900, Clifford Heath scribed:
> David G. Andersen wrote:
> 
> > popen("zcat foo.gz", "r") faster than GzipReader.each_line
> 
> I had a similar problem which was discussed here at length a year or
> so ago. If you avoid the block setup and use a fixed-length read, it's
> quite a bit quicker. Still nowhere near as fast as Perl though :-(.

Ahh, thanks.  So the problem is really in GzipReader's each_line
handling.  It's actually pretty close to as fast as it could go
when doing a fixed-length read.  Byte-counting only, fixed-length
read;  popen and gzipreader both take 1.4 seconds on my test file.
A zcat to /dev/null takes 1.18 seconds.  Piping to 'wc' takes 1.83
seconds.  No complaints.

gzfile_read is fast.
gzfile_read_more is fast (used by gzfile_read).
But gzreader_gets... is a dog.  It does a memcmp()
on each byte of the input string to test it against
the delimiter - yow!  So, it looks like zlib's "gets"
needs the equivalent of rb_io_getline_fast.  Would
be nice if that were easily re-used, but the FILE *
access is buried pretty deep inside of it.

Guess I'll have to dig up some spare time next week. :)

  -Dave

-- 
work: dga@lcs.mit.edu                          me:  dga@pobox.com
      MIT Laboratory for Computer Science           http://www.angio.net/