From: Jeremy Bopp <jeremy@...>
Date: 2011-02-04T10:06:06+09:00
Subject: Re: Zlib::GzipReader and multiple compressed blobs in a single stream

On 02/03/2011 06:12 PM, Jos Backus wrote:
> On Fri, Feb 04, 2011 at 07:38:04AM +0900, Jeremy Bopp wrote:
>> That's great!  How does the performance compare to zcat with your data?
> 
> Comparable:
> 
> % time zcat gz > /dev/null
> zcat gz > /dev/null  0.29s user 0.00s system 99% cpu 0.296 total
> % time ./gzr > /dev/null  
> ./gzr > /dev/null  0.31s user 0.07s system 99% cpu 0.383 total
> % 

Excellent.

>> BTW, this implementation does require that you have enough memory to
>> hold all of the gzipped file data at once.  That will be a problem with
>> sufficiently large files or constrained resources.
>  
> Using the file directly should avoid that. Since we have a File, we don't need
> the StringIO object:
> 
>     require 'stringio'
>     require 'zlib'
> 
>     def inflate(filename)
>       File.open(filename) do |file|
> 	zio = file
> 	loop do
> 	  io = Zlib::GzipReader.new zio
> 	  puts io.read
> 	  unused = io.unused
> 	  io.finish
> 	  break if unused.nil?
> 	  zio.pos -= unused.length
> 	end
>       end
>     end
> 
>     inflate "gz"

The only case where I could see this failing now is if you were given a
non-seekable IO such as a socket or a pipe from which to read.  Of
course, I apparently haven't been thinking of solutions to these
problems myself very well, but you'll probably figure out something
pretty quick. ;-)

-Jeremy