Re: Zlib::GzipReader and multiple compressed blobs in a single stream

On 02/02/2011 07:33 PM, Eric Hodel wrote:
> On Jan 28, 2011, at 15:09, Jos Backus wrote:
>> I'm trying to inflate a set of concatenated gzipped blobs stored in a single
>> file. As it stands, Zlib::GzipReader only inflates the first blob. It
>> appears that the unused instance method would return the remaining data,
>> ready to be passed into Zlib::GzipReader, but it yields an error:
>>
>> method `method_missing' called on hidden T_STRING object
>>
>> What could be going on here?
> 
> It's a bug, the internal buffer that libz uses is dup'd, but this is not enough to make it safe for use by ruby.  I have filed a ticket and attached a stupid patch:
> 
> http://redmine.ruby-lang.org/issues/show/4360

Once your fix is in place and GZipReader#unused works correctly, is
there any convenient way to take the returned string and continue
processing it along with the remaining file contents with an instance of
GzipReader?

From my testing, it appears that GzipReader.open in Ruby 1.9 always
rewinds any IO object you give it before inflating any data, so you
can't use that method to create your instance if you need to start
reading from anywhere other than the beginning of the stream.
GzipReader.new doesn't have that problem, but there isn't any easy way
to make use of that unused data from the earlier processing along with
the remaining file contents.  According to the documentation, you could
create an IO-like wrapper that will first feed in that unused data
followed by the real file data, and GzipReader.new should be able to use
that, but that's a bit of a mess.

If all that really is a design limitation of GzipReader, having the
unused data isn't very useful when attempting to inflate concatenated
gzip blobs as zcat does.  You may be able to make it work with a little
judicious hacking, but it's certainly more effort than it should be.
Maybe a ZcatReader is needed to plaster over things?

BTW, why do GzipReader.open and GzipReader.new behave so differently
with regard to the IO object you pass into them?  They're a little
closer in operation under Ruby 1.9 than they were under Ruby 1.8, but
the difference is still surprising given the idiom followed by File.open
and File.new where File.open is really just a simple wrapper around
File.new that can help ensure that File#close is called at the end of
your block.

-Jeremy