From: Michal Suchanek <hramrach@...>
Date: 2008-07-16T18:57:10+09:00
Subject: Re: [rubyzip + open-uri] reading zipfiles from a url?

On 13/07/2008, David Masover <ninja@slaphack.com> wrote:
> On Wednesday 02 July 2008 06:49:26 Janus Bor wrote:
>
>  > url = "http://www.cibiv.at/%7Ephuong/vien/8a375.zip"
>  > zip_file = Zip::ZipFile.open(url)
>
>
> Maybe there is a right way to do this...
>
>  I'm going to argue that it would be difficult at best. Zip files store
>  metadata, such as compressed file location, at the end of the file. After
>  that, you'd want to seek somewhere in the middle. Since open-uri is probably
>  meant to issue a single, straightforward HTTP request (that is, ask for the
>  whole file, from beginning to end), I'm not sure this would work well.
>
>  But that's just an educated guess.
>
>  Depending on how portable you need this to be, you might consider the
>  FUSE-based HTTPFS:
>
>  http://httpfs.sourceforge.net/
>
>  When I last tried this, performance was pretty terrible -- no caching, and it
>  would fetch the file in blocks, so _many_ separate HTTP requests. But it
>  would probably do what you want.

This is how FUSE is designed. The caching is supposed to happen in the
kernel upper layers. The requests are sent as received from the kernel
so there is nothing httpfs can do about the granularity. In practice
the kernel requests up to ~16k chunks but probably only when the
application does large block reads.

I tried keep-alive which should speed up subsequent requests. However,
the sockets can then hang, and it takes time to detect that. Still
this should happen only when there are network problems anyway.

It is very nice for mounting CD or DVD images, extracting a single
file from a zip could also be faster. If you want all the files anyway
it's probably better to just download the zip.

Also last time I looked the httpfs at SF was broken, there was a bug
around the SSL #ifdefs in read/write. You may need to define or
undefine USE_SSL or fix the code.

FUSE is theoretically portable to *BSD but not the code there because
it uses undefined behaviour of directory operations to make the
underlying directory still visible after the mount.

Thanks

Michal