From: Hugh Sasse <hgs@...>
Date: 2008-11-08T01:51:50+09:00
Subject: Re: how to quickly find a string towards the end of a large io object



On Fri, 7 Nov 2008, bwv549 wrote:

> I really appreciate all the comments.  They are very helpful.  I
> should have been more specific:
> 
> In this case, I am dealing with a file that will usually be >= 100
> MB.  It may easily be up to 1GB or more.  The file is in binary form,
> but about 9/10ths of the way there is a small text file embedded into
> it.  I need to check a parameter in the file before processing the
> complete binary format (which I've decoded).  At this point I only
> expect to see files, but I was hoping to make it generic to any IO

You can do that with read.

> object.  I would settle on a solution for a file (rather than an IO
> object), but I would like it to be cross-platform and prefer the
> solution in ruby if it is fast enough (who wouldn't).

Other languages may be faster of course, but first make it work...
> 
> Again, the basic layout of the file:
> 
> ................................my_param = X........
let the length of your data       ^^^^^^^^^^^^ be k

First try reading the whole lot. If it's too big you'll get an exception.
Otherwise use rindex on the buffer from the successful read.

If it's a stream you'll have to read it on order anyway,
so read it in chunks with a 2*k overlap so you can definitely
fit the whole string in a chunk, and boundaries between reads
don't matter.  That should work on 10GB files.  You might want
to read in multiples of 512 bytes on Unix, maybe 4k (cluster size??)
on Windows, not sure.  

> 
> I need the value 'X' following the 'my_para = '
> 
> Thanks again!
> 
        Hugh