Re: how to quickly find a string towards the end of a large io object

On Thu, 6 Nov 2008, Lloyd Linklater wrote:

> bwv549 wrote:
> > How do I scan starting at the end of a big io object to find a string
> > that is always closer to then end?
        [...]
> > In other words, I have a massive IO object that I cannot reverse and I
> > need to very quickly find a string towards the end (but still fairly
> > far from the end).  What's the fastest way to do this?
> > 
> > Thanks in advance!
> 
> They are correct that more info is needed.  If it is just a huge amount 
> of text and it absolutely *had* to run faster, in other languages I 
> would break up the string and have separate threads search.

Other things to take into account would include:
 * Will be big string be kept for some time?
 * Will there be multiple searches on it?
 * Is it bigger than RAM + swap?

You might find that if you're doing multiple searches that a suffix array
(see Programming Pearls) is useful. You may wish to look at Ferret. You
may want to call out to another program to do this if is is obtained from
a file.

ri String#rindex gives:

---------------------------------------------------------- String#rindex
     str.rindex(substring [, fixnum])   => fixnum or nil
     str.rindex(fixnum [, fixnum])   => fixnum or nil
     str.rindex(regexp [, fixnum])   => fixnum or nil
------------------------------------------------------------------------
     Returns the index of the last occurrence of the given _substring_,
     character (_fixnum_), or pattern (_regexp_) in _str_. Returns +nil+
     if not found. If the second parameter is present, it specifies the
     position in the string to end the search---characters beyond this
     point will not be considered.

        "hello".rindex('e')             #=> 1
        "hello".rindex('l')             #=> 3
        "hello".rindex('a')             #=> nil
        "hello".rindex(101)             #=> 1
        "hello".rindex(/[aeiou]/, -2)   #=> 1

Does this actually search from the end backwards?  Not checked the source
(recently enough || at all) to know, but I'd expect so.

        HTH
        Hugh