From: Stuart Clarke <stuart.clarke1986@...>
Date: 2010-07-01T20:58:49+09:00
Subject: Re: Fast searching of large files

Thanks.

This seems to be pretty much the best logic for me, however it takes a 
good 20 minutes to scan a 2Gb file.

Any ideas?

Thanks

Michael Fellinger wrote:
> On Thu, Jul 1, 2010 at 6:47 PM, Stuart Clarke
> <stuart.clarke1986@gmail.com> wrote:
>> Hey all,
>>
>> Could anyone advise me on a fast way to search a single, but very large
>> file (1Gb) quickly for a string of text? Also, is there a library to
>> identify the file offset this string was found within the file?
> 
> You can use IO#grep like this:
> File.open('qimo-2.0-desktop.iso', 'r:BINARY'){|io|
> io.grep(/apiKey/){|m| p io.pos => m } }
> 
> The pos is the position the match ended, so just substract the string 
> length.
> The above example was a file with 700mb, took around 40s the first
> time, 2s subsequently, so disk I/O is the limiting factor in terms of
> speed (as usual).
> Oh, and also don't use binary encoding if you are dealing with another 
> one ;)

-- 
Posted via http://www.ruby-forum.com/.