From: Don Wood <dwood@...>
Date: 2009-04-17T02:31:49+09:00
Subject: Re: Tokenizing a large file

Caleb Clausen wrote:
> On 4/15/09, Don Wood <dwood@biped.us> wrote:
>> And here is what I would like to do.
>> array.  (It would be nice if String#scan could do this as well.)  This
>> isn�t a big issue, it just causes my machine to overflow to the swap
>> file periodically.  I could easily fix that with a couple DIMMs, but I
>> can�t help thinking that there should be a better way.
> 
> The sequence gem permits scanning a file directly with a regexp.
> Something like this should work:
> 
> require 'rubygems'
> require 'sequence'
> require 'sequence/file'
> tokens= []
> fh=Sequence::File.new(open(filename))
> until fh.eof?
>   tokens<<fh.scan(/'[^']*'|"[^"]*"|[(:)]|[^(:)\s]+/) #or yield token
> up to the caller...
>   fh.scan "\n"
> end
> fh.close
> 
> As I don't know your data format, I'm not sure if this is right. I'm
> assuming that your tokens are separated by newlines, but if it's more
> complicated than that, you will have to fiddle with the argument to
> the 2nd scan. (As Sequence doesn't have String#scan's bump-a-long
> behavior, you have to explicitly match the things between scanned
> patterns yourself.)
> 
> Note that Sequence::File#scan will match patterns only up to a certain
> size (4k bytes, I think). This is an inevitable consequence of using a
> Regexp against a file; you wouldn't want arbitrary amounts of
> backtracking in a 1GB+ file. Java had this restriction as well, last
> time I knew (several years ago).

Thanks Caleb,

This looks like exactly what I needed.  I'm not sure I understand the 
point of the second scan though.  The first scan should already ignore 
unquoted whitespace, including "\n".  (At least that is how it currently 
works when I scan a string.)  I don't think that I will get anywhere 
near the per-token 4k limit.
-- 
Posted via http://www.ruby-forum.com/.