From: Joel VanderWerf <vjoel@...>
Date: 2009-04-16T06:43:48+09:00
Subject: Re: Tokenizing a large file

Eric Hodel wrote:
> On Apr 15, 2009, at 09:48, Don Wood wrote:
>> I have a large file that I need to tokenize.  The method I am using now
>> is fast, but eats up a ton of memory by reading in the entire file first
>> as a String.  I would also like to reuse existing tokens for duplicates.
>> (I have no control over the file format, but this Regex works well for
>> what I need.)
>>
>> Here is what I am doing today.
>>
>> tokens= File.read(filename).scan(/'[^']*'|"[^"]*"|[(:)]|[^(:)\s]+/)
>>
>> And here is what I would like to do.
>>
>> tokens= []
>> File.open(filename) do |fh|
>>  fh.scan(/'[^']*'|"[^"]*"|[(:)]|[^(:)\s]+/) do |token|
>>    tokens << i=tokens.index(token) ? tokens[i] : token
>>  end
>> end
>>
>> So what I would like to have is a scan method for File objects that
>> yields the tokens when called with a block, instead of returning an
>> array.  (It would be nice if String#scan could do this as well.)  This
>> isn�t a big issue, it just causes my machine to overflow to the swap
>> file periodically.  I could easily fix that with a couple DIMMs, but I
>> can�t help thinking that there should be a better way.
> 
> You should look at StringScanner in strscan.rb, it'll allow you to 
> intern your tokens like you want.

I was going to suggest that, but:

$ irb -r strscan
irb(main):001:0> StringScanner.new(File.open('tmp/t'))
TypeError: can't convert File into String
         from (irb):1:in `initialize'
         from (irb):1:in `new'
         from (irb):1

Is there some way to use StringScanner with an open file?

(also, my ruby 1.8.6 only comes with ext/strscan, not lib/strscan.rb... 
maybe we're talking about different things)

-- 
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407