From: sabbyxtabby@... (Sabby and Tabby)
Date: 2003-09-19T09:16:02+09:00
Subject: Re: Why IO#readlines does'nt accept a Regexp?

"Robert Klemme" <bob.news@gmx.net> wrote:

> For large Files this is more efficient:
> 
> words=[]
> IO.foreach("myfile") do |line|
>   words.push( *line.scan( /\w+/oi ) )
> end

The /oi modifiers aren't necessary.

> If you have many repeating words you can save even more mem:
> 
> cache = Hash.new {|h,k| h[k]=k}
> words = []
> 
> IO.foreach("myfile") do |line|
>   words.push( *( line.scan( /\w+/oi ).map {|w| cache[w]} ) )
> end

The #map isn't doing what you think it is doing.  To remove repeating
words from the list:

  saw = Hash.new {|h,k| h[k] = true; false}
  words = []

  IO.foreach("myfile") do |line|
    words.push(*( line.scan(/\w+/).reject {|w| saw[w]} ))
  end

Or if word order isn't a concern:

  cache = {}

  IO.foreach("myfile") do |line|
    line.scan(/\w+/).each {|w| cache[w] = 1}
  end

  words = cache.keys