From: Robert Klemme <shortcutter@...>
Date: 2011-11-11T21:50:28+09:00
Subject: Re: The fastest way to read files

2011/11/11 Noé Alejandro <casanejo@gmail.com>:
> I mean, the final processing is about compare (preprocessed) content of
> each pair of texts. So, I open a file, I remove blanks and so on, and I
> record the information in a data structure. Then I open other file,
> remove blanks and so on, and I record this new information in other data
> structure. Now I have preprocessed information of a pair of texts, and
> then I apply other processing to it.
>
> I repeat previous steps for each text.

Aha.  I assume you do your analysis based on words.  In that case
something like this might be efficient:

# ensure every word is only once in memory
words = Hash.new {|h,k| k.freeze; h[k] = k}
...

words_in_file = []

File.foreach a_file_name do |line|
  line.scan(/\w+/) do |word|
    word.downcase!
    words_in_file << words[word]
  end
end

Kind regards

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/