From: James Tucker <jftucker@...>
Date: 2008-04-06T05:07:39+09:00
Subject: Re: memory allocation failure: heavy files


On 5 Apr 2008, at 20:58, Michael Linfield wrote:
> Seems like any combo I try, eventually it lands me in a NoMemory  
> error.
> I've tried:
>
> ####
> file = File.open(datafile)
>
>  file.each{|data| @store << data}
>  file.close()
>
>  @large = @store.to_s.split(" ")
> ####
> File.open(datafile) do |file|
> data = file.gets.to_s.split(" ")
> end
> ####
> (this appears to be the most inefficient one)
> file = File.readlines(datafile)
> file.each{|data| @store << data}
> @large = @store.to_s.split(" ")
> ####
>
> And this is with only a 24mb datafile, to which effect I had to split
> the file.
> The total size of the file is around 250-300mb.
>
> Code inefficiency issue?
>
> Thanks,
>
> - Mac
> -- 
> Posted via http://www.ruby-forum.com/.
>



You'll want to buffer data reads, and do the splits on data that's  
buffered properly and completely. Many of the operations there slurp  
or are never dropping data. If you have a *very* long line, there may  
be more work required, too, also dependent on ram.