From: Roger Pack <rogerpack2005@...>
Date: 2008-10-10T14:21:59+09:00
Subject: Re: Memory leak problem

Elias Orozco wrote:
> I have a txt file with some data that i need to import to de database.
> I'm using ruby to import those data but i have a major problem, when i
> 've
> a large amount of data i run out of memory.
> 
> 
>   File.open("#{RAILS_ROOT}/public/files/Neighborsville.TXT").each() do
> |line|
>       @stringArray = line.split("|")
>       @i += 1
>       puts @i
>       @pid = @stringArray[0]
>       @chain_id = @stringArray[1]
>       @business = Business.find_by_pid_and_chain_id(@pid,@chain_id);
>       #Check PID + CHAIN_ID
>       @business.pid = @stringArray[0]
>       @business.chain_id = @stringArray[1]
>       @business.cityname = @stringArray[17]
>       @business.state = @stringArray[18]
>       @business.business =
> Business.find_by_pid_and_chain_id(@pid,@chain_id);
>       @business.city = City.new
>       @business.business_category = get_category_id(@stringArray[40])
>       @business.address = @stringArray[8] +" "+   @stringArray[9] +" "+
> @stringArray[10]+" "+ @stringArray[11] +" "+@stringArray[12]+"
> "+@stringArray[13]+" "+@stringArray[14]
>       if @chain_id == nil
>         @chain_id = ""
>       end
>       business.save
>       end
>     end
> 
> 
> I belive that ruby use in every cycle of the do new blocks of memories
> for my instances
> of Business. 

Yes that's right.  You've possibly run into an infamous "ruby's GC is 
broken!" bug. Then again, I could be wrong.
Question:
don't you want @business.save?

Anyway ways to avoid this:
you may be able to use ar extensions, which allows for multiple inserts 
into the DB.  Oh wait, except that you are doing multiple updates. Never 
mind.
In that case, as gross as it seems, you could try forking once per loop 
[or once every x lines of the input file]--that way the forked process 
will die [with its high RAM consumption] allowing the parent process to 
continue [and fork more].

require 'forkoff'
   File.open("#{RAILS_ROOT}/public/files/Neighborsville.TXT").each() do
 |line|
    [1].forkoff {
         # do your stuff
    }
   end

Maybe it won't help.  I know with huge data sets it helps me avoid the 
GC.  I've never tried it with sql and rails.
Good luck.
-=R
-- 
Posted via http://www.ruby-forum.com/.