From: khaines@...
Date: 2007-01-04T02:08:54+09:00
Subject: Re: Memory Leak Madness

On Thu, 4 Jan 2007, Brandon Casci wrote:

> Do you mean this by locking?
>
> @mutex.synchronize do
>  @myarray << station
> end
>
> Becuase I do that in two spots, although I do .clear out that
> frequently.

Do you have situations where you end up with a lot of threads?  Do you 
then have a lot of threads all using the same Mutex?  There's an issue 
with the way Ruby handles the memory allocated to an array when values are 
shifted off of it.  In short, Mutex uses an array to manage the queue of 
waiting threads.  It pushes onto it and shifts them off of it, and if you 
have a lot of threads, you will see what seems to be inexplicable memory 
usage as a consequence.  Also be aware that if you use shift() on arrays 
elsewhere, your arrays are using more memory than you think.

The current best fix is to use the fastthread library if you can.  It 
replaces the Ruby threading support items like Mutex with much faster C 
versions.

If, for whatever reason, you can't do this, you can override the 
definitions of the Thread lock() and unlock() methods to use unshift and 
pop instead of push and shift for placing items into the queue and taking 
them off.  This doesn't hold a candle to all that's being done with 
fasthread, but it does eliminate a bad RAM usage issue if you really can't 
use fasthread for whatever reason.

Something like this:

class Thread

   def lock
     while (Thread.critical = true; @locked)
       @waiting.unshift Thread.current
       Thread.stop
     end
     @locked = true
     Thread.critical = false
     self
   end

   def unlock
     return unless @locked
     Thread.critical = true
     @locked = false
     begin
       t = @waiting.pop
       t.wakeup if t
     rescue ThreadError
       retry
     end
     Thread.critical = false
     begin
       t.run if t
     rescue ThreadError
     end
     self
   end

end


Kirk Haines