From: "ara.t.howard" <ara.t.howard@...>
Date: 2008-01-01T07:27:27+09:00
Subject: Re: deadlock in ThreadPool using backtick


On Dec 31, 2007, at 1:59 PM, Charles Oliver Nutter wrote:

>
> Isn't this largely just working around a lack of concurrency in  
> Ruby's threading model (or perhaps a bug in backtick handling)?  
> With concurrent threads I can implement the same thing in a trivial  
> amount of code and not have to worry about backticks et al  
> blocking, nor about spinning up processes. In fact with native  
> threads the original ThreadPool from Ruby Cookbook ought to work  
> fine for basically anything you want to throw at it (including  
> backticks).
>
> Perhaps backtick needs to be fixed in MRI so it doesn't block the  
> entire runtime?

in general you may be correct - but i think the OP's code has a race  
condition.  if you look at the stacktrace it's not reading stdout  
(backticks) that's got everyone's undies in a wedgie - it's the  
shutdown method.  so basically the original code starts out by firing  
off a bunch of threads, limited at the number 2.  let's assume it  
safely got two running, but that none of have completed, that leaves  
every other thread blocked here

       @pool_mutex.synchronize do
         while @pool.size >= @max_size
           # Sleep until some other thread calls @pool_cv.signal.
           @pool_cv.wait(@pool_mutex)

so let's assume they've all just went to sleep...

now the main code starts to shutdown, calling

   def shutdown
     @pool_mutex.synchronize ...

so it has the lock

now the first two finish and try to take them selves out of the @pool  
by doing

    ensure
      @pool_mutex.synchronize do
        @pool.delete(Thread.current)

and of course go to sleep since the main thread has that lock

now the main thread goes to sleep via

   @pool_cv.wait(@pool_mutex) ...

so now everyone is asleep and no one will be signaled - deadlock.

i think the backticks are a red herring and that the shutdown method  
would need to be reworked, at minimum.


threads are just too complicated without Queue IMHO.  the classic  
race condition you see all over the place is

   Thread.new{ thread_working }

   something_that_depends_on_thread_working

but that only works sometimes, specifically when the thread_working  
gets started before the next line - which is not always true. etc. etc.

i just glanced at the code so maybe i've overlooked something - but i  
think the OP's code is racing, not ruby.

cheers.

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama