From: Alex Stahl <astahl@...5.com>
Date: 2010-09-28T18:16:31+09:00
Subject: Re: Point me to help w/ multithreading in 1.9.2-p0

No prob... but I'm not sure it's quite what you're looking for.  I'm
refactoring code I can't get to work instead of finding the root cause
in the threading.  (BTW - any thread insight would be appreciated based
on this write-up - cuz there's still problems!  I'm starting to consider
that a Threaded Load Tester Gem might be handy...)

After staring at the screen for too long, I took a break to ponder
whether the organization of my threads was fundamentally flawed.  It
occurred to me that one of the problems I was having - a deadlock -
could be avoided by instantiating the threads only when needed.

If you review my original code snippet, I spin up all the threads w/ a
proc object in initialize(), then .stop them, then perform via method
a .call on each element of the thread's api array, then a .wakeup
followed by the .join.

...
1.upto @count do
  @threads << Thread.new do
    Thread.current[:hi5] = Hi5fbapi.new 'redacted params'
    Thread.current["api_calls"] = []
    apis.each do |api|
      Thread.current["api_calls"] << get_call(api) #pushes a proc obj
    end
    Thread.stop
  end
end
...
@threads.each do |thr|
  thr[:api_calls].each do |api|
    p api.call thr[:hi5], @session
  end
end
@threads.each {|t| t.wakeup.join}
...

On 1.8.7 w/ a single core processor, this is a highly deterministic
sequence, and would not deadlock.  Once deployed to a multi-core VM
running 1.9.2-p0 (selected specifically for concurrency), not so much.  

There I encountered more deadlocks and also "NoMethodError"s from
Sinatra (undefined method `bytesize' for #<Thread:0xa36ae0c dead>).
This would occur during the .each where I would .join.  So it's trying
to join a 'dead' thread.  Except that if I add anything to prevent that,
such as 

... unless t.status == 'dead'...

it would still deadlock or NoMethodError.

But further testing showed that adding any operation to the main thread,
prior to calling .join, would prevent the deadlock:

...
@threads.each do |thr|
  p thr.inspect
...

Anyway, I rewrote things where the threads are created in the method,
not initialize(), so '@api_calls' is already populated by procs, and it
works fine at small scale:

...
thr = []
1.upto @count do
  thr << Thread.new do
    @api_calls.each do |api|
      p api.call @hi5, @session
      Thread.pass
    end
  end
end
thr.each {|t| t.join}
...

This runs fine on 1.8.7/single and 1.9.2/multi at like 5-10 threads.
But when I ramp up to, say, 5,000 (it is a *load* test!), 1.8.7 is fine
but 1.9.2 segfaults.

Even 500 threads on 1.9.2 is segfaulting right now (but not 1.8.7).  I
get the handy output: 

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension
libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

I'll write it up tomorrow.

Some open questions:
1. When .join is called on a multi-core system, what qualifies as the
calling thread?  The .main on the main processor, or the thread which
instantiates my ThreadedLoadTester object?  (i.e. what if
ThreadedLoadTester is created from a sinatra thread which itself isn't
main?)

2. Sinatra regularly reports a 'NoMethodError' for 'bytesize' when the
last thread is dead but joined to the main thread.  But only when the
main thread originates w/in sinatra, and not an inline call.

3. Is there a theoretical maximum to the number of concurrent threads
which can be created which all access a network interface?  This is
admittedly a poor theory - what might really cause a segfault in 1.9.2
when 500 threads all try to access the network?

Thanks for asking :)
-Alex