From: Alex Stahl <astahl@...5.com>
Date: 2010-09-29T19:01:55+09:00
Subject: Re: Point me to help w/ multithreading in 1.9.2-p0

Thanks for the note, Eric.  Each thread is only opening one file
descriptor, so I haven't encountered a limit there.  And a refactor
eliminated the segfaults.

Although, after further rewriting, I'm now bumping into a maximum of
~2,950 threads when running on 1.9.2-p0 (on Ubuntu 10.04).  When I run
w/ >2,950, there's one of three errors I encounter:

-can't create Thread (11) (ThreadError) (from my class)
-out of memory error (from my class; sorry, missed a chance to copy)
-Cannot assign requested address - connect(2) (Errno::EADDRNOTAVAIL)
(from HTTP class)

Changed up the class to be a little more concise:

  def run_threads()
    thr = []
    1.upto @count do
      thr << Thread.new do
        @tests.each do |test|
          p test.call @obj, @session
          Thread.pass
        end
      end
    end
    thr.each {|t| t.join}
  end

So what limits might I be bumping into now?  The "can't create...' seems
to be the most common error - what could cause that?  Is my best course
of action to d/l source, grep for that string, and analyze from there?
Or is this possibly a bug?  Or am I beyond expected threading usage??

Thanks,
Alex


On Tue, 2010-09-28 at 14:37 -0500, Eric Wong wrote:
> Alex Stahl <astahl@hi5.com> wrote:
> > 3. Is there a theoretical maximum to the number of concurrent threads
> > which can be created which all access a network interface?  This is
> > admittedly a poor theory - what might really cause a segfault in 1.9.2
> > when 500 threads all try to access the network?
> 
> Are you opening multiple file descriptors per thread?  If you exceed
> 1024 file descriptors per-process, then the select() interface will
> overrun the buffers and segfault.  You can fork() your process to get
> around this (and get more CPU/memory concurrency), or do all your IO
> over something like Rev[1] or EventMachine[2] which use epoll or kqueue.
> 
> Which OS is this on?  There may be some lingering pthreads portability
> issues for non-NPTL.  Definitely talk to ruby-core about this.
> 
> 
> [1] - http://rev.rubyforge.org/
> [2] - http://rubyeventmachine.com/
>