From: Anders Lindgren <andlind@...>
Date: 2007-03-06T17:15:22+09:00
Subject: Thread reading from a pipe blocks other threads, why?

Hi!

I'm implementing an application where the main Ruby program spawns off
a number of external programs using pipes. Each pipe is handled by its
own thread. Well, so far so good... Unfortunately, I'm having problems
since when one thread calls "pipe.each_line" the other (!) threads
hang. Well, if I add a "Process.waitpid" call before the call to
pipe.each_line then the external command will sooner or later hang
since noone consumes anything in the pipe.

To exemplify, below are two ruby programs. The first, lotsoflines.rb,
will print so much output that the pipe will be filles (in the real
world this is typically not written in Ruby). The second, test.rb,
without the "waitpid" will block the thread printing "BG Thread"; with
the "waitpid" line the background process will never terminate.

So, what I would need is a way to read from the pipe *without blocking
other threads*, or a wait until there is data to read, or similar. All
suggestions are welcome!

Oh, btw, I need this to work both under win32 and unix.

    -- Anders Lindgren

------------------------------------ lotsoflines.rb
ARGV[0].to_i.times do puts "A LINE" end
sleep 2
------------------------------------ test.rb
cmd = "ruby lotsoflines.rb 300"

Thread.new do
  while true
    sleep 0.1
    puts "BG Thread"
  end
end

puts "Before popen"

IO.popen(cmd) do |pipe|

  puts "Inside popen"

  # With this line, the background thread will be blocked, with this
  # line the process at the other end of the pipe will hang...

  Process.waitpid(pipe.pid)

  pipe.each_line do |x|
    puts "XXX:" + x
  end
end

puts "After popen"