From: Michal Suchanek <hramrach@...>
Date: 2008-08-06T21:08:55+09:00
Subject: Threaded IO trouble

Hello

I do not understand the internals of Ruby IO but what I get here looks
really awful.

I am interfacing an application that processes lines of text and
buffers part of the output until EOF (or EOF tag). I thought using an
additional thread would be the right thing to do in this case: one
thread writes stuff, and as the application starts to output stuff
another thread wakes up to read it.

There are two problems here:

1) only reading inside the separate thread is possible, writing causes
stdin to be read into the program

2) reading inside the thread produces deadlocks

Attaching the files required to reproduce both failures.

=================== mock.rb ===================
buf=[]
while l=gets
  buf << l
  puts buf.shift if buf.length >3
end
=================== garbage ===================
ilks nvd
sdh v
df bhl
dj fbo
fj wr;oug
 rbfg
=================== testt.rb ===================
def unidentified l
  false
end

def try_analyze *words
  return [] if words.length < 1
  analyzer = IO.popen( 'ruby -w mock.rb', IO::RDWR | IO::SYNC )
  res = []
  t = Thread.new( (IO::for_fd analyzer.fileno) ,res){|fd,ary|
    while l = fd.gets do
      break if  l =~ %r|^</csts>$|
      if l =~ /^$/
        STDERR << "try_analyze: Empty line in output! \n"
        next
      else
        ary.push l
      end
    end
    fd.close rescue nil # hopefully prevents zombie hordes
  }
  words.each{|w|
    STDERR.puts "try_analyze: #{w}"
    analyzer.puts "<f>"+w
  }
  analyzer.puts "</csts>"
  analyzer.close_write
  t.join
  res.map{|w|
    if unidentified w then
      nil
    else if w !~ /^<f/
      STDERR << "try_analyze: JUNK: #{w} \n"
      nil
    else
      w
    end end
  }
end

1.upto(10000){|_|
  try_analyze *%w(a b c d e f g h i)
}
=================== testt2.rb ===================
def unidentified l
  false
end

def try_analyze *words
  return [] if words.length < 1
  analyzer = IO.popen 'ruby -w mock.rb'
  res = []

  t = Thread.new( (IO::for_fd analyzer.fileno) ,words){|fd,ary|
    fd.sync = true
    ary.each{|w|
      STDERR.puts "try_analyze: #{w}"
      fd.puts "<f>"+w
    }
    fd.puts "</csts>"
    analyzer.close_write
  }
  analyzer.sync = true
  while l = analyzer.gets do
    break if  l =~ %r|^</csts>$|
    if l =~ /^$/
      STDERR << "try_analyze: Empty line in output! \n"
      next
    else
      res.push l
    end
  end
  fd.close rescue nil # hopefully prevents zombie hordes
  res.map{|w|
    if unidentified w then
      nil
    else if w !~ /^<f/
      STDERR << "try_analyze: JUNK: #{w} \n"
      nil
    else
      w
    end end
  }
end

1.upto(10){|_|
  try_analyze *%w(a b c d e f g h i)
}
======================================

Use:

$ cat garbage | ruby -w  testt2.rb
testt2.rb:47: warning: `*' interpreted as argument prefix
try_analyze: a
try_analyze: JUNK: ilks nvd

try_analyze: JUNK: sdh v

try_analyze: JUNK: df bhl

try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a
try_analyze: a

$ cat garbage | ruby -w  testt.rb

[lots of output snipped]

deadlock 0x53d4: sleep:F(4)  - testt.rb:14
deadlock 0x31704: sleep:S (main) - testt.rb:27
testt.rb:27:in `write': Thread(0x31704): deadlock (fatal)
        from testt.rb:27:in `puts'
        from testt.rb:27:in `try_analyze'
        from testt.rb:25:in `each'
        from testt.rb:25:in `try_analyze'
        from testt.rb:45
        from testt.rb:44:in `upto'
        from testt.rb:44


Thanks

Michal