From: Michal Suchanek <hramrach@...>
Date: 2008-08-06T22:02:51+09:00
Subject: Re: Threaded IO trouble

On 06/08/2008, Michal Suchanek <hramrach@centrum.cz> wrote:
> Hello
>
>  I do not understand the internals of Ruby IO but what I get here looks
>  really awful.
>
>  I am interfacing an application that processes lines of text and
>  buffers part of the output until EOF (or EOF tag). I thought using an
>  additional thread would be the right thing to do in this case: one
>  thread writes stuff, and as the application starts to output stuff
>  another thread wakes up to read it.
>
>  There are two problems here:
>
>  1) only reading inside the separate thread is possible, writing causes
>  stdin to be read into the program
>
>  2) reading inside the thread produces deadlocks
>
>  Attaching the files required to reproduce both failures.
>
>  =================== mock.rb ===================
>  buf=[]
>  while l=gets
>   buf << l
>   puts buf.shift if buf.length >3
>  end
Actually the mockup misses a line here:

 while l = buf.shift do puts l end

However, the program reproduces the errors anyway.
>  =================== garbage ===================
>  ilks nvd
>  sdh v
>  df bhl
>  dj fbo
>  fj wr;oug
>   rbfg
>  =================== testt.rb ===================
>  def unidentified l
>   false
>  end
>
>  def try_analyze *words
>   return [] if words.length < 1
>   analyzer = IO.popen( 'ruby -w mock.rb', IO::RDWR | IO::SYNC )
>   res = []
>   t = Thread.new( (IO::for_fd analyzer.fileno) ,res){|fd,ary|
>     while l = fd.gets do
>       break if  l =~ %r|^</csts>$|
>       if l =~ /^$/
>         STDERR << "try_analyze: Empty line in output! \n"
>         next
>       else
>         ary.push l
>       end
>     end
>     fd.close rescue nil # hopefully prevents zombie hordes
>   }
>   words.each{|w|
>     STDERR.puts "try_analyze: #{w}"
>     analyzer.puts "<f>"+w
>   }
>   analyzer.puts "</csts>"
>   analyzer.close_write
>   t.join
>   res.map{|w|
>     if unidentified w then
>       nil
>     else if w !~ /^<f/
>       STDERR << "try_analyze: JUNK: #{w} \n"
>       nil
>     else
>       w
>     end end
>   }
>  end
>
>  1.upto(10000){|_|
>   try_analyze *%w(a b c d e f g h i)
>  }
>  =================== testt2.rb ===================
>  def unidentified l
>   false
>  end
>
>  def try_analyze *words
>   return [] if words.length < 1
>   analyzer = IO.popen 'ruby -w mock.rb'
>   res = []
>
>   t = Thread.new( (IO::for_fd analyzer.fileno) ,words){|fd,ary|
>     fd.sync = true
>     ary.each{|w|
>       STDERR.puts "try_analyze: #{w}"
>       fd.puts "<f>"+w
>     }
>     fd.puts "</csts>"
>     analyzer.close_write
>   }
>   analyzer.sync = true
>   while l = analyzer.gets do
>     break if  l =~ %r|^</csts>$|
>     if l =~ /^$/
>       STDERR << "try_analyze: Empty line in output! \n"
>       next
>     else
>       res.push l
>     end
>   end
>   fd.close rescue nil # hopefully prevents zombie hordes
>   res.map{|w|
>     if unidentified w then
>       nil
>     else if w !~ /^<f/
>       STDERR << "try_analyze: JUNK: #{w} \n"
>       nil
>     else
>       w
>     end end
>   }
>  end
>
>  1.upto(10){|_|
>   try_analyze *%w(a b c d e f g h i)
>  }
>  ======================================
>
>  Use:
>
>  $ cat garbage | ruby -w  testt2.rb
>  testt2.rb:47: warning: `*' interpreted as argument prefix
>  try_analyze: a
>  try_analyze: JUNK: ilks nvd
>
>  try_analyze: JUNK: sdh v
>
>  try_analyze: JUNK: df bhl
>
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>  try_analyze: a
>
>  $ cat garbage | ruby -w  testt.rb
>
>  [lots of output snipped]
>
>  deadlock 0x53d4: sleep:F(4)  - testt.rb:14
>  deadlock 0x31704: sleep:S (main) - testt.rb:27
>  testt.rb:27:in `write': Thread(0x31704): deadlock (fatal)
>         from testt.rb:27:in `puts'
>         from testt.rb:27:in `try_analyze'
>         from testt.rb:25:in `each'
>         from testt.rb:25:in `try_analyze'
>         from testt.rb:45
>         from testt.rb:44:in `upto'
>         from testt.rb:44
>
>
>  Thanks
>
>  Michal
>
>