fatal flaw in popen4 on windows? [WAS] Re: Nonblocking IO read

On Thu, 2 Nov 2006, Robert Klemme wrote:

> Tom Pollard wrote:
>> Anyway, you would only need nonblocking IO if you wanted to read bits of 
>> the stderr stream before the command exited, but that doesn't sound like 
>> what you're want.
>
> Actually this is not correct: if there is a lot written to stderr then you
> need to read that concurrently.  If you do not do that then the process will
> block on some stderr write operation that fills up the pipe and you get a
> deadlock because your code waits for process termination.

not only is that true but, afaik, it's why popen4 cannot even work on windows!
this program will eventually hang on either windows or unix


     harp:~ > cat a.rb
     require 'rubygems'
     require 'popen4'

     n = (ARGV.shift || 4242).to_i
     ruby = ARGV.shift || 'ruby'

     system "ruby -e 42" or abort "ruby not in your path!"

     STDOUT.sync = STDERR.sync = true

     program = <<-program
       #{ n }.times do
         t = Time.now.to_f
         STDOUT.puts t
         STDERR.puts t
       end
     program


     POpen4.popen4(ruby) do |stdout, stderr, stdin, pid|
       STDOUT.puts pid

       stdin.puts program
       stdin.close

       Thread.new{ stdout.each{|line| STDOUT.puts line} }
       #
       # uncomment and it won't hang!!!
       #
       #Thread.new{ stderr.each{|line| STDERR.puts line} }
     end

     puts 'done'



however, on windows it will always hang - even if the line above is
uncommented.  this is because if one popen4s a process it's __essential__, as
robert correctly points out, to continually consume any stdout or stderr
produced - otherwise the program will eventually get stuck in EPIPE and you'll
be waiting for this stuck program.

and here's the rub: you cannot reliably consume both stdout and stderr under
windows using threads or, afaik, nonblocking io.  perhaps the new nonblock_*
methods could help with this?  it'd no doubt be a major reworking...

*** i'm really hoping someone will chime in here and prove me wrong ***

for reference i'm including the code for my open4 lib's spawn method: which
illustrates the logical concept of what must be done to avoid a subprocess
blocked writing to it's parent's pipes...


#
# for complete code see
# http://codeforpeople.com/lib/ruby/open4/open4-0.9.1/lib/open4.rb
# http://rubyforge.org/frs/?group_id=1024&release_id=7556
#

   def spawn arg, *argv 
#--{{{
     argv.unshift(arg)
     opts = ((argv.size > 1 and Hash === argv.last) ? argv.pop : {})
     argv.flatten!
     cmd = argv.join(' ')

     getopt = getopts opts

     ignore_exit_failure = getopt[ 'ignore_exit_failure', getopt['quiet', false] ]
     ignore_exec_failure = getopt[ 'ignore_exec_failure', !getopt['raise', true] ]
     exitstatus = getopt[ %w( exitstatus exit_status status ) ]
     stdin = getopt[ %w( stdin in i 0 ) << 0 ]
     stdout = getopt[ %w( stdout out o 1 ) << 1 ]
     stderr = getopt[ %w( stderr err e 2 ) << 2 ]
     pid = getopt[ 'pid' ]
     timeout = getopt[ %w( timeout spawn_timeout ) ]
     stdin_timeout = getopt[ %w( stdin_timeout ) ]
     stdout_timeout = getopt[ %w( stdout_timeout io_timeout ) ]
     stderr_timeout = getopt[ %w( stderr_timeout ) ]
     status = getopt[ %w( status ) ]
     cwd = getopt[ %w( cwd dir ), Dir.pwd ]

     exitstatus =
       case exitstatus
         when TrueClass, FalseClass
           ignore_exit_failure = true if exitstatus
           [0]
         else
           [*(exitstatus || 0)].map{|i| Integer i}
       end

     stdin ||= '' if stdin_timeout
     stdout ||= '' if stdout_timeout
     stderr ||= '' if stderr_timeout

     started = false

     status =
       begin
         Dir.chdir(cwd) do
           Timeout::timeout(timeout) do
             popen4(*argv) do |c, i, o, e|
               started = true

               %w( replace pid= << push update ).each do |msg|
                 break(pid.send(msg, c)) if pid.respond_to? msg
               end

#
# this is the critical bit!!!
#

               te = ThreadEnsemble.new c

               te.add_thread(i, stdin) do |i, stdin|
                 relay stdin, i, stdin_timeout
                 i.close rescue nil
               end

               te.add_thread(o, stdout) do |o, stdout|
                 relay o, stdout, stdout_timeout
               end

               te.add_thread(e, stderr) do |o, stderr|
                 relay e, stderr, stderr_timeout
               end

               te.run
             end
           end
         end
       rescue
         raise unless(not started and ignore_exec_failure)
       end

     raise SpawnError.new(cmd, status) unless
       (ignore_exit_failure or (status.nil? and ignore_exec_failure) or exitstatus.include?(status.exitstatus))

     status
#--}}}
   end


kind regards.

-a
-- 
my religion is very simple.  my religion is kindness. -- the dalai lama

-a
-- 
my religion is very simple.  my religion is kindness. -- the dalai lama