Re: Reading stdout & stderr from a pipe with popen3

On Jul 3, 2008, at 2:07 PM, Matt Mencel wrote:

> A coworker suggested I run the original "zmprov" command as "zmprov  
> 2>&1" so that stderr is returned in stdout.  Which will work...but I  
> was just wondering if I could get the code to work as designed.  If  
> stdout is returning data...read it and do something....if stderr is  
> returning data then do something with that.  I'm wanting to keep one  
> from blocking my ability to read the other.


the issue is even worse than you describe, the program can easily  
become blocked if it's stdout or stderr pipes get full - to fix the  
situation you need to use threads, one processing both stdout and  
stderr asynchronously where each may have to trigger actions on  
stdin.  the general pattern is


   q = Queue.new

   err = Thread.new do
     Thread.current.abort_on_exception = true

     while(( line = stderr.gets ))
       ...
       q.push :somthing if some_condition_on(line)
     end
     q.push :stderr_done
   end

   out = Thread.new do
     Thread.current.abort_on_exception = true

     while(( line = stdout.gets ))
       ...
       q.push :something if some_condition_on(line)
     end
     q.push :stdout_done
   end

   in = Thread.new do
     Thread.current.abort_on_exception

     while(( command = q.pop ))
       ...
       break if stdout_done and stderr_done
     end
   end

   in.join

so basically have one thread sending commands down stdin.  start a  
thread each for stdout and stderr, each doing their own processing, if  
they encounter something which means input needs to be send push it  
onto a queue to allow the stdin thread to do it on their behave.  this  
of course ignores exceptional conditions and coordination between the  
stdout and stderr threads, but it's one approach.


the big conditions any solution needs to handle are having no output  
on either stderr or stdout or being blocked on a write to either due a  
full pipe, which is why this cannot work safely:

   loop do
     handle stdout.gets
     handle stderr.gets
   end

check out open4 and session for examples of using threads to process  
both stdout and stderr concurrently.

   http://codeforpeople.com/lib/ruby/
   # gem install open4 session

cheers.


a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being  
better. simply reflect on that.
h.h. the 14th dalai lama