From: Matt Mencel <MR-Mencel@...>
Date: 2008-07-18T01:53:59+09:00
Subject: Re: Reading stdout & stderr from a pipe with popen3

OK...using what ara.t.howard suggested, I've rewritten a block of my code and it looks something like this...

>>>>CODE
require "open3"
require "thread"

def runner(stdin, stdout, stderr, cmd)
  queue = Queue.new
  stdin.puts(cmd)
  errthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( line = stderr.gets ))
      queue.push(line)
    end
    queue.push :stderr_done
  end
  
  outthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( line = stdout.gets ))
      queue.push(line)
    end
    queue.push :stdout_done
  end
  
  inthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( stuff = queue.pop ))
      puts stuff
      break if :stdout_done and :stderr_done
    end
  end
  
  inthd.join
  errthd.exit
  outthd.exit
end

stdin, stdout, stderr = Open3.popen3("zmprov")
runner(stdin, stdout, stderr, "selectMailbox shares")
runner(stdin, stdout, stderr, "getAllFolders")
<<<<CODE

It almost works.  I found that if I didn't add the errthd.exit and outthd.exit lines, the program would get stuck on inthd.join(waiting for the break conditions?) most of the time...very rarely would it complete.  But the output is always incomplete.  If I run the commands directly from the command line it look something like this...

>>>>GOOD OUTPUT
$ zmprov
prov> selectMailbox shares
mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59
mbox shares@domain.com> getAllFolders
        Id  View      Unread   Msg Count  Path
----------  ----  ----------  ----------  ----------
         1  conv           0           0  /
        16  docu           0           0  /Briefcase
        10  appo           0           0  /Calendar
        14  mess           0           0  /Chats
         7  cont           0           0  /Contacts
         6  mess           0           0  /Drafts
        13  cont           0           2  /Emailed Contacts
       257  appo           0           0  /evite
         2  mess           0           0  /Inbox
         4  mess           0           0  /Junk
        12  wiki           0           0  /Notebook
         5  mess           0           2  /Sent
        15  task           0           0  /Tasks
         3  conv           0           0  /Trash
<<<<GOOD OUTPUT

When I run my program though it looks more like this...

>>>>>BAD OUTPUT
prov> mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59
mbox shares@domain.com>         Id  View      Unread   Msg Count  Path
<<<<<BAD OUTPUT

The first line always seems to come out right... but the 'getAllFolders' command never comes out correctly.  Maybe 1 in 10 times it will return a line or two more than just the 'mbox' line, but most of the time it's just that line of the output...the first line returned from the 'getAllFolders' command.  So the thread for standard out doesn't seem to be grabbing all the data out of the pipe?  I tried adding a short pause 'sleep 2' above the threads to see if maybe there was some slow response from them getting in the way, but that didn't really change anything.

Any help or ideas anyone can offer?

Thanks,
Matt


----- "ara.t.howard" <ara.t.howard@gmail.com> wrote:

> 
> the issue is even worse than you describe, the program can easily  
> become blocked if it's stdout or stderr pipes get full - to fix the  
> situation you need to use threads, one processing both stdout and  
> stderr asynchronously where each may have to trigger actions on  
> stdin.  the general pattern is
> 
> 
>    q = Queue.new
> 
>    err = Thread.new do
>      Thread.current.abort_on_exception = true
> 
>      while(( line = stderr.gets ))
>        ...
>        q.push :somthing if some_condition_on(line)
>      end
>      q.push :stderr_done
>    end
> 
>    out = Thread.new do
>      Thread.current.abort_on_exception = true
> 
>      while(( line = stdout.gets ))
>        ...
>        q.push :something if some_condition_on(line)
>      end
>      q.push :stdout_done
>    end
> 
>    in = Thread.new do
>      Thread.current.abort_on_exception
> 
>      while(( command = q.pop ))
>        ...
>        break if stdout_done and stderr_done
>      end
>    end
> 
>    in.join
> 
> so basically have one thread sending commands down stdin.  start a  
> thread each for stdout and stderr, each doing their own processing, if
>  
> they encounter something which means input needs to be send push it  
> onto a queue to allow the stdin thread to do it on their behave.  this
>  
> of course ignores exceptional conditions and coordination between the 
> 
> stdout and stderr threads, but it's one approach.
> 
> 
> the big conditions any solution needs to handle are having no output 
> 
> on either stderr or stdout or being blocked on a write to either due a
>  
> full pipe, which is why this cannot work safely:
> 
>    loop do
>      handle stdout.gets
>      handle stderr.gets
>    end
> 
> check out open4 and session for examples of using threads to process 
> 
> both stdout and stderr concurrently.
> 
>    http://codeforpeople.com/lib/ruby/
>    # gem install open4 session
> 
> cheers.
> 
> 
> a @ http://codeforpeople.com/
> --
> we can deny everything, except that we have the possibility of being 
> 
> better. simply reflect on that.
> h.h. the 14th dalai lama