Re: Slow reading output from Open3.popen3??

On 05/14/2013 05:38 PM, Thomas Luedeke wrote:
> I've never tried using this function before, but I am now using as part
> of a code compilation build script (using 'make' on UNIX).
>
> It looks something like this:
>
> ==================================
>
> stdin, stdout, stderr = Open3.popen3( 'make codename' )
> stdin.close
>
> puts "Reading STDOUT"
> outfile.puts stdout.read
> stdout.close
>
> ==================================
>
> I'm trying to get the output from the 'make' command (which is quite
> large), so I can dump it to a file for later diagnostics.

There are a couple ways to tackle what you want to do, but one big
problem with what you're doing here is that you're trying to read *all*
of the output from stdout into memory at once.  I'm not sure how big
"quite large" is, but you should probably do this line by line or in
limited block sizes instead.

stdout.each_line do |line|
  outfile.write(line)
end

OR

while (data = stdout.read(1024)).size > 0
  outfile.write(data)
end

> However, when it gets to the stdout.read, it seems to hang (or at least
> take so long I've never seen it finish).
>
> Any thoughts on why this is happening?  Is there a better approach for
> me to try?

You never read from stderr in your example.  I'm wondering if the
subprocess isn't actually blocking while trying to write something to
stderr.  Because you never read from it, the buffer for the pipe
connected to stderr may fill up.  If that were to happen while the
subprocess was writing to stderr, the subprocess would block while
trying to write and never get to close its end of the stdout pipe and
exit.  Thus your read from stdout will block forever.

You need to read from both stdout and stderr in your script to avoid
this problem; however, this can get complicated since you need to avoid
blocking while reading from either one of them.  You could also skip
stderr entirely and just use IO.popen, in which case the stderr output
will go to the terminal or wherever your script's stderr was wired to
go.  Finally, you could use IO.popen while redirecting the subprocess'
stderr to its stdout so that you can read both streams at once:

IO.popen('make codename 2>&1', 'r') do |pipe|
  pipe.each_line do |line|
    outfile.write(line)
  end
end

Given your stated goal of sending the output of make to a file for later
diagnostics, you would probably be better off simply redirecting the
output directly to a file and skipping ruby entirely:

bash$ make codename >/path/to/make.log 2>&1

The above when run in the bash shell would send both the stdout and
stderr into the file /path/to/make.log.  If you still wanted to see the
output on screen at the same time as logging to the file, you can use
the tee program:

bash$ make codename 2>&1 | tee /path/to/make.log

The same thing happens here as before but you'll also see the output in
the terminal while make runs.

-Jeremy