Re: Slow reading output from Open3.popen3??

On Wed, May 15, 2013 at 5:11 AM, Jeremy Bopp <[e-mail]> wrote:

> On 05/14/2013 05:38 PM, Thomas Luedeke wrote:
> > I've never tried using this function before, but I am now using as part
> > of a code compilation build script (using 'make' on UNIX).
> >
> > It looks something like this:
> >
> > ==================================
> >
> > stdin, stdout, stderr = Open3.popen3( 'make codename' )
> > stdin.close
> >
> > puts "Reading STDOUT"
> > outfile.puts stdout.read
> > stdout.close
> >
> > ==================================
> >
> > I'm trying to get the output from the 'make' command (which is quite
> > large), so I can dump it to a file for later diagnostics.
>
> There are a couple ways to tackle what you want to do, but one big
> problem with what you're doing here is that you're trying to read *all*
> of the output from stdout into memory at once.  I'm not sure how big
> "quite large" is, but you should probably do this line by line or in
> limited block sizes instead.
>
> stdout.each_line do |line|
>   outfile.write(line)
> end
>

If you use #each_line then I would also use #puts for output because both
are line oriented.


> OR
>
> while (data = stdout.read(1024)).size > 0
>   outfile.write(data)
> end
>

That doesn't work because #read returns nil at EOF.  You just need

while data = stdout.read(1024)
  outfile.write(data)
end

OR, a tad more efficient

data = ""

while stdout.read(1024, data)
  outfile.write(data)
end


> > However, when it gets to the stdout.read, it seems to hang (or at least
> > take so long I've never seen it finish).
> >
> > Any thoughts on why this is happening?  Is there a better approach for
> > me to try?
>
> You never read from stderr in your example.  I'm wondering if the
> subprocess isn't actually blocking while trying to write something to
> stderr.  Because you never read from it, the buffer for the pipe
> connected to stderr may fill up.  If that were to happen while the
> subprocess was writing to stderr, the subprocess would block while
> trying to write and never get to close its end of the stdout pipe and
> exit.  Thus your read from stdout will block forever.
>

Right!

You need to read from both stdout and stderr in your script to avoid
> this problem; however, this can get complicated since you need to avoid
> blocking while reading from either one of them.  You could also skip
> stderr entirely and just use IO.popen, in which case the stderr output
> will go to the terminal or wherever your script's stderr was wired to
> go.  Finally, you could use IO.popen while redirecting the subprocess'
> stderr to its stdout so that you can read both streams at once:
>
> IO.popen('make codename 2>&1', 'r') do |pipe|
>   pipe.each_line do |line|
>     outfile.write(line)
>   end
> end
>

I wouldn't want to do that because then I cannot differenciate between
regular and error output.


> Given your stated goal of sending the output of make to a file for later
> diagnostics, you would probably be better off simply redirecting the
> output directly to a file and skipping ruby entirely:
>
> bash$ make codename >/path/to/make.log 2>&1
>

Agree.


> The above when run in the bash shell would send both the stdout and
> stderr into the file /path/to/make.log.  If you still wanted to see the
> output on screen at the same time as logging to the file, you can use
> the tee program:
>
> bash$ make codename 2>&1 | tee /path/to/make.log
>
> The same thing happens here as before but you'll also see the output in
> the terminal while make runs.


And here's a solution using open3:

require 'open3'

Open3.popen3( %w{make codename} ) do |stdin, stdout, stderr, t|
  stdin.close
  err_thr = Thread.new { IO.copy_stream(stderr, outfile) }
  puts "Reading STDOUT"
  IO.copy_stream(stdout, outfile)
  err_thr.join
end

Note: I also used an Array for the make command invocation because that
avoids parsing issues in the shell because it omits the shell altogether
(see Process.spawn).

You might also prefer a more line based approach

def copy_lines(str_in, str_out)
  str_in.each_line {|line| str_out.puts line}
end

Open3.popen3( 'make codename' ) do |stdin, stdout, stderr, t|
  stdin.close
  err_thr = Thread.new { copy_lines(stderr, $stderr) }
  puts "Reading STDOUT"
  copy_lines(stdout, $stdout)
  err_thr.join
end

Kind regards

robert

-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/