From: Robert Klemme Date: 2013-05-16T01:16:59+09:00 Subject: Re: Slow reading output from Open3.popen3?? --20cf30363d3f36395204dcc41484 Content-Type: text/plain; charset=ISO-8859-1 On Wed, May 15, 2013 at 5:11 AM, Jeremy Bopp wrote: > On 05/14/2013 05:38 PM, Thomas Luedeke wrote: > > I've never tried using this function before, but I am now using as part > > of a code compilation build script (using 'make' on UNIX). > > > > It looks something like this: > > > > ================================== > > > > stdin, stdout, stderr = Open3.popen3( 'make codename' ) > > stdin.close > > > > puts "Reading STDOUT" > > outfile.puts stdout.read > > stdout.close > > > > ================================== > > > > I'm trying to get the output from the 'make' command (which is quite > > large), so I can dump it to a file for later diagnostics. > > There are a couple ways to tackle what you want to do, but one big > problem with what you're doing here is that you're trying to read *all* > of the output from stdout into memory at once. I'm not sure how big > "quite large" is, but you should probably do this line by line or in > limited block sizes instead. > > stdout.each_line do |line| > outfile.write(line) > end > If you use #each_line then I would also use #puts for output because both are line oriented. > OR > > while (data = stdout.read(1024)).size > 0 > outfile.write(data) > end > That doesn't work because #read returns nil at EOF. You just need while data = stdout.read(1024) outfile.write(data) end OR, a tad more efficient data = "" while stdout.read(1024, data) outfile.write(data) end > > However, when it gets to the stdout.read, it seems to hang (or at least > > take so long I've never seen it finish). > > > > Any thoughts on why this is happening? Is there a better approach for > > me to try? > > You never read from stderr in your example. I'm wondering if the > subprocess isn't actually blocking while trying to write something to > stderr. Because you never read from it, the buffer for the pipe > connected to stderr may fill up. If that were to happen while the > subprocess was writing to stderr, the subprocess would block while > trying to write and never get to close its end of the stdout pipe and > exit. Thus your read from stdout will block forever. > Right! You need to read from both stdout and stderr in your script to avoid > this problem; however, this can get complicated since you need to avoid > blocking while reading from either one of them. You could also skip > stderr entirely and just use IO.popen, in which case the stderr output > will go to the terminal or wherever your script's stderr was wired to > go. Finally, you could use IO.popen while redirecting the subprocess' > stderr to its stdout so that you can read both streams at once: > > IO.popen('make codename 2>&1', 'r') do |pipe| > pipe.each_line do |line| > outfile.write(line) > end > end > I wouldn't want to do that because then I cannot differenciate between regular and error output. > Given your stated goal of sending the output of make to a file for later > diagnostics, you would probably be better off simply redirecting the > output directly to a file and skipping ruby entirely: > > bash$ make codename >/path/to/make.log 2>&1 > Agree. > The above when run in the bash shell would send both the stdout and > stderr into the file /path/to/make.log. If you still wanted to see the > output on screen at the same time as logging to the file, you can use > the tee program: > > bash$ make codename 2>&1 | tee /path/to/make.log > > The same thing happens here as before but you'll also see the output in > the terminal while make runs. And here's a solution using open3: require 'open3' Open3.popen3( %w{make codename} ) do |stdin, stdout, stderr, t| stdin.close err_thr = Thread.new { IO.copy_stream(stderr, outfile) } puts "Reading STDOUT" IO.copy_stream(stdout, outfile) err_thr.join end Note: I also used an Array for the make command invocation because that avoids parsing issues in the shell because it omits the shell altogether (see Process.spawn). You might also prefer a more line based approach def copy_lines(str_in, str_out) str_in.each_line {|line| str_out.puts line} end Open3.popen3( 'make codename' ) do |stdin, stdout, stderr, t| stdin.close err_thr = Thread.new { copy_lines(stderr, $stderr) } puts "Reading STDOUT" copy_lines(stdout, $stdout) err_thr.join end Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/ --20cf30363d3f36395204dcc41484 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Wed, May 15, 2013 at 5:11 AM, Jeremy Bopp <= jeremy@bopp.net>= ; wrote:
On 05/14/2013 05:38 PM, Thomas Luedeke w= rote:
> I've never tried using this function before, but I am now using as= part
> of a code compilation build script (using 'make' on UNIX).
>
> It looks something like this:
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> stdin, stdout, stderr =3D Open3.popen3( 'make codename' )
> stdin.close
>
> puts "Reading STDOUT"
> outfile.puts stdout.read
> stdout.close
>
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> I'm trying to get the output from the 'make' command (whic= h is quite
> large), so I can dump it to a file for later diagnostics.

There are a couple ways to tackle what you want to do, but one big problem with what you're doing here is that you're trying to read *= all*
of the output from stdout into memory at once. =A0I'm not sure how big<= br> "quite large" is, but you should probably do this line by line or= in
limited block sizes instead.

stdout.each_line do |line|
=A0 outfile.write(line)
end

If you use #each_line then I = would also use #puts for output because both are line oriented.
=A0
OR

while (data =3D stdout.read(1024)).size > 0
=A0 outfile.write(data)
end

That doesn't work because= #read returns nil at EOF. =A0You just need

while data =3D stdout.read(1024)
=A0 outfile.write(da= ta)
end

OR, a tad more efficien= t

data =3D ""

while stdout.read(1024, data)
=A0 out= file.write(data)
end
=A0
> However, when it gets to the stdout.read, it seems to hang (or at leas= t
> take so long I've never seen it finish).
>
> Any thoughts on why this is happening? =A0Is there a better approach f= or
> me to try?

You never read from stderr in your example. =A0I'm wondering if t= he
subprocess isn't actually blocking while trying to write something to stderr. =A0Because you never read from it, the buffer for the pipe
connected to stderr may fill up. =A0If that were to happen while the
subprocess was writing to stderr, the subprocess would block while
trying to write and never get to close its end of the stdout pipe and
exit. =A0Thus your read from stdout will block forever.

Right!=A0

You need to read from both stdout and stderr in your script to avoid
this problem; however, this can get complicated since you need to avoid
blocking while reading from either one of them. =A0You could also skip
stderr entirely and just use IO.popen, in which case the stderr output
will go to the terminal or wherever your script's stderr was wired to go. =A0Finally, you could use IO.popen while redirecting the subprocess'= ;
stderr to its stdout so that you can read both streams at once:

IO.popen('make codename 2>&1', 'r') do |pipe|
=A0 pipe.each_line do |line|
=A0 =A0 outfile.write(line)
=A0 end
end

I wouldn't want to do tha= t because then I cannot differenciate between regular and error output.
=A0
Given your stated goal of sending the output of make to a file for later diagnostics, you would probably be better off simply redirecting the
output directly to a file and skipping ruby entirely:

bash$ make codename >/path/to/make.log 2>&1
=
Agree.
=A0
The above when run in the bash shell would send both the stdout and
stderr into the file /path/to/make.log. =A0If you still wanted to see the output on screen at the same time as logging to the file, you can use
the tee program:

bash$ make codename 2>&1 | tee /path/to/make.log

The same thing happens here as before but you'll also see the output in=
the terminal while make runs.

And her= e's a solution using open3:

require '= ;open3'

Open3.popen3( %w{make codename} ) do |= stdin, stdout, stderr, t|
=A0 stdin.close
=A0 err_thr =3D Thread.new { IO.copy_stream(= stderr, outfile) }
=A0 puts "Reading STDOUT"
= =A0 IO.copy_stream(stdout, outfile)
=A0 err_thr.join
en= d
=A0
Note: I also used an Array for the make command invocation b= ecause that avoids parsing issues in the shell because it omits the shell a= ltogether (see Process.spawn).

You might also prefer a more line based approach

def copy_lines(str_in, str_out)
=A0 = str_in.each_line {|line| str_out.puts line}
end

Open3.popen3( 'make codename' ) do |stdin,= stdout, stderr, t|
=A0 stdin.close
=A0 err_thr =3D Thread.new { copy_lines(stderr, $stderr) }
=A0 puts "Reading STDOUT"
=A0=A0copy_lines(stdout, $stdout)
=A0 err_thr.join
end

Kind regards

robert

--
remember.guy= do |as, often| as.you_can - without end
http://blog.rubybestpractice= s.com/
--20cf30363d3f36395204dcc41484--