From: Robert Klemme <shortcutter@...>
Date: 2006-11-02T07:10:15+09:00
Subject: Re: Nonblocking IO read

Tom Pollard wrote:
> 
> On Nov 1, 2006, at 11:30 AM, Robert Klemme wrote:
>> Tom Pollard wrote:
>>> Anyway, you would only need nonblocking IO if you wanted to read bits 
>>> of the stderr stream before the command exited, but that doesn't 
>>> sound like what you're want.
>>
>> Actually this is not correct: if there is a lot written to stderr then 
>> you need to read that concurrently.  If you do not do that then the 
>> process will block on some stderr write operation that fills up the 
>> pipe and you get a deadlock because your code waits for process 
>> termination.
> 
> I guess I can see that, though I can't think of a program that I'd 
> expect to be able to generate enough stderr output to clog a pipe.

A typical pipe buffer size is 4k which can get filled pretty fast.

 >  In
> any case, my response would be to merge stdout and stderr, rather than 
> use non-blocking IO.  If you're just reading one stream while the 
> command is executing, you don't need to worry about blocking.

Merging both from outside the subprocess is certainly possible from Ruby 
(via a shell) but I am not sure, whether there is a portable solution 
(one of the popenN methods?).

 >  I'm
> certainly with Ara in recommending that if you can avoid non-blocking 
> IO, you should.

I second that.

> At the risk of starting an unrelated discussion ("stderr considered 
> harmful"), my feeling has long been that stderr is misused by most 
> people, and that the only context in which it makes any sense is for 
> small commandline tools that you expect to use in a pipeline.  For apps 
> like that, it's helpful to keep error messages out of your stdout 
> stream.  For most apps, however, I don't think it makes any sense to 
> write error messages to a separate file.  Error messages should be 
> written to the app's main log file or output file, where the user will 
> be looking for their results.  That way, nonfatal error messages also 
> appear naturally in the proper sequence with other output.  I work with 
> a lot of scientist programmers who don't think much about issues like 
> this and (typically) write their error messages to stderr just because 
> it's there.  I'm not sure that's relevant to the OP's situation or not.  
> (Probably not.)

All true.  But if you do not know what the program does or how it is 
implemented you better deal with potential output to stderr (either by 
merging, see above, or by making sure that stderr and stdout are read) 
because otherwise the consequences might be somewhat catastrophic.  And 
this could also mean that your program at some point in the future 
simply stops working because another piece of software has changed.

Kind regards

	robert