From: Michael Malone <michael.malone@...>
Date: 2009-03-17T08:03:43+09:00
Subject: Re: does IO.read block?

Robert Klemme wrote:
> On 16.03.2009 21:00, Michael Malone wrote:
>>> One problem that could hit you is that you might not get the 
>>> concurrency right: you need to make sure that stderr and stdout are 
>>> read concurrently (if there can be output on both) because if the 
>>> pipe on one of them fills up your child process is blocked.  Maybe 
>>> your issue lies in that area.
>>>
>> I have set STDOUT.sync = true  Is that enough?
>
> No, definitively not!  This covers the *sending* side but the locking 
> issue I described is caused by the receiving side not reading all the 
> data.  Assume an application as a child process that frequently writes 
> to stdout and stderr.  The parent process at the other end of those 
> two (!) pipes only reads the stdout pipe.  Now what happens is this: 
> once the pipe's buffer (OS and configuration dependent, often 4k) 
> fills up the write call blocks and the child hangs - indefinitely.
>
>>  It should be, because I don't have any output originating from the 
>> child processes or any child threads, only the parent/calling 
>> thread.  But I've been wrong before...
>>
>> Though I think I have narrowed the problem _very_ slightly.  The 
>> thread/process pair that blocks indefinitely blocks on the 
>> read_end.read() call.  Which is weird and annoying.
>
> And you are sure you close stdout in the client or the client terminates?
        pw = IO::pipe
        pr = IO::pipe
        pe = IO::pipe
        read_end,write_end = IO.pipe
        pid = fork() do
           begin
              pw[1].close
              STDIN.reopen(pw[0])
              pw[0].close

              pr[0].close
              STDOUT.reopen(pr[1])
              pr[1].close

              pe[0].close
              STDERR.reopen(pe[1])
              pe[1].close

That's the code I used to make sure STDOUT, STDIN and STDERR weren't 
causing full buffer hangs.  I borrowed it from a process detach method 
elsewhere, so it might not be exactly what I want.  However, it had no 
effect.  The problem still exists.

Righto, here's the deal:  Either the fork call doesn't work (but doesn't 
fail either -> a valid pid is returned)
if I do a puts either side of the fork() (with and without the STDOUT 
re-direction) then on the process that blocks, the puts just inside the 
fork() doesn't run.  This indicates that the fork() call fails, however 
it doesn't return nil like the docs say. 

An interesting effect I noticed, is if I remove my call to 
Process.wait(pid) then it displays almost identical behaviour!  It runs 
perfectly some times, and occasionally just hangs.  However, if I'm not 
collecting child processes, then I would expect to hit my ulimit at the 
same place every time and for the fork() call to fail noisily.  The docs 
don't mention anything on this.  Is it possible that my child processes 
aren't being collected correctly, thus the rlimit is being hit, but Ruby 
*believes* the process is collected, so its internal rlimit is not 
reached, so it calls fork() regardless?  Is that at all sane?  Though 
presumably, the external fork() call *would* fail, causing the internal 
call to fail.  And yes, I'm speaking as if the two are different, but 
they might not be.  I suppose it doesn't really make sense to duplicate 
it.  In fact, the call directly after the block passed to fork() doesn't 
execute.  Is it possible that my Thread is hanging on the fork() call?  
What would that indicate?  How could I fix it?

No straw will be left un-grasped at!

Michael

=======================================================================
This email, including any attachments, is only for the intended
addressee.  It is subject to copyright, is confidential and may be
the subject of legal or other privilege, none of which is waived or
lost by reason of this transmission.
If the receiver is not the intended addressee, please accept our
apologies, notify us by return, delete all copies and perform no
other act on the email.
Unfortunately, we cannot warrant that the email has not been
 altered or corrupted during transmission.
=======================================================================