From: Robert Citek <robert.citek@...>
Date: 2009-11-07T04:15:15+09:00
Subject: Re: reading from an external process with IO.popen

On Fri, Nov 6, 2009 at 12:50 PM, Robert Klemme
<shortcutter@googlemail.com> wrote:
> On 11/06/2009 04:01 PM, Robert Citek wrote:
>> BTW, the cat example works as expected, but the using sed doesn't
>> work.  That is, there is no output from sed until the pipe closes.
>> There seems to be some buffering going on.  I'm guessing it's from the
>> Ruby side since I don't see this when run from the shell.  But that's
>> just a guess.
>
> The shell closes the pipe as well.  It is sed that is doing the buffering
> and you have no control over it unless it provides an option to control
> this.

Yes, it appears that the external program is controlling the
buffering.  When I tried the same process with the program I really
wanted to use, IO.popen worked pretty much the way I wanted it to.
The pattern was this:

foo = io.popen("external_program", "w+")
while data = gets
  prepare data
  foo.puts(data)
  while not end of record
    newdata += foo.readlines
  end
  process newdata
end
foo.close

Turns out that the program I used has a signal to signify the end of a
chunk of data.  So the program knows when I am finished sending data
and it can start crunching away.  And I know when I can stop reading
data from the pipe and begin processing it.  This saves the time of
repeatedly having to open and close the pipe.

>> Of course, it's entirely possible that IO.popen is not the "right" way
>> to tackle this and I have not discovered the Ruby way, yet.
>
> No, it's the right way but your expectations cannot be met in all cases.

It's nice to know that I'm at least on the right track, or on one of
many possible right tracks.

Thanks for your help.

Regards,
- Robert