From: Ilmari Heikkinen <kig@...>
Date: 2005-05-14T06:33:18+09:00
Subject: Re: Deal with large inputs in a popen (was Re: Get back data from a child (with exec))


On 13.5.2005, at 23:40, Lawrence Oluyede wrote:

>
> How can I deal with large inputs to an external process
> popen driven? With 5k of data sent to its standard
> input blocks.
>
> -- 
> Lawrence
> http://www.oluyede.org/blog
>

You need to send the data in blocks lequal to the pipe buffer size.
Something like pipe.write(input.read(4096)) until input.eof?

Also, if the external process outputs to its stdout, it will block once 
it
has written pipe buffer bytes, so you need to read from the pipe aswell.

So, one thread for writing, one thread for reading.


Btw, maybe popened IOs should do this automatically under covers?

test_popen.rb

d = ' '*32000
IO.popen('cat','r+'){|c|
   Thread.new{ c.write(d); c.close_write }
   puts c.read.size
}

Here's a way to fix the hang:

class IO
   class << self
     alias_method :real_popen, :popen
     def popen(*args,&block)
       io = real_popen(*args)
       def io.write(data)
         i = wb = 0
         (wb += super(data[i,4096]); i+=4096) while i < data.size
         wb
       end
       if block_given?
         block.call(io)
       else
         io
       end
     end
   end
end