From: Ashley Moran <work@...>
Date: 2006-08-30T16:34:04+09:00
Subject: Re: Evading the limit of a pipe's standard input

On Tuesday 29 August 2006 21:06, Gennady Bystritsky wrote:
> Make sure the command you spawn in IO.popen is actually reading out
> stuff from its stdin in parallel. The limit a pipe have on the system
> level is the amount of the unread data it can hold. When the data is
> read out, it makes room for more data. However, if your reader is stuck
> for some reason, the writer will wait too.
>
> If you say that psql accepts only limited amount of data from its stdin,
> it is more of its problem, rather than pipe's. To check it, try to feed
> the big file to psql via redirection, like in:
>
> psql -U test test < "your big file"
>
> Hope I am not too off from what you meant,
> Gennady.

Hi Gennady

I was not very clear in my e-mail, I didn't think psql was limited in the size 
of file it can read.  You were right it was just the buffer filling up.  I 
had actually tried reading from the buffer but not in parallel.  This seems 
to work:

   MAX_SUCCESSFUL_TIMES = 3047

   query = ["BEGIN WORK;"]
   (MAX_SUCCESSFUL_TIMES + 1).times do
     query << "INSERT INTO people (name) VALUES ('Fred');"
   end
   query << "COMMIT;"

   IO.popen("psql -U test test","r+") do |pipe|
     Thread.new { loop { pipe.read } }                # fixes it
     query.each { |statement| pipe.puts statement }
   end

I assume pipe.read blocks, so the thread doesn't turn the CPU into a heating 
element.

Cheers
Ashley

-- 
"If you do it the stupid way, you will have to do it again"
  - Gregory Chudnovsky