From: Robert Klemme <bob.news@...>
Date: 2006-03-06T18:48:40+09:00
Subject: Re: Piping binary data to an external program

|MKSM| wrote:
> Hello,
>
> I'm working on a logparser and i've run into some issues. It will
> parse OpenBSD PF logs. They are tcpdump format logs and BSD normally
> compress them.
>
> Here is the usage I have in mind:
>
> "gzip -cd log.gz | ruby logparser.rb --today"
>
> I have the following code:
>
> Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
> in_io.write($stdin.read)
> in_io.close
> $log = out_io.read
> }
>
> The script freezes on the open3 line and doesn't continue. I've tested
> several other methods but it doesn't seem to work.
>
> Any suggestions on how this can be done?

It's likely that you run into a deadlock caused by pipe buffer sizes.  I
suggest to not read and write the whole content but to do it in chunks.
Also, I'd separate the reading and writing code into two threads.

Untested:

Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
t = Thread.new(in_io) do |out|
  while ( buff = $stdin.read( 1024 ) )
    out.write(buff)
  end
  out.close
end
$log = ""
while ( b = out_io.read(1024))
  $log << b
end
}


HTH

Kind regards

    robert