Re: thread/fork/tk nightmares

On Thu, 25 Mar 2004, Xavier wrote:

> Nice idea IMHO.
> Funny, I had to write something similar yesterday to get rid of open3
> warnings that were triggered when publishing my objects with drb because
> of its forks. This is what I did. It is much simpler as I do not need to
> read stdout while the spawned process runs, nor write to its stdin. I only
> need to get stderr and stdout separately.
> 
> def runrun(rcmd)
>   # Get a temp file
>   tmp = Tempfile.new("runrun")
>   # Redirect stderr into temp file
>   po = IO.popen("#{rcmd} 2>#{tmp.path}")
>   stdo = po.readlines
>   po.close
>   # Reopen temp file and read STDERR
>   tmp.open
>   stdr = tmp.readlines
>   # Close and unlink temp file
>   tmp.close!
>   # return STDOUT and STDERR
>   [stdo, stdr]
> end
> 
> 
> Regards,
> Xavier.

i also saw that the ruby-dev suumary 23175-23213 by Masayoshi Takahashi spoke
of related issues...

  ...
  [ruby-dev:23212] warning: fork terminates thread

    Tanaka Akira proposed to suppress "fork terminates thread"
    warnings on using folk with thread as follows:

      % ruby -ve 'Thread.new { sleep }; pid = fork {}; Process.wait pid'
      ruby 1.9.0 (2004-03-22) [i686-linux]
      -e:1: warning: fork terminates thread at -e:1

    Rationale for the proposal is:

      * If the warning is for chang of specification of fork in
        Ruby 1.8, it should be suppressed on Ruby 1.9.

      * fork must be used when system and popen are inadequant.
        In such a case, the warning is unsolicited.

      * Even if the user doesn't use thread explicitly, some libraries
        such like timeout use thread implicitly.

    Matz agreed to do it.
  ...

my situation is that i have a long running peice of code that i need to start
in a thread (so the rest of a tk app can continue)... the code drives about 30
minutes of external processes for which i need the stdout and stderr.  this is
made more difficult by the fact that i need to send multiple commands to the
external processes depending on their outputs (sort of expect like) and thus
cannot close the stdin of any of the processes...  i've tried using select,
threads doing normal reads, etc - but any combination seems to result in the
thread getting all the output from the processes at once - not bit by bit.
this is obviously very undsirable for a gui which is supposed to be monitoring
the progress of these processes... starting the external processes in the
background via system and redirecting their stdin, stdout, stderr to temporary
fifos eliminates the fork and my problems...  this little program, although
contrived, illustrates the problem:


if it uses open3 the output comes all at once, not incrementally

  ~/eg/ruby > spawn.rb open3
  o @ 1080227752.76307: "42\n42\n"

if it uses spawn the output comes incrementally

  ~/eg/ruby > spawn.rb spawn
  o @ 1080227758.40758: "42\n"
  o @ 1080227760.00291: "42\n"


the code (sorry for length)


----
#!/usr/bin/env ruby
require 'open3'
require 'tmpdir'
require 'io/wait'

$VERBOSE=nil

module Spawn
  class << self
    def spawn cmd
      ipath, opath, epath = tmpfifo, tmpfifo, tmpfifo
      system "#{ cmd } < #{ ipath } 1> #{ opath } 2> #{ epath } &"
      [open(ipath, 'w'), open(opath, 'r'), open(epath, 'r')]
    end
    def tmpfifo
      path = nil
      42.times do |i|
        tpath = File.join(Dir.tmpdir, "#{ $$ }.#{ rand }.#{ i }")
        system "mkfifo #{ tpath }"
        next unless $? == 0
        path = tpath
        at_exit{ File.unlink(path) rescue STDERR.puts("rm <#{ path }> failed") }
        break
      end
      raise "could not generate tmpfifo" unless path
      path
    end
  end
end

mode = ARGV.shift || 'open3'

program = <<-stmts
  echo 42
  sleep 1
  echo 42
  sleep 1
  echo __done__         # signal end of stdout
  echo __done__ 1>&2    # signal end of stderr
stmts

done = %r/^\s*__done__\s*$/o

thread =
  Thread.new do
    i,o,e = (mode =~ /open3/io ?  Open3::popen3('sh') : Spawn::spawn('sh'))

    program.each{|stmt| i.puts stmt}
    i.flush

    rfds = [o,e]

    loop do
      break if rfds.empty?
      rs, = select rfds, nil, nil
      rs.each do |r|
        rfds.delete(r) and next if r.eof?
        line = ''
        line << r.getc while r.ready?
        rfds.delete(r) and line[done]='' if line =~ done
        next if line.empty?
        printf "%s @ %s: %s\n", (r == o ? 'o' : 'e'), Time.now.to_f, line.inspect
      end
    end
  end

thread.join
----



-a
-- 
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: [phone]
| ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
| URL     :: http://www.ngdc.noaa.gov/stp/
| TRY     :: for l in ruby perl;do $l -e "print \"\x3a\x2d\x29\x0a\"";done 
===============================================================================