From: "Ara.T.Howard" <Ara.T.Howard@...>
Date: 2005-07-22T01:54:10+09:00
Subject: Re: [BUG] Segmentation fault with a threads/forks script

On Thu, 21 Jul 2005, Lucas Nussbaum wrote:

> Hi,
>
> I experience a reproducable crash (each time) with prunner.rb at
> http://blop.info/bazaar/prunner.rb. The script starts several commands
> at the same time. When running with a large number of commands, it
> exits with :
>
> ./prunner.rb:51: [BUG] Segmentation fault
> ruby 1.8.2 (2005-04-11) [i386-linux]
>
> Aborted
>
> To reproduce, create a large file with one command per line.
> for i in $(seq 1 2000); do echo hostname done > cmds
> Then run prunner.rb like this :
> cat cmds |head -n 1500 |./prunner.rb
> 1500 can be increased if it doesn't crash for you.
>
> Of course, I expect it to go wrong at some time, but it could probably
> do this in a cleaner way.
>
> Can somebody confirm the bug ? Or better, fix it ? :-)
> -- 
> | Lucas Nussbaum
> | lucas@lucas-nussbaum.net   http://www.lucas-nussbaum.net/ |
> | jabber: lucas@nussbaum.fr             GPG: 1024D/023B3F4F |

looks like you can work around it by just closing every thing as you use it:

   harp:~ > curl http://fortytwo.merseine.nu/prunner.rb > prunner.rb

   harp:~ > for i in $(seq 1 2000);do echo 'date'; done > cmds

   harp:~ > wc -l cmds
      2000 cmds

   harp:~ > ruby prunner.rb < cmds > /dev/null

   harp:~ > echo $?
   0

   harp:~ > ls prunner.*out* | wc -l
      2000

   harp:~ > cat prunner.out-.0
   # 0 : date
   Thu Jul 21 10:49:07 MDT 2005

   harp:~ > cat prunner.out-.1999
   # 1999 : date
   Thu Jul 21 10:49:21 MDT 2005

(prunner.rb inlined below)

hth.

-a
-- 
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| My religion is very simple.  My religion is kindness.
| --Tenzin Gyatso
===============================================================================

===============================================================================
file: prunner.rb
===============================================================================
#!/usr/bin/ruby -w

require 'optparse'
require 'thread'

#
# prunner : read commands from stdin, and execute each of them in parallel
#

class Hash
   def getopt k, default = nil
     return self[k] if self.has_key? k
     k = "#{ k }"
     return self[k] if self.has_key? k
     k = k.intern
     return self[k] if self.has_key? k
     return default
   end
end

class Command
   class << self
     def gen_cid
       @cid = defined?(@cid) ? (@cid + 1) : 0
     end
   end

   attr_accessor :command, :cid, :prefix, :path, :exit_status

   def initialize command, opts = {}
     @command = command.strip
     @cid = self.class.gen_cid
     @prefix = opts.getopt 'prefix', "#{ $$ }_command.out-"
     @path = "#{ @prefix }.#{ @cid }"
     @header = opts.getopt 'header'
     @mutex = Mutex::new
     @lines = []
     @update_idx = 0
     @thread = nil
     @exit_status = -1
   end
   def start
     @thread =
       Thread::new(@command, Thread::current) do |cmd, cur|
         begin
           IO::popen("{ #{ cmd } ;} 2>&1") do |pipe|
             File::open(@path, 'w') do |f|
               f << self if @header
               while((line = pipe.gets))
                 synchronize{ @lines << line }
                 f << line
               end
             end
           end
           @exit_status = $?.exitstatus
         rescue Exception => e
           cur.raise e
         end
       end
   end
   def synchronize(*a, &b)
     @mutex.synchronize(*a, &b)
   end
   def join(*a, &b)
     @thread.join(*a, &b)
   end
   def update
     report = nil
     synchronize do
       report = @lines[@update_idx .. -1]
       @update_idx = @lines.size
     end
     report
   end
   def update?
     @update_idx < @lines.size
   end
   def to_s
     "# #{ @cid } : #{ @command }\n"
   end
   alias label to_s
end

class Main
   def initialize env = ENV.to_hash, argv = ARGV.clone
     @env, @argv = env, argv
     @cmds = []
     @header = true
     @verbose = true
     @interval = 1
     @prefix = 'prunner.out-'
     @viewthread = nil
     parse_options
   end
   def parse_options
     OptionParser::new do |opts|
       opts.banner = "echo command | prunner.rb [options]"
       opts.separator ''
       opts.on('-h', '--suppress-header', 'suppress header in output files'){
         @header = false
       }
       opts.on('-q', '--quiet', 'run quietly'){
         @verbose = false
       }
       opts.on('-i', '--interval', 'output interval'){|i|
         @interval = Float i
       }
       opts.on('-p', '--prefix PREFIX', "prefix for output files (default #{ @prefix })"){|p|
         @prefix = p
       }
       opts.on_tail('h', '--help', 'Show this message') {
         puts opts
         exit
       }
       opts.parse!(@argv)
     end
   end
   def main
     STDIN.each do |line|
       line.strip!
       next if line.empty?
       c =
         Command::new(line,
           :verbose => @verbose,
           :header => @header,
           :prefix => @prefix
         )
       c.start
       @cmds << c
     end

     if @verbose
       @viewthread =
         Thread::new do
           loop do
             reports = @cmds.map{|c| [c.label, c.update] if c.update?}.compact
             exit if reports.empty?
             reports.each do |label, report|
               print label
               report.each{|line| print line}
             end
             sleep @interval
           end
         end
     end

     @cmds.each{|c| c.join}
     @viewthread.join if @viewthread
     exit
   end
end

if $0 == __FILE__
   STDOUT.sync = true
   Main::new(ENV, ARGV).main
end