From: John Stoffel Date: 2009-02-14T07:04:35+09:00 Subject: DRb::DRbBadScheme when using drbunix sockets, why? Hi, I've been slowly hacking on my parallel recursive readdir() implementation using a limited number of parallel processes, since as you all know, Ruby threads are not truly parallel. Anyway, I'm now getting wierd errors like the following: Starting DRb on server Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 counter.count=1 And it's not really clear to my *why* I'm getting these errors, or how I can trap them so I'm not bothered by them. The basic code uses both DRb and Slave libraries to make the interprocess communication simpler, and to have a central counting server to regulate the number of sub-processes I'll be using at any one point in time. So, first off, here's a full set of output of my code on a test directory. Thanks for any hints or suggestions. I *think* I need to use begin ... rescue ... end blocks possibly, but it's not clear where. Oh yeah, I'm running this all on a CentOS 5.2 (RHEL5.2) Final system, against a small 350Mb test directory tree. Thanks, John ------------------------------- log ---------------------------------- $ ./readdir-drb.rb tmp Starting DRb on server Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/stoffj-test_17986 counter.count=1 Threaded! Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Threaded! Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 Exception `DRb::DRbBadScheme' at /usr/lib/ruby/1.8/drb/drb.rb:814 - drbunix:///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 size = 372104960 size = 373477128 Exception `NameError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:409 - uninitialized constant Slave::DBb Exception `RuntimeError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:519 - already shutdown exit (SystemExit) /usr/lib/ruby/site_ruby/1.8/slave.rb:265:in `exit' /usr/lib/ruby/site_ruby/1.8/slave.rb:265:in `cling' /usr/lib/ruby/site_ruby/1.8/slave.rb:259:in `call' /usr/lib/ruby/site_ruby/1.8/slave.rb:259:in `on_cut' /usr/lib/ruby/site_ruby/1.8/slave.rb:251:in `initialize' /usr/lib/ruby/site_ruby/1.8/slave.rb:251:in `new' /usr/lib/ruby/site_ruby/1.8/slave.rb:251:in `on_cut' /usr/lib/ruby/site_ruby/1.8/slave.rb:265:in `cling' /usr/lib/ruby/site_ruby/1.8/slave.rb:415:in `initialize' /usr/lib/ruby/site_ruby/1.8/slave.rb:593:in `new' /usr/lib/ruby/site_ruby/1.8/slave.rb:593:in `object' ./readdir-drb.rb:114:in `readdir' ./readdir-drb.rb:86:in `foreach' ./readdir-drb.rb:86:in `readdir' ./readdir-drb.rb:207 Total size: 373477128 B Total size: 364723 KB Total size: 356 MB Total size: 0 GB Exception `Errno::ENOENT' at /usr/lib/ruby/1.8/fileutils.rb:1281 - No such file or directory - /tmp/slave_proc_537506070_17986_17988_0_0.125356081811465 Exception `RuntimeError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:519 - already shutdown rt3.taec.com:~/src/Tools/philesight-20081120$ Exception `NameError' at /usr/lib/ruby/site_ruby/1.8/slave.rb:409 - uninitialized constant Slave::DBb Exception `Errno::ENOENT' at /usr/lib/ruby/1.8/fileutils.rb:1281 - No such file or directory - /tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 Exception `IOError' at /usr/lib/ruby/1.8/drb/unix.rb:92 - stream closed Exception `Errno::EBADF' at /usr/lib/ruby/1.8/drb/unix.rb:92 - Bad file descriptor - ///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 /usr/lib/ruby/1.8/drb/unix.rb:92:in `close': Bad file descriptor - ///tmp/slave_proc_537512250_17988_17989_0_0.20115187038118 (Errno::EBADF) from /usr/lib/ruby/1.8/drb/unix.rb:92:in `close' from /usr/lib/ruby/1.8/drb/drb.rb:1433:in `run' from /usr/lib/ruby/1.8/drb/drb.rb:1427:in `start' from /usr/lib/ruby/1.8/drb/drb.rb:1427:in `run' from /usr/lib/ruby/1.8/drb/drb.rb:1347:in `initialize' from /usr/lib/ruby/1.8/drb/drb.rb:1627:in `new' from /usr/lib/ruby/1.8/drb/drb.rb:1627:in `start_service' from /usr/lib/ruby/site_ruby/1.8/slave.rb:396:in `initialize' ... 28 levels... from ./readdir-drb.rb:114:in `readdir' from ./readdir-drb.rb:86:in `foreach' from ./readdir-drb.rb:86:in `readdir' from ./readdir-drb.rb:207 --------------------------- source -------------------------------------- And here's my source code, excuse my lack of Ruby knowledge, I'm a newb to Ruby, though not to programming. Hopefully that is shown. :] #!/usr/bin/ruby require 'getoptlong' require 'thread' require 'slave' require 'drb' require 'drb/unix' $VERSION = "v1.0"; $max_slaves = 3 $count = 50 # Local Socket - need to worry about multiple runs of this script, so add on # unique PID. $URI = "drbunix:///tmp/stoffj-test_" + Process.pid.to_s slave_cnt_mutex = Mutex.new opts = GetoptLong.new( [ "--help", "-h", GetoptLong::NO_ARGUMENT], [ "--kids", "-k", GetoptLong::REQUIRED_ARGUMENT] ) #--------------------------------------------------------------------- class Counter def initialize(max=1) @slaves = [] @max = max @count = 1 @count_mutex = Mutex.new end def count @count end def max @max end # Increment the count of slaves, returning 1 if incremented, 0 if not. def increment ok = nil @count_mutex.synchronize do if (@count < @max) then @count += 1 ok = 1 end end ok end # Decrement the count of slaves def decrement @count_mutex.synchronize do if (@count > 1) then @count -= 1 end end @count end end #--------------------------------------------------------------------- class ReadDir def initialize(server) @server = server # Slave pool for this level of readdir() recursion. @kids = [] end def readdir(dir) #puts "readdir(#{dir})" size_file = {} size_dir = {} size_total = 0 # Traverse the directory and collect the size of all files and # directories begin Dir.foreach(dir) do |f| #print " #{f}," if(f != "." && f != "..") then f_full = addpath(dir, f) stat = File.lstat(f_full) if(!stat.symlink?) then if(stat.file?) then #puts " File: #{f}" size = File.size(f_full) size_file[f] = size size_total += size end if(stat.directory?) then #puts "DIR= #{f}" if (@server.max <= 1) then puts " no threads." size = readdir(f_full) if (size > 0) then size_dir[f] = size size_total += size end else ok = @server.increment if (ok) puts " Threaded!" @kids << Slave.object(:async => true) { size = readdir(f_full) puts "size = #{size}" # Duh... return the size from the slave properly size } else #puts " no free threads, do anyway" size = readdir(f_full) if(size > 0) then size_dir[f] = size size_total += size end end end end end end end end @kids.each { |kid| size_total += kid.value } #Puts "Dir: #{dir} = #{size_total}" return size_total end end #--------------------------------------------------------------------- # Read a directory and add to the database; this function is recursive # for sub-directories #--------------------------------------------------------------------- def usage puts puts "usage: readdir-drb [--kids NUM] " puts " defaults to #{$max_kids} children" puts puts " version: #{$version}" puts end #--------------------------------------------------------------------- def addpath(a, b) return a + b if(a =~ /\/$/) return a + "/" + b end #--------------------------------------------------------------------- # Main #--------------------------------------------------------------------- $DEBUG = true opts.each do |opt,arg| case opt when "--kids" $max_slaves = arg.to_i else usage exit end end if ARGV.length != 1 puts "Missing dir argument (try --help)" exit 0 end dir = ARGV.shift # Start the DRb service. puts "Starting DRb on server" DRb.start_service $URI, Counter.new($max_slaves) # Child # Fire up the first slave process which will do the work of readdir() DRb.start_service counter = DRbObject.new_with_uri $URI puts "counter.count=#{counter.count}" # Fire up a new Kid Class readdir. kid = ReadDir.new(counter) # Now let's try to do a recursive readdir() algorith with threads. size = kid.readdir(dir) sizekb = size / 1024; sizemb = sizekb / 1024; sizegb = sizemb / 1024; puts "" puts "Total size: #{size} B" puts "Total size: #{sizekb} KB" puts "Total size: #{sizemb} MB" puts "Total size: #{sizegb} GB" -- Posted via http://www.ruby-forum.com/.