From: Brad Volz <bradv@...>
Date: 2008-09-13T15:45:32+09:00
Subject: how can i find lingering file descriptors?

Hello,

I seem to have an issue with file descriptors that aren't being closed  
when I attempt to put some parallelization into one of my scripts.  I  
am trying to make use of the forkoff gem, but I guess that I am not  
using it correctly.

If it's useful, here is what the code looks like currently:

     def measure_n ( start_time, stop_time, direction, pattern,  
records )
       counters = [ :flows, :packets, :octets ]

       # single process
       stats = Hash.new
       counters.each { |c| stats[c] = 0 }

       puts "starting single process execution"
       single_start = Time.now.to_f
       pattern.each do |p|
         r = self.measure_1( start_time, stop_time, direction,  
"#{direction} AS #{p.to_s}", records)
         counters.each { |c| stats[c] += r[c] }
       end
       single_stop = Time.now.to_f
       puts stats.inspect
       puts "single exeution time: #{single_stop - single_start}"

       # multiple processes
       stats = Hash.new
       counters.each { |c| stats[c] = 0 }

       puts "starting multi process execution"
       multi_start = Time.now.to_f
       asn_stats = pattern.forkoff! :processes => 4 do |asn|
         a = Netflow::Nfdump.new
         a.measure_1(start_time,stop_time,direction,"#{direction} AS  
#{asn}",records)
       end
       asn_stats.each do |asn|
         counters.each { |c| stats[c] += asn[c] }
       end

       multi_stop = Time.now.to_f
       puts stats.inspect
       puts "multi execution time: #{multi_stop - multi_start}"

       return stats
     end


The part that I find really odd, is that I can replace the block that  
I am attempting to parallelize with a simple:-

	puts "#{asn}"

And the script dies in the same place -- at the 255'th element in the  
array, which corresponds well to the number of file descriptors that I  
can use:-

	bradv:bvolz:$ ulimit -a | grep files
	open files                      (-n) 256

Since I see this problem with a simple 'puts' does that mean that the  
issue is not in my code, and perhaps lies elsewhere?  Or have I  
misunderstood how to make use of the forkoff gem?  In either case, how  
can I figure out what these open file descriptors are?

Thanks,

Brad