[ruby-core:19524] Re: Too Many Files Error -- Test Case Produced.

On 26 okt 2008, at 13.54, C.E. Thornton wrote:

> Core,
>
> I have attached a cut-down version of my
> dir scanner.  It simply starts with a single
> directory and scans the entire dir tree listing
> the directories.
>
> 186 -- Runs without error
>          (> 140,000 directories scanned!)
> 190 -- Runs without error           (> 140,000 directories scanned!)
>
> 187 -- (Latest) FAILS after approx 3200 Directories!
>          Fails with a "Too many open files" Message.
>
> Chuck T.
>
> --  
> Competency and chastity have much in common,
> they both encompass their own punishment! -- C.E. Thornton --  
> Hawthorne Press --
>
> #!/usr/bin/ruby
> #
> # Demo File showing problem with Ruby 187 (Latest)
> #
> #
> # Run in a Very Large Dir Tree (at least 5000 directories)
> #
> # Usage:  tmfiles [Initial Dir]   # Start at specified dir
> #         tmfiles                 # Default Directory is "/"
> #
> #
> # I am running on a AMD2300 (X2) using CENTOS 4.5 Linux
> # and 2.0 GB of Memory.
> #
> # Fails between 3000-4000 files (depending on starting dir)
> # for Ruby 187
> #
> # Works without failure on Ruby 190 ( ~ 23,000 Directories)
> #
> #
> #
> d_img  = Array.new                         # Directory Entries
> f_img  = Array.new                         # File Entries
> #
> dir_cnt  = 1
> err_cnt  = 0
> #
> if ARGV[0].nil?
>   cur_dir = "/"
> else
>  cur_dir = (ARGV[0])                        # Load Initial Directory  
> to process
> end
> #
> until cur_dir.nil?
>    begin
>        d = Dir.new(cur_dir)
>        d.each do |x|
>            if x == '.' or x == '..' ; else
>                if cur_dir == "/"
>                    y = cur_dir + x
>                else
>                    y = cur_dir + "/" + x
>                end
>                puts "Directory #{dir_cnt} =  #{y}"
>
>                begin
>                   if File.lstat(y).symlink? then
>                    #  puts "lstat.symlink? True -- Always Handle  
> #{y} as a 'file'"
>                   else
>                       if File.stat(y).directory?
>                            dir_cnt += 1          # Count Dir
>                            d_img.push(y)         # Add dir to Dir List
>                       end
>                   end
>                rescue
>                     $stderr.printf("Status Error Reading File <%s>  
> -- Dir Count = %d\n", y, dir_cnt)
>                end
>            end
>         end
	A d.close here will fix the problem.
The problem is that directory handles are not garbage collected and  
automatically closed often enough.
There are one other alternative:  Use Dir.open(cur_dir) do | d  |
o
>
>     rescue
>         err_cnt += 1
>         $stderr.printf("Status Error Reading Directory <%s> -- Dir  
> Count = %d -- Error Count = %d\n", cur_dir, dir_cnt, err_cnt)
>         if err_cnt > 10
>             puts "Error Limit of 10  Exceeded!"
>             exit
>         end
>     end
> #
>     cur_dir = d_img.shift
> end
> #
> puts "Finished Processing Directories"

------------------------------------------------------
"Home is not where you are born, but where your heart finds peace" -
Tommy Nordgren, "The dying old crone"
[e-mail]