From: Scott Ellsworth <scott@...>
Date: 2005-07-19T09:25:54+09:00
Subject: Re: How do I reduce the memory usage of a script?

In article <3jmmgnFqqsq6U1@individual.net>,
 "Robert Klemme" <bob.news@gmx.net> wrote:

> Scott Ellsworth wrote:
> > In article <8HKdneMVh9MZDUjfSa8jmA@karoo.co.uk>,
> >  "daz" <dooby@d10.karoo.co.uk> wrote:
> >
> >>           if (!found) then
> >>              puts "Unknown  #{f}"
> >>           end
> >>           data = nil
> >>           GC.start   # garbage collect
> >
> > This did seem to drop the memory usage on my MacOS X 10.4.2 system.
> >
> > I will investigate the Find.find command next to see if I can get rid
> > of some recursion.  An array of 5500 paths should not be _that_ big,
> > at least in comparison with four or five levels of directory depth.
> 
> The problem might be that the data is still around while you enter the
> recursion.  If you want to verify that this is the case you can simply do
> data = nil after processing.  But: You definitely need to throw out the
> recursion from propath() - otherwise you'll be processing directories over
> and over again (I smell something like O(n*n) here)!

I have removed the recursion - see below.

A question, though,  Is the String.scan method I used the best way to do 
the scan this block of data?  Every file is going to contain the string 
'QuickTime' somewhere in the first few MB, and I want from the last 
nonprintable character before it to the next nonprintable character 
after.  I only need to read from disk until I find that string, and once 
I find it, I need only the bytes before, plus a version number 
afterwards.  I certainly do not need to manipulate more than a few 
hundred characters around that magic string, and once I have read, I do 
not need to go back.

NB - nonprintable here is defined as [[:alnum:]_., ]

work@boggle:Desktop$ time ./detectEncoding.rb > songs.txt

real    3m30.563s
user    0m26.229s
sys     0m23.746s

New code:

#!/usr/bin/env ruby
require 'find'
re = /[[:alnum:]_., ]{9,}/
Find.find("/Users/work/Music/iTunes/iTunes Music/") do |f|
   if File.file?(f) && File.fnmatch("*.m4a",f) then
      found = false
      data = IO.read(f, 65536*8)
      data.scan(re) do |string|
         if (string =~ /QuickTime/) then
            filename = File.basename(f)
            dirname = File.dirname(f)
            puts "#{string}   #{dirname}"
#           puts "#{string}   #{dirname}  #{filename}"
            found = true
            break
         end
      end
      if (!found) then
         puts "Unknown  #{f}"
      end
      data = nil
      GC.start   # garbage collect
   end
end

Scott

-- 
Scott Ellsworth
scott@alodar.nospam.com
Java and database consulting for the life sciences