From: Ned Konz <ned@...>
Date: 2001-08-16T03:07:19+09:00
Subject: [ruby-talk:19812] Re: how to speed up hash script?

On Wednesday 15 August 2001 10:32 am, you wrote:
> I'm in the process of writing a script that will read data from a 17MB
> climatology file and then hash the information according to date-time.
> The script below takes approximately 14 seconds to run. An analogous
> Python script takes 3 seconds to run. Is there a simple way to speed
> up the Ruby script? I'm interested in the possibility of using Ruby as
> an alternative to Python.

Hi Gerry,

First, why are you slurping the whole file into memory at once? This seems 
inefficient, since you're only populating the hash one line at a time.

Second, did you profile your program to see where the time went? Here's what 
I got when I used my line profiler (omitting lines that have 0% of the time):

$ ruby -r lineprofile2.rb climo.rb climo.dat

 Line time%  time  count

    5   5%   0.78      1     @lines = @fp.readlines()

   14  95%  16.14  49152       @dict[@lines[i][0...@len_date]] = i

OK, so re-write it to not slurp the file:

class Climo
  DATE_FORMAT = "YYMMDDHH"
  DATE_LEN = 8
  def initialize(in_file)
    @dict = Hash.new()
    
    # Read in data
    # Store line indices in dictionary based on date field
    fp = File.new(in_file, "r")
    i = 1
    first_line = fp.readline
    first_date_line = fp.readline
    line = ''
    fp.each_line { |line|
      @dict[ line[0...DATE_LEN] ] = i
      i += 1
    }
    fp.close()
    
    first_date = first_date_line[0...DATE_LEN]
    last_date = line[0...DATE_LEN]
    puts "first_date: ", first_date
    puts "last_date: ", last_date
  end

end

And profile it again. Here's where all the time goes (note that we're now 
down from about 17 seconds to about 13.5):

   14   0%   0.00      1     fp.each_line { |line|
   15  64%   8.65  49151       @dict[ line[0...DATE_LEN] ] = i
   16  36%   4.83  49151       i += 1
   17                        }

So you have to ask yourself if you really need to store the line numbers. 
Without them, it's even faster (down to 8.9 seconds from your original 16.9):

   14   0%   0.00      1     fp.each_line { |line|
   15 100%   8.91  49151       @dict[ line[0...DATE_LEN] ] = nil
   16                        }

Anyway, you could probably get it faster. (sorry if I missed something 
obvious; this is from someone who's written two Ruby programs total.)

-- 
Ned Konz
currently: Stanwood, WA
email:     ned@bike-nomad.com
homepage:  http://bike-nomad.com