From: "Ara.T.Howard" <ahoward@...>
Date: 2004-07-16T01:32:28+09:00
Subject: Re: my ruby code won't go as fast as my perl code

On Thu, 15 Jul 2004, denis wrote:

> "Dave Burt" <burtdav@hotmail.com> wrote in message news:<y8pJc.1797$K53.1255@news-server.bigpond.net.au>...
>
>>
>> The main slow bit seems to be the string concatenation and comparison
>> (...+$4+$5+$6 >= start_date).
>>

minimize IO and use the fast stringscanner library:


   ~ > parse.rb csv/
   Read 29696 lines in 11.699876 seconds
   Wrote 29696 lines in 0.03556 seconds

   ~ > parse.pl csv/
   Read 29696 lines in 7 seconds
   Wrote in 0 seconds

   ~ > diff -u perl.out ruby.out


here's the code(s).  note that your perl script had two bugs in it - times were
not reported correctly and the first line containing a valid starting date was
not written to file.  the below assumes (like your code does) that the input is
sorted in ascending order (probably not a good assumption since it will fail
silently if not):


   ~ > cat parse.rb
   #!/usr/bin/env ruby
   require 'strscan'
   dir = ARGV.shift

   mm = Hash.new
   i = '00'
   %w(JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC).each do |mmm|
    mm[mmm] = i = i.succ
   end

   start_date = '20040101000000'  # "yyyymmddhhmmss"
   date_regex = /^(\d\d)-(\w\w\w)-(\d\d) (\d\d):(\d\d):(\d\d).*$\n/o
   anyline = %r/^.*$\n/o
   a = []
   t = Time.new
   buf = nil

   Dir.foreach(dir) do |path|
     next if path[0] == ?.

     buf = IO.read(File.join(dir, path))
     s = StringScanner.new buf

     while s.rest?
       if s.scan date_regex
         date = "#{ s[3] >= '87' ? '19' : '20' }#{ s[3] }#{ mm[s[2]] }#{ s[1] }#{ s[4] }#{ s[5] }#{ s[6] }"
         if date >= start_date
           a << s[0]
           a << s.scan(anyline) while s.rest?
         end
       else
         s.scan anyline
       end
     end
   end

   t = Time.now - t;
   puts "Read #{ a.size } lines in #{ t } seconds"
   t = Time.now
   File.open('ruby.out', 'w'){|f| a.each{|e| f.print e}}
   t = Time.new - t;
   puts "Wrote #{ a.size } lines in #{ t } seconds"; # 3 seconds

   ~ > cat parse.pl
   #!/usr/bin/env perl
   $dir = shift;
   $start_date = '20040000000000';  # "yyyymmddhhmmss"

   @months = qw(JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC);
   %mm = {};
   for ($i = 0; $i < 12; $i++) {
    $mm{$months[$i]} = sprintf('%.2d', $i)
   }
   undef @months;

   @a = ();
   $t = time;

   opendir DIR, $dir;
   while ($_ = readdir DIR) {
    next if /^\./;  # skip dotfiles
    open IN, "$dir/$_";
    while (<IN>) {
     /^(\d\d)-(\w\w\w)-(\d\d) (\d\d):(\d\d):(\d\d)/;
     $cc = ($3 ge '87' ? '19' : '20');
     if ("$cc$3$mm{$2}$1$4$5$6" ge $start_date) {
      push @a, $_;
      while (<IN>) {
       push @a, $_;
      }
     }
    }
    close IN;
   }
   closedir DIR;

   $t = time - $t;
   print "Read " . scalar(@a) . " lines in $t seconds$/";  # 4 seconds
   $t = time;
   open OUT, ">perl.out";
   print OUT @a;
   $t = time - $t;
   print "Wrote in $t seconds$/"; # 3 seconds


i generated the data sets with this:

   ~ > cat gendata.rb
   require 'fileutils'
   dir = ARGV.shift
   FileUtils.mkdir_p dir

   t_start = Time.mktime(1987)
   t_end = Time.now
   delta_t = t_end - t_start
   t_fmt = '%d-%b-%y %H:%M:%S'  # like 31-DEC-03 23:59:59

   1024.times do |fn|
     path = File.join dir, "#{ fn }.csv"
     open(path, 'w') do |f|
       time = t_start + rand(delta_t)
       1024.times do |lineno|
         row = time.strftime(t_fmt).upcase, rand(42), rand(42), rand(42), rand(42)
         f.puts(row.join(','))
         time += rand(42)
       end
     end
   end

it makes 1024 files, each of 1024 lines containing ordered tuples of a format
like your input data.


-a
--
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| A flower falls, even though we love it;
| and a weed grows, even though we do not love it. 
|   --Dogen
===============================================================================