From: Ernie <erne@...>
Date: 2004-07-16T01:42:18+09:00
Subject: Re: my ruby code won't go as fast as my perl code


"Dave Burt" <burtdav@hotmail.com> wrote in message
news:y8pJc.1797$K53.1255@news-server.bigpond.net.au...
> I realise I'm doing this a perlish way, but my question is, is it possible
> to do this operation in Ruby in a time more comparable to what the Perl
> version's getting? (That's about 4 seconds; my Ruby code runs in about 17
> seconds over the same data set, which is far smaller than the production
> data set.)
>
> Basically, we have CSV files with a date like 31-DEC-03 23:59:59 as the
> first field (always in order), and the task is to grab into an array (to
> later process further) just the parts of each file that fall after a given
> date.
>
> The main slow bit seems to be the string concatenation and comparison
> (...+$4+$5+$6 >= start_date).
>
> ################################################################
I built 3 files all with only 9 lines of data, one of the files had a line
that fails the regex text.
I read the files and find the appropriate lines 1000 times in just under 4
seconds on a Pentium 850  Windows XP machine running Ruby 1.81-12.

In the code below I use interpolation, rather than concatenation which
speeds things up a little  about 1.5 seconds in the trial.
I also take advantage of a couple of Ruby features.  Array#delete_if  to
eliminate the lines that fail the regex.
I add a function to class Array that does a binary search for the right
place in the array.  This eliminates some searching through the file.
This will speed up your search.  Of course the increase in speed will depend
on how many lines exist in each file and how many lines precede the one
where you want to start.

You could also write a function in Perl  that would do a binary search as
well.

Since you have sorted dates to begin with, there is no reason not to do a
binary search.
Please reply to the group with your results if you try this on your data .

Ernie


class Array
def findGE(start_date, date_regex, mm)
     starter=0
     ender=self.length
     while true do
      pt=(ender-starter)/2 + starter
       date_regex =~ self[pt]
               #if ($3>='87'?'19':'20')+$3+mm[$2]+$1+$4+$5+$6 >= start_date
               if "#{$3>='87'?'19':'20'}#{$3}#{mm[$2]}#{$4}#{$5}#{$6}" >=
start_date
                    ender=pt
               else
                    starter=pt
               end
          if (ender-starter) <= 1
               date_regex =~ self[starter]
               #return starter if ($3>='87'?'19':'20')+$3+mm[$2]+$1+$4+$5+$6
>= start_date
               return starter if
"#{$3>='87'?'19':'20'}#{$3}#{mm[$2]}#{$4}#{$5}#{$6}" >= start_date
               date_regex =~ self[ender]
               #return ender if ($3>='87'?'19':'20')+$3+mm[$2]+$1+$4+$5+$6
>= start_date
               return ender if
"#{$3>='87'?'19':'20'}#{$3}#{mm[$2]}#{$4}#{$5}#{$6}" >= start_date
               return false
          end
     end
end
end
mm = Hash.new
i = '00'
%w(JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC).each do |mmm|
 mm[mmm] = i = i.succ
end
start_date = '20040000000000'
date_regex = /^(\d\d)-(\w\w\w)-(\d\d) (\d\d):(\d\d):(\d\d)/
dir="C:/dataTest"
t=Time.now.to_f
1..1000.times do
 a=[]
 aFinal=[]
 Dir.open(dir).each do |file|
  next if file[0] == ?.
  File.open(dir + '/' + file){|f|
  a=f.readlines}
  a.delete_if{|line| not date_regex =~ line}
  z=a.findGE(start_date, date_regex, mm)
  aFinal = aFinal + a[z...a.length] if z
 end
end
tend=Time.now.to_f
puts "#{tend-t}"

Here is one file, (the one with the bad line)

12-APR-98 21:59:59, aaaa,bbbb,cccc,dddd
30-JUL-99 20:05:35, cccc,ffff,gggg,hhhh
27-JAN-00 15:15:45, xxxx,ffff,cccc,dddd
28-FEB-01 12:30:20, zzzz,bbbb,dddd,gggg
31-DEC-03 23:59:59, xxxx,xxxxx,yyyyy,zzzzz
01-JAN-04 00:01:00, aaaa,bbbb,cccc,dddd
01-FEB-04 00:01:05, bbbb,cccc,dddd,xxxx
05-MAR-04 05:01:59, aaaa,bbbb,cccc,dddd
08-APR-04 05:15:35, aaaa,bbbb,cccc,xxxx
nnyy,aaa,bbb,ccc,xxx