From: Ryan Davis Date: 2002-09-23T17:17:04+09:00 Subject: Re: Speed up suggestions --Apple-Mail-4-774728434 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII; format=flowed On Monday, September 23, 2002, at 12:39 AM, Tomas Brixi wrote: > Thanks all for speedup tips. > > I have tried all of them and the fastest one is attached. > Results: > ruby : 115 sec -> 62 sec (wow :-) > python : 60 sec -> 53 sec Are you sure you included the right script? I just expanded your weblog to 4.5 megs and ran the script you included and got a time of 80 seconds (via time). I then added the rest of the changes I suggested and ran my version in 26 seconds (via time). Included below are the scripts and the output. The changes were: 1) get rid of all calls to index. 2) put all regexen inline. > But there could be generally conditions put on more fields. What then? > Use String.split to get the fields and then match single fields or > build a all_in_one regex and try to match the whole line? That really depends. If you can order the conditions to exclusion in such a way that you can avoid the split, you probably want to go that way. But I'd just measure and see. --Apple-Mail-4-774728434 Content-Disposition: attachment; filename=parse.rb.orig Content-Transfer-Encoding: quoted-printable Content-Type: application/octet-stream; x-unix-mode=0644; name="parse.rb.orig" =0D=0Aoutf=20=3D=20File.new(=20"found.log",=20"w"=20)=0D=0Ar=20=3D=20= Regexp.new(=20"xxx\\xxxxx",=20Regexp::IGNORECASE=20)=0D=0Arc=20=3D=20= Regexp.new(=20"^#"=20)=0D=0At=20=3D=20Time.new=0D=0Aputs=20t=0D=0A= File.new(=20"WEB.log"=20).each_line=20do=0D=0A=09|line|=0D=0A=09next=20= if=20line=20=3D~=20rc=0D=0A=09t1=20=3D=20line.index("\t")+1;=20pom=20=3D=20= line[t1..(line.index("\t",=20t1)-1)]=0D=0A=09if=20pom=20=3D~=20r=0D=0A=09= =09outf.puts(=20line=20)=0D=0A=09end=0D=0Aend=0D=0Ate=20=3D=20Time.new=0D= =0Aputs=20te=0D=0Aprint=20"Time:=20"=0D=0Aputs=20(te=20-=20t).to_s=0D=0A= outf.close()=0D=0A= --Apple-Mail-4-774728434 Content-Disposition: attachment; filename=parse.rb.time Content-Transfer-Encoding: 7bit Content-Type: application/octet-stream; x-unix-mode=0664; name="parse.rb.time" % cumulative self self total time seconds seconds calls ms/call ms/call name 59.55 12.10 12.10 1 12100.00 20300.00 IO#each_line 31.05 18.41 6.31 27540 0.23 0.30 IO#puts 9.30 20.30 1.89 27547 0.07 0.07 IO#write 0.05 20.31 0.01 3 3.33 3.33 Kernel.puts 0.00 20.31 0.00 1 0.00 0.00 NilClass#to_s 0.00 20.31 0.00 1 0.00 0.00 Time#- 0.00 20.31 0.00 2 0.00 0.00 File#new 0.00 20.31 0.00 1 0.00 0.00 Float#to_s 0.00 20.31 0.00 1 0.00 20320.00 #toplevel 0.00 20.31 0.00 2 0.00 0.00 File#initialize 0.00 20.31 0.00 1 0.00 0.00 Kernel.print 0.00 20.31 0.00 1 0.00 0.00 IO#close 0.00 20.31 0.00 2 0.00 0.00 Time#to_s 0.00 20.31 0.00 2 0.00 0.00 Object#initialize 0.00 20.31 0.00 2 0.00 0.00 Time#new Mon Sep 23 01:09:05 PDT 2002 Mon Sep 23 01:09:31 PDT 2002 Time: 26.383407 real 0m26.508s user 0m19.110s sys 0m1.270s --Apple-Mail-4-774728434 Content-Disposition: attachment; filename=parse.rb Content-Transfer-Encoding: quoted-printable Content-Type: application/octet-stream; x-unix-mode=0644; name="parse.rb" =0D=0Aoutf=20=3D=20File.new(=20"found.log",=20"w"=20)=0D=0At=20=3D=20= Time.new=0D=0Aputs=20t=0D=0AFile.new(=20"WEB.log"=20).each_line=20do=0D=0A= =09|line|=0D=0A=09next=20if=20line=20=3D~=20/^#/=0D=0A=0D=0A=09if=20line=20= =3D~=20/^[^\t]*\t([^\t]*)\t/=20then=0D=0A=09=09pom=20=3D=20$1=0D=0A=09=09= if=20pom=20=3D~=20/xxx\\xxxxx/i=20then=0D=0A=09=09=09outf.puts(=20line=20= )=0D=0A=09=09end=0D=0A=09end=0D=0Aend=0D=0Ate=20=3D=20Time.new=0D=0Aputs=20= te=0D=0Aprint=20"Time:=20"=0D=0Aputs=20(te=20-=20t).to_s=0D=0A= outf.close()=0D=0A= --Apple-Mail-4-774728434 Content-Disposition: attachment; filename=parse.rb.orig.time Content-Transfer-Encoding: 7bit Content-Type: application/octet-stream; x-unix-mode=0664; name="parse.rb.orig.time" % cumulative self self total time seconds seconds calls ms/call ms/call name 68.85 45.03 45.03 1 45030.00 65380.00 IO#each_line 9.86 51.48 6.45 82620 0.08 0.08 String#=~ 8.18 56.83 5.35 71604 0.07 0.07 String#index 4.72 59.92 3.09 35802 0.09 0.09 Fixnum#+ 4.68 62.98 3.06 35802 0.09 0.09 String#[] 3.67 65.38 2.40 35802 0.07 0.07 Fixnum#- 0.02 65.39 0.01 2 5.00 5.00 Object#initialize 0.00 65.39 0.00 2 0.00 0.00 File#new 0.00 65.39 0.00 3 0.00 0.00 Kernel.puts 0.00 65.39 0.00 2 0.00 0.00 Regexp#initialize 0.00 65.39 0.00 7 0.00 0.00 IO#write 0.00 65.39 0.00 1 0.00 0.00 NilClass#to_s 0.00 65.39 0.00 1 0.00 0.00 Float#to_s 0.00 65.39 0.00 1 0.00 0.00 Time#- 0.00 65.39 0.00 2 0.00 0.00 File#initialize 0.00 65.39 0.00 1 0.00 0.00 Kernel.print 0.00 65.39 0.00 1 0.00 0.00 IO#close 0.00 65.39 0.00 2 0.00 0.00 Time#to_s 0.00 65.39 0.00 1 0.00 65400.00 #toplevel 0.00 65.39 0.00 2 0.00 5.00 Time#new 0.00 65.39 0.00 2 0.00 0.00 Regexp#new Mon Sep 23 01:04:37 PDT 2002 Mon Sep 23 01:05:57 PDT 2002 Time: 80.671118 real 1m20.765s user 1m0.810s sys 0m4.650s --Apple-Mail-4-774728434--