From: Robert Klemme Date: 2006-03-09T23:48:54+09:00 Subject: Re: Searching for a very fast string parser ------=_NextPart_000_0085_01C64390.8F0BBAA0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Robert Klemme wrote: > Caleb Clausen wrote: >> OK, so first off, your sample implementation seemed to have several >> bugs in it. After fixing those, I thought you might be able to save >> some time by glomming all the regexp's together, obviating the need >> for StringScanner altogether. However, that doesn't seem to have >> actually made any difference... > > I don't buy this. A single plain RX is usually faster than a more > complex solution. Even on a machine with constant high load (I had > no different available at the moment) I get a significant difference > (north of 6%): > >>> 15:22:14 [source]: /c/temp/ruby/logscan.rb > Rehearsal ------------------------------------------------ > strscan 5.969000 0.000000 5.969000 ( 6.095000) > rx 5.828000 0.000000 5.828000 ( 5.951000) > rx with conv 5.860000 0.000000 5.860000 ( 5.922000) > -------------------------------------- total: 17.657000sec > > user system total real > strscan 5.953000 0.000000 5.953000 ( 6.043000) > rx 5.547000 0.000000 5.547000 ( 5.747000) > rx with conv 5.765000 0.000000 5.765000 ( 5.924000) > > (script attached) > >> if anything it seems to have been a >> little slower. I don't know why. And the great big long Regexp is >> considerably harder to read. > > Using %r{} and /x makes a great deal in readability (see script). > > Kind regards > > robert I redid the test on an idle Linux machine with Ruby 1.8.1 and the StringScanner is actually faster: [root@fox tmp]# ./logscan.rb Rehearsal ------------------------------------------------ strscan 2.990000 0.000000 2.990000 ( 2.991096) rx 4.870000 0.000000 4.870000 ( 4.868536) rx with 4.280000 0.010000 4.290000 ( 4.284334) rx with conv 5.240000 0.000000 5.240000 ( 5.459702) -------------------------------------- total: 17.390000sec user system total real strscan 3.000000 0.000000 3.000000 ( 2.999783) rx 4.870000 0.000000 4.870000 ( 4.899242) rx with 4.300000 0.010000 4.310000 ( 4.869835) rx with conv 5.240000 0.000000 5.240000 ( 5.442722) Apparently I have to correct myself... robert ------=_NextPart_000_0085_01C64390.8F0BBAA0 Content-Type: application/octet-stream; name="logscan.rb" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="logscan.rb" #!/usr/bin/env ruby=0A= =0A= require 'benchmark'=0A= require 'strscan'=0A= =0A= REP =3D 100_000=0A= =0A= LINE =3D "1140908573.050732 rule 19/0(match): pass unkn(255) on sis1: = 80.202.226.15.50000 > 192.168.0.6.52525: UDP, length 64"=0A= =0A= Benchmark.bmbm(10) do |bb|=0A= bb.report "strscan" do=0A= REP.times do=0A= s =3D StringScanner.new(LINE)=0A= time =3D s.scan(/\d+\.\d+/)=0A= s.pos =3D 23=0A= rule_no =3D s.scan(/\d+/)=0A= s.skip(/\S+\s/)=0A= stat =3D s.scan(/\w+/)=0A= s.skip(/\s\S+\son\s/)=0A= interface =3D s.scan(/\w+\:/)=0A= s.skip(/\s/)=0A= out_ip =3D s.scan(/(?:\d+\.){4}/)=0A= out_port =3D s.scan(/\d+/)=0A= s.skip(/ > /)=0A= in_ip =3D s.scan(/(?:\d+\.){4}/)=0A= in_port =3D s.scan(/\d+/)=0A= s.pos +=3D 2=0A= proto =3D s.scan(/\w+/)=0A= end=0A= end=0A= =0A= bb.report "rx" do=0A= REP.times do=0A= if %r{=0A= \A=0A= (\d+\.\d+) # time=0A= \s+rule\s+=0A= (\d+) # rule=0A= /\d+\(\w+\):\s+=0A= (\w+) # stat=0A= \s+\w+\(\d+\)\s+on\s+=0A= (\w+) # iface=0A= :\s+=0A= (\d+(?:\.\d+){3}) # ip1=0A= \.=0A= (\d+) # port1=0A= \s+>\s+=0A= (\d+(?:\.\d+){3}) # ip2=0A= \.=0A= (\d+) # port2=0A= :\s+=0A= (\w+) # proto=0A= }x =3D~ LINE then=0A= time =3D $1=0A= rule_no =3D $2=0A= stat =3D $3=0A= interface =3D $4=0A= out_ip =3D $5=0A= out_port =3D $6=0A= in_ip =3D $7=0A= in_port =3D $8=0A= proto =3D $9=0A= end=0A= end=0A= end=0A= =0A= bb.report "rx with \s" do=0A= REP.times do=0A= if %r{=0A= \A=0A= (\d+\.\d+) # time=0A= \srule\s=0A= (\d+) # rule=0A= /\d+\(\w+\):\s=0A= (\w+) # stat=0A= \s\w+\(\d+\)\son\s=0A= (\w+) # iface=0A= :\s=0A= (\d+(?:\.\d+){3}) # ip1=0A= \.=0A= (\d+) # port1=0A= \s>\s=0A= (\d+(?:\.\d+){3}) # ip2=0A= \.=0A= (\d+) # port2=0A= :\s=0A= (\w+) # proto=0A= }x =3D~ LINE then=0A= time =3D $1=0A= rule_no =3D $2=0A= stat =3D $3=0A= interface =3D $4=0A= out_ip =3D $5=0A= out_port =3D $6=0A= in_ip =3D $7=0A= in_port =3D $8=0A= proto =3D $9=0A= end=0A= end=0A= end=0A= =0A= bb.report "rx with conv" do=0A= REP.times do=0A= if %r{=0A= \A=0A= (\d+\.\d+) # time=0A= \s+rule\s+=0A= (\d+) # rule=0A= /\d+\(\w+\):\s+=0A= (\w+) # stat=0A= \s+\w+\(\d+\)\s+on\s+=0A= (\w+) # iface=0A= :\s+=0A= (\d+(?:\.\d+){3}) # ip1=0A= \.=0A= (\d+) # port1=0A= \s+>\s+=0A= (\d+(?:\.\d+){3}) # ip2=0A= \.=0A= (\d+) # port2=0A= :\s+=0A= (\w+) # proto=0A= }x =3D~ LINE then=0A= time =3D $1.to_f=0A= rule_no =3D $2.to_i=0A= stat =3D $3=0A= interface =3D $4=0A= out_ip =3D $5=0A= out_port =3D $6.to_i=0A= in_ip =3D $7=0A= in_port =3D $8.to_i=0A= proto =3D $9=0A= end=0A= end=0A= end=0A= =0A= end=0A= =0A= =0A= __END__=0A= s =3D StringScanner.new(line)=0A= time =3D s.scan(/\d+\.\d+/)=0A= s.pos =3D 23=0A= rule_no =3D s.scan(/\d+/)=0A= s.skip(/\S+\s/)=0A= stat =3D s.scan(/\w+/)=0A= s.skip(/\s\S+\son\s/)=0A= interface =3D s.scan(/\w+\:/)=0A= s.skip(/\s/)=0A= out_ip =3D s.scan(/(?:\d+\.){4}/)=0A= out_port =3D s.scan(/\d+/)=0A= s.skip(/ > /)=0A= in_ip =3D s.scan(/(?:\d+\.){4}/)=0A= in_port =3D s.scan(/\d+/)=0A= s.pos +=3D 2=0A= proto =3D s.scan(/\w+/)=0A= # end=0A= =0A= puts time, rule_no, stat, interface, out_ip, out_port, in_ip, in_port, = proto=0A= puts "----"=0A= =0A= %r{=0A= \A=0A= (\d+\.\d+) # time=0A= \s+rule\s+=0A= (\d+) # rule=0A= /\d+\(\w+\):\s+=0A= (\w+) # stat=0A= \s+\w+\(\d+\)\s+on\s+=0A= (\w+) # iface=0A= :\s+=0A= (\d+(?:\.\d+){3}) # ip1=0A= \.=0A= (\d+) # port1=0A= \s+>\s+=0A= (\d+(?:\.\d+){3}) # ip2=0A= \.=0A= (\d+) # port2=0A= :\s+=0A= (\w+) # proto=0A= }x =3D~ line and puts $1, $2, $3, $4, $5, $6, $7, $8, $9=0A= =0A= 1140908573.050732=0A= 19=0A= pass=0A= sis1:=0A= 80.202.226.15.=0A= 50000=0A= 192.168.0.6.=0A= 52525=0A= UDP=0A= ------=_NextPart_000_0085_01C64390.8F0BBAA0--