From: Robert Klemme <bob.news@...>
Date: 2006-03-09T23:33:46+09:00
Subject: Re: Searching for a very fast string parser

------=_NextPart_000_0036_01C6438E.7A64EF10
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

Caleb Clausen wrote:
> OK, so first off, your sample implementation seemed to have several
> bugs in it. After fixing those, I thought you might be able to save
> some time by glomming all the regexp's together, obviating the need
> for StringScanner altogether. However, that doesn't seem to have
> actually made any difference...

I don't buy this.  A single plain RX is usually faster than a more complex
solution.  Even on a machine with constant high load (I had no different
available at the moment) I get a significant difference (north of 6%):

>> 15:22:14 [source]: /c/temp/ruby/logscan.rb
Rehearsal ------------------------------------------------
strscan        5.969000   0.000000   5.969000 (  6.095000)
rx             5.828000   0.000000   5.828000 (  5.951000)
rx with conv   5.860000   0.000000   5.860000 (  5.922000)
-------------------------------------- total: 17.657000sec

                   user     system      total        real
strscan        5.953000   0.000000   5.953000 (  6.043000)
rx             5.547000   0.000000   5.547000 (  5.747000)
rx with conv   5.765000   0.000000   5.765000 (  5.924000)

(script attached)

> if anything it seems to have been a
> little slower. I don't know why. And the great big long Regexp is
> considerably harder to read.

Using %r{} and /x makes a great deal in readability (see script).

Kind regards

    robert

------=_NextPart_000_0036_01C6438E.7A64EF10
Content-Type: application/octet-stream;
	name="logscan.rb"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
	filename="logscan.rb"

#!/usr/bin/env ruby=0A=
=0A=
require 'benchmark'=0A=
require 'strscan'=0A=
=0A=
REP =3D 100_000=0A=
=0A=
LINE =3D "1140908573.050732 rule 19/0(match): pass unkn(255) on sis1: =
80.202.226.15.50000 > 192.168.0.6.52525: UDP, length 64"=0A=
=0A=
Benchmark.bmbm(10) do |bb|=0A=
  bb.report "strscan" do=0A=
    REP.times do=0A=
      s =3D StringScanner.new(LINE)=0A=
      time =3D s.scan(/\d+\.\d+/)=0A=
      s.pos =3D 23=0A=
      rule_no =3D s.scan(/\d+/)=0A=
      s.skip(/\S+\s/)=0A=
      stat =3D s.scan(/\w+/)=0A=
      s.skip(/\s\S+\son\s/)=0A=
      interface =3D s.scan(/\w+\:/)=0A=
      s.skip(/\s/)=0A=
      out_ip =3D s.scan(/(?:\d+\.){4}/)=0A=
      out_port =3D s.scan(/\d+/)=0A=
      s.skip(/ > /)=0A=
      in_ip =3D s.scan(/(?:\d+\.){4}/)=0A=
      in_port =3D s.scan(/\d+/)=0A=
      s.pos +=3D 2=0A=
      proto =3D s.scan(/\w+/)=0A=
    end=0A=
  end=0A=
=0A=
  bb.report "rx" do=0A=
    REP.times do=0A=
      if %r{=0A=
      \A=0A=
      (\d+\.\d+)              # time=0A=
      \s+rule\s+=0A=
      (\d+)                   # rule=0A=
      /\d+\(\w+\):\s+=0A=
      (\w+)                   # stat=0A=
      \s+\w+\(\d+\)\s+on\s+=0A=
      (\w+)                   # iface=0A=
      :\s+=0A=
      (\d+(?:\.\d+){3})       # ip1=0A=
      \.=0A=
      (\d+)                   # port1=0A=
      \s+>\s+=0A=
      (\d+(?:\.\d+){3})       # ip2=0A=
      \.=0A=
      (\d+)                   # port2=0A=
      :\s+=0A=
      (\w+)                   # proto=0A=
      }x =3D~ LINE then=0A=
        time      =3D $1=0A=
        rule_no   =3D $2=0A=
        stat      =3D $3=0A=
        interface =3D $4=0A=
        out_ip    =3D $5=0A=
        out_port  =3D $6=0A=
        in_ip     =3D $7=0A=
        in_port   =3D $8=0A=
        proto     =3D $9=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
  bb.report "rx with conv" do=0A=
    REP.times do=0A=
      if %r{=0A=
      \A=0A=
      (\d+\.\d+)              # time=0A=
      \s+rule\s+=0A=
      (\d+)                   # rule=0A=
      /\d+\(\w+\):\s+=0A=
      (\w+)                   # stat=0A=
      \s+\w+\(\d+\)\s+on\s+=0A=
      (\w+)                   # iface=0A=
      :\s+=0A=
      (\d+(?:\.\d+){3})       # ip1=0A=
      \.=0A=
      (\d+)                   # port1=0A=
      \s+>\s+=0A=
      (\d+(?:\.\d+){3})       # ip2=0A=
      \.=0A=
      (\d+)                   # port2=0A=
      :\s+=0A=
      (\w+)                   # proto=0A=
      }x =3D~ LINE then=0A=
        time      =3D $1.to_f=0A=
        rule_no   =3D $2.to_i=0A=
        stat      =3D $3=0A=
        interface =3D $4=0A=
        out_ip    =3D $5=0A=
        out_port  =3D $6.to_i=0A=
        in_ip     =3D $7=0A=
        in_port   =3D $8.to_i=0A=
        proto     =3D $9=0A=
      end=0A=
    end=0A=
  end=0A=
=0A=
end=0A=
=0A=
=0A=
__END__=0A=
s =3D StringScanner.new(line)=0A=
time =3D s.scan(/\d+\.\d+/)=0A=
s.pos =3D 23=0A=
rule_no =3D s.scan(/\d+/)=0A=
s.skip(/\S+\s/)=0A=
stat =3D s.scan(/\w+/)=0A=
s.skip(/\s\S+\son\s/)=0A=
interface =3D s.scan(/\w+\:/)=0A=
s.skip(/\s/)=0A=
out_ip =3D s.scan(/(?:\d+\.){4}/)=0A=
out_port =3D s.scan(/\d+/)=0A=
s.skip(/ > /)=0A=
in_ip =3D s.scan(/(?:\d+\.){4}/)=0A=
in_port =3D s.scan(/\d+/)=0A=
s.pos +=3D 2=0A=
proto =3D s.scan(/\w+/)=0A=
# end=0A=
=0A=
puts time, rule_no, stat, interface, out_ip, out_port, in_ip, in_port, =
proto=0A=
puts "----"=0A=
=0A=
%r{=0A=
\A=0A=
(\d+\.\d+)              # time=0A=
\s+rule\s+=0A=
(\d+)                   # rule=0A=
/\d+\(\w+\):\s+=0A=
(\w+)                   # stat=0A=
\s+\w+\(\d+\)\s+on\s+=0A=
(\w+)                   # iface=0A=
:\s+=0A=
(\d+(?:\.\d+){3})       # ip1=0A=
\.=0A=
(\d+)                   # port1=0A=
\s+>\s+=0A=
(\d+(?:\.\d+){3})       # ip2=0A=
\.=0A=
(\d+)                   # port2=0A=
:\s+=0A=
(\w+)                   # proto=0A=
}x =3D~ line and puts $1, $2, $3, $4, $5, $6, $7, $8, $9=0A=
=0A=
1140908573.050732=0A=
19=0A=
pass=0A=
sis1:=0A=
80.202.226.15.=0A=
50000=0A=
192.168.0.6.=0A=
52525=0A=
UDP=0A=

------=_NextPart_000_0036_01C6438E.7A64EF10--