From: |MKSM| <mksm.sama@...>
Date: 2006-03-09T03:09:44+09:00
Subject: Re: Searching for a very fast string parser

On 3/8/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
> On Thu, 9 Mar 2006, |MKSM| wrote:
>
> > Hello,
> >
> > I want to parse a log file containing several line in the same format.
> > My log files are about 50mb each (350k lines) so i need something
> > quite fast. The current (and fastest) solution i came up with is using
> > StringScanner.
> >
> > I save what i get into variables and then pass them all into a Struct
> > i created. Each new struct is then passed into an Array that holds all
> > structs.
> >
> >
> > Here's my test code:
> >
> > require 'strscan'
> >
> > a = "1140908573.050732 rule 19/0(match): pass unkn(255) on sis1:
> > 80.202.226.15.50000 > 192.168.0.6.52525: UDP, length 64"
> >
> > s = StringScanner.new(a)
> > time = s.scan(/\d+\.\d+/)
> > s.pos += 23
> > rule_no = s.scan(/\d+/)
> > s.skip(/[\d\D]*?\s/)
> > stat = s.scan(/\w+/)
> > s.skip(/.*on\s/)
> > interface = s.scan(/\w+\:/)
> > s.skip(/\D+?\s/)
> > out_ip = s.scan(/(\d+\.){3}\d{0,3}/)
> > s.pos += 1
> > out_port = s.scan(/\d+/)
> > s.skip(/\D+/)
> > in_ip = s.scan(/(\d+\.){3}\d{0,3}/)
> > s.pos += 1
> > in_port = s.scan(/\d+/)
> > s.pos += 2
> > proto = s.scan(/\w+/)
> > proto
> > s.pos += 1
> >
> > Running that on a 10k times loop it takes about 0.6 seconds to
> > complete. Is there a better/faster way on doing it?
> >
> > Regards,
> >
> > Ricardo.
>
> can you put a demo log file on the web somewhere?
>
> -a
>
> --
> knowledge is important, but the much more important is the use toward which it
> is put.  this depends on the heart and mine the one who uses it.
> - h.h. the 14th dali lama
>
>
I'm sorry, the log file i have comes from a live firewall. I'd rather
not release it.

The log is only consisted by several line such as the one i used in the code.

Regards,

Ricardo