From: ara.t.howard@...
Date: 2006-03-09T03:02:44+09:00
Subject: Re: Searching for a very fast string parser

On Thu, 9 Mar 2006, |MKSM| wrote:

> Hello,
>
> I want to parse a log file containing several line in the same format.
> My log files are about 50mb each (350k lines) so i need something
> quite fast. The current (and fastest) solution i came up with is using
> StringScanner.
>
> I save what i get into variables and then pass them all into a Struct
> i created. Each new struct is then passed into an Array that holds all
> structs.
>
>
> Here's my test code:
>
> require 'strscan'
>
> a = "1140908573.050732 rule 19/0(match): pass unkn(255) on sis1:
> 80.202.226.15.50000 > 192.168.0.6.52525: UDP, length 64"
>
> s = StringScanner.new(a)
> time = s.scan(/\d+\.\d+/)
> s.pos += 23
> rule_no = s.scan(/\d+/)
> s.skip(/[\d\D]*?\s/)
> stat = s.scan(/\w+/)
> s.skip(/.*on\s/)
> interface = s.scan(/\w+\:/)
> s.skip(/\D+?\s/)
> out_ip = s.scan(/(\d+\.){3}\d{0,3}/)
> s.pos += 1
> out_port = s.scan(/\d+/)
> s.skip(/\D+/)
> in_ip = s.scan(/(\d+\.){3}\d{0,3}/)
> s.pos += 1
> in_port = s.scan(/\d+/)
> s.pos += 2
> proto = s.scan(/\w+/)
> proto
> s.pos += 1
>
> Running that on a 10k times loop it takes about 0.6 seconds to
> complete. Is there a better/faster way on doing it?
>
> Regards,
>
> Ricardo.

can you put a demo log file on the web somewhere?

-a

-- 
knowledge is important, but the much more important is the use toward which it
is put.  this depends on the heart and mine the one who uses it.
- h.h. the 14th dali lama