From: |MKSM| <mksm.sama@...>
Date: 2006-03-09T02:29:59+09:00
Subject: Searching for a very fast string parser

Hello,

I want to parse a log file containing several line in the same format.
My log files are about 50mb each (350k lines) so i need something
quite fast. The current (and fastest) solution i came up with is using
StringScanner.

I save what i get into variables and then pass them all into a Struct
i created. Each new struct is then passed into an Array that holds all
structs.


Here's my test code:

require 'strscan'

a = "1140908573.050732 rule 19/0(match): pass unkn(255) on sis1:
80.202.226.15.50000 > 192.168.0.6.52525: UDP, length 64"

s = StringScanner.new(a)
time = s.scan(/\d+\.\d+/)
s.pos += 23
rule_no = s.scan(/\d+/)
s.skip(/[\d\D]*?\s/)
stat = s.scan(/\w+/)
s.skip(/.*on\s/)
interface = s.scan(/\w+\:/)
s.skip(/\D+?\s/)
out_ip = s.scan(/(\d+\.){3}\d{0,3}/)
s.pos += 1
out_port = s.scan(/\d+/)
s.skip(/\D+/)
in_ip = s.scan(/(\d+\.){3}\d{0,3}/)
s.pos += 1
in_port = s.scan(/\d+/)
s.pos += 2
proto = s.scan(/\w+/)
proto
s.pos += 1

Running that on a 10k times loop it takes about 0.6 seconds to
complete. Is there a better/faster way on doing it?

Regards,

Ricardo.