From: Peter Schrammel <peter.schrammel@...>
Date: 2006-11-17T08:05:05+09:00
Subject: Re: regular expression too big

Paul Lutus wrote:
> Yes, unless he is matching whole words, he is stuck with regexes. There is
> very likely to be a refactoring for this problem, and it would have to
> start with a clear statement of the problem to be solved.
> 
Sorry, my fault.

The problem is to match a whole bunch (>70000) of words (later regexps)
on a string.
The actual implementation is to concat the word with | and create a big
regexp. word1|word2|word3|word4 ...
This went well until I tested with some 100 words. Now I have the 'big
regexp problem' problem. The solution has to work with regexps as words
as well like: word1(the|a|an)word11|regexp2|...
So the currently working solutions are:
-use ruby 1.9 (the performance is far below perl, but I have to
benchmark this.)
-loop through the words/regexp and match each of them on the string
(don't ask for performance here).
-perhaps I'll realy pipe the problem into a
perl-implemented-matching-server (not that bad idea)

peter