From: brabuhr@...
Date: 2006-11-18T05:57:34+09:00
Subject: Re: regular expression too big

Peter Schrammel wrote:
> >> got  problem with big regexes:
> >> I have a regex of about 70000+ words concated with '|' that I'd like to
> >> match as a regex. /bla|blub|foo|bar|.....(70000)/
> >>
> >> But unfortunately ruby gives me a 'regular expression too big' if I'm
> >> trying to build such a thing.
> >> I had a look at the regex.c code and saw the limit of 1 << 16 bytes for
> >> regexes. Is there a way around this (without going down to 2000 words) ?
> >>
> >> Thanks for any hint

Jeffrey Schwab wrote:
> > You could optimize the regex a little for size, e.g. by factoring out
> > common prefixes:
> >
> >     (b(l(a|ub)|ar)|foo)...

Peter Schrammel wrote:
> Thought of that.

Have you seen:
"Converts a list of words to a regular expression with minimum
backtracking by joining words with common prefixes. It is a port
of the Perl module MakeRegex.pm by Hakan Kjellerstrand with
some improvements."
    http://raa.ruby-lang.org/project/makeregex/

YMMV; I have never used it on anything like the scale you are.