From: Eric Mahurin <eric.mahurin@...>
Date: 2005-11-04T23:22:50+09:00
Subject: Re: [RFC] Method of feeding input to regexp matching

------=_Part_31275_24459338.1131114167250
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On 11/3/05, Nikolai Weibull <mailing-lists.ruby-core@rawuncut.elitemail.org=
>
wrote:
>
> I would very much like to be able to provide a Regexp object input from
> some input source other than a "fixed" string. Examples of such sources
> would be a file bufferer that reads chunks from a file and can then make
> them accessible to the Regexp object as it needs them. At the moment
> there's no way to do this. The exact semantics of such a method might
> be quite complicated, as it can be hard to maintain the MatchData. How
> this is to be done is beyond this initial query, but I would like to
> know if anyone besides me sees any merit in this.
>

Since I've been doing parsers, I've thought about this a bit too. Because o=
f
the problems you've mentioned, I've decided not to incorporated regexes in
my parser stuff. The only reason I would want to would be for performance
reasons in the lexers (my competing method is more consistent, flexible, an=
d
more readable - for complex stuff). Regexes being confined to a String is
likely the reason I can't find any racc examples where the lexer reads in a=
n
IO/File rather than a String.

I have put some hacky solutions in my Cursor classes (external
iterator/stream stuff), but I'm not using them in Grammar (parser/lexer
stuff). Here are the methods I came up with (after some discussion with
Caleb Clausen):

# scan for a pattern (\A anchored) with a finite length (specify max length
of the pattern)
scan_pattern(pattern,len=3D1,hold=3Dfalse,buffer=3Dnil)

# scan until a finite pattern is found (specify max length of the pattern)
# - kind of works like IO#gets(aString)
scan_pattern_until(pattern,len=3D1,hold=3Dfalse,buffer=3Dnil,init=3D16)

# scan while a loop pattern matches (specify max length of an iteration -
finite)
scan_pattern_while(pattern,len=3D1,hold=3Dfalse,buffer=3Dnil,init=3D16)

With these, you might match a multi-line comment (could be any length) like
this:

cursor.scan_pattern(/\A\/\*/,2,false,buf=3D"") && cursor.scan_pattern_until
(/\*\//,2,false,buf)

buf should contain the multi-line comment if this was successful.


I see several solutions to the problems being discussed:

1. live with - do something like the above

2. you should be able to specify a get-more-data method when a regexp hits
the end of a string

3. you should have access to whether the regexp hit the end of string (pass
or fail)

4. Regexp should be duck-typed (with a String optimization and possibly
String buffering for other types) such that it can operate on anything that
responds to a subset of String methods (minimal: #[]). It wouldn't be too
difficult to make an IO (responding to #pos) handle a string-like #[]
method.

------=_Part_31275_24459338.1131114167250--