From: Francis Cianfrocca <garbagecat10@...>
Date: 2006-05-13T13:12:23+09:00
Subject: Re: lightweigth lexer?

------=_Part_39316_23038438.1147493540170
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Well, yes, that's what I meant by "not much more." In my experience (writin=
g
scanners for programming languages) the start states and greedy matching
were the most important features beyond the basic NFA. Unshifting input
seems like more of a job for the parser than the scanner, but different
strokes, I guess. Are you going to give this a try?

On 5/13/06, Meador Inge <meadori@gmail.com> wrote:
>
> > If you look closely at what traditional tools like lex and flex do, you
> > could almost take the input to lex and execute it directly in Ruby!
> They do look very similar.
>
> > It's not much more than a case statement from regular expressions.
> It is a bit more that a case statement on regular expressions:). You have
> to
> do a bit of work to manage the input buffer and ensure that the longest
> match is made (e.g. greedy) for each token. Also, if you want some of the
> other traditional features of lex and flex you have to add features such
> as
> start states, putting characters back into the input stream, reject
> matches,
> etc...
>
> > I'm not sure what you mean by "lightweight"
> I more or less meant simple and easy to use (maybe I should have just sai=
d
> so:)) without the full range of features of a typical lex\flex knockoff.
>
>
> On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
> >
> > If you look closely at what traditional tools like lex and flex do, you
> > could almost take the input to lex and execute it directly in Ruby! It'=
s
> > not
> > much more than a case statement from regular expressions. I'm not sure
> > what
> > you mean by "lightweight" but if you mean "fast": a lot of people who
> > wrote
> > language compilers back in the day (I'm one of them) would often write
> > scanners in hand-tuned C rather than using lex.
> >
> > On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
> > >
> > > Is there lightweight lexer for Ruby? I have seen a few lexer
> > > generators out there, but I want something simple that can be used
> > > without generating the lexer first. I really just need the basic
> > > functionaliy of matching a pattern and associating an action with tha=
t
> > > pattern. For example, I was thinking something like:
> > >     class MyLexer < Lexer
> > >        rule /a/ do
> > >           # action for pattern a
> > >        end
> > >        rule /b/ do
> > >           # action for pattern b
> > >        end
> > >        rule /c/ do
> > >           # action for pattern c
> > >        end
> > >     end
> > >
> > >     l =3D MyLexer.new(input_stream)
> > >     l.each_token do |token|
> > >        # do something with token
> > >     end
> > >
> > > I have something similar to the above that I have created. However, I
> > > wanted to check and see what is out there first before I invested too
> > > much time in it. If no one knows of anything else and\or is intereste=
d
> > > in what I have described let me know and I will clean it up and
> > > publish it.
> > >
> > > --Meador
> > >
> > >
> >
> >
>
>

------=_Part_39316_23038438.1147493540170--