From: Meador Inge <meadori@...>
Date: 2006-05-13T13:00:16+09:00
Subject: Re: lightweigth lexer?

------=_Part_8739_29467500.1147492813503
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

> If you look closely at what traditional tools like lex and flex do, you
> could almost take the input to lex and execute it directly in Ruby!
They do look very similar.

> It's not much more than a case statement from regular expressions.
It is a bit more that a case statement on regular expressions:). You have t=
o
do a bit of work to manage the input buffer and ensure that the longest
match is made (e.g. greedy) for each token. Also, if you want some of the
other traditional features of lex and flex you have to add features such as
start states, putting characters back into the input stream, reject matches=
,
etc...

> I'm not sure what you mean by "lightweight"
I more or less meant simple and easy to use (maybe I should have just said
so:)) without the full range of features of a typical lex\flex knockoff.


On 5/12/06, Francis Cianfrocca <garbagecat10@gmail.com> wrote:
>
> If you look closely at what traditional tools like lex and flex do, you
> could almost take the input to lex and execute it directly in Ruby! It's
> not
> much more than a case statement from regular expressions. I'm not sure
> what
> you mean by "lightweight" but if you mean "fast": a lot of people who
> wrote
> language compilers back in the day (I'm one of them) would often write
> scanners in hand-tuned C rather than using lex.
>
> On 5/12/06, Meador Inge <meadori@gmail.com> wrote:
> >
> > Is there lightweight lexer for Ruby? I have seen a few lexer
> > generators out there, but I want something simple that can be used
> > without generating the lexer first. I really just need the basic
> > functionaliy of matching a pattern and associating an action with that
> > pattern. For example, I was thinking something like:
> >     class MyLexer < Lexer
> >        rule /a/ do
> >           # action for pattern a
> >        end
> >        rule /b/ do
> >           # action for pattern b
> >        end
> >        rule /c/ do
> >           # action for pattern c
> >        end
> >     end
> >
> >     l =3D MyLexer.new(input_stream)
> >     l.each_token do |token|
> >        # do something with token
> >     end
> >
> > I have something similar to the above that I have created. However, I
> > wanted to check and see what is out there first before I invested too
> > much time in it. If no one knows of anything else and\or is interested
> > in what I have described let me know and I will clean it up and
> > publish it.
> >
> > --Meador
> >
> >
>
>

------=_Part_8739_29467500.1147492813503--