From: Eric Mahurin <eric.mahurin@...>
Date: 2005-11-06T07:57:23+09:00
Subject: Re: parser performance comparisions

------=_Part_43477_18019238.1131231440761
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

On 11/5/05, Jim Freeze <jim@freeze.org> wrote:
>
> On 11/5/05, Eric Mahurin <eric.mahurin@gmail.com> wrote:
> > Other comparisons may be needed, but this one is very important.
> > > It says that Grammar can play equally with RACC in terms of speed,
> > > which is important if you are the new kid on the block.
> > >
> > > What I would like to see is grammar speed up a bit so that the
> > > one-token-at-a-time
> > > is as fast as Racc with the C extension. Then it would almost be a no
> > > brainer
> > > to use Grammar instead of Racc.
> >
> > Why is the one-token-at-a-time Grammar lexer important? If you already
> have
> > a Regexp lexer (for RACC or whatever), it is very easy to make it a
> lexer
> > for a Grammar parser - just make it an object that has a read1next
> method
> > (like a Cursor) - similar to next_token with yacc. Until I start
> generating
> > C code, it will be hard to compete against a Regexp lexer since it does
> > stuff in C.
>
> Memory consumption.
> From the data that you showed, it looks like I should avoid
> Regexp if I am concerned about memory usage.
>

Yes, but you have the same problem in racc (and more since typical usage is
to generate the entire token stream in memory first). The Regexp lexers
could be brought down to more reasonable memory levels if you read in a
block (or line) at a time to match against. In this particular example a
line a time wouldn't help since I put the entire 2.6MB expression on a line
(not realistic of course). With reading in a block (fixed number of
characters) or line at a time, you'd have to deal with tokens that can span
multiple blocks or lines. You'd lose some of the Regexp advantage but
probably not much. The racc memory usage could also be helped if you don't
generate the entire token stream in memory first. I'm not sure why there ar=
e
no racc examples with low memory usage (don't read the entire file into a
string and don't generate the entire token stream in memory up front).

------=_Part_43477_18019238.1131231440761--