From: Steven Jenkins <steven.jenkins@...>
Date: 2005-11-06T09:55:11+09:00
Subject: Re: parser performance comparisions

Eric Mahurin wrote:
> Do you even need a separate thread? Why can't you just have #next_token just
> parse the next token and return it? From what I saw of your lexer, it looked
> like that would be possible, but then again I have almost no racc experience
> (I've only learned enough to try to benchmark against it). When you go
> mutli-threaded, you do have to make sure your doing enough work in each
> thread because thread switching is relatively expensive. For my
> multi-threaded test, increasing my token buffer from no buffering (lexer and
> parser synchronized) to a 300 token buffer made a couple orders of magnitude
> difference in performance.

If you want to limit memory consumption, a separate thread and 
SizedQueue are sufficient. That might not give the highest performance, 
but it's the cleanest code. If performance is an issue, then yes, you 
could have next_token read another line from the input file if the 
current buffer is empty.

> To me, if a parser is reading in a whole file into memory and/or generating
> all the tokens of the file in memory up front, it doesn't seem like a "real"
> parser. Regexp fits in this camp (parses from a string - in memory). With
> some of features that are being added to regexes (look at some of perl's
> advanced/beta features where you can embed code and do recursion), you could
> call a regex a parser, but it doesn't seem like one to me because it needs
> the whole thing it is parsing in memory. It just seems strange to me that
> the common racc usage does things this way. You could actually optimize a
> parser quite a bit if you knew everything it was parsing was in memory
> (regexes make this assumption for doing backtracking - similar to lookahead
> for a standard parser).

Again, only if performance and memory usage are issues. I'd guess that 
for most applications using racc, they aren't. And if they really are, 
Ruby might not be the best choice anyway. Ruby/racc will never seriously 
compete with C/yacc in speed. For my particular application, the largest 
file I'll ever encounter can be parsed in less than one second, so I put 
exactly zero effort into making it faster.

Steve