From: Vidar Hokstad <vidar.hokstad@...>
Date: 2006-11-29T09:55:08+09:00
Subject: Re: "yield" and "old-way iteration"


Bjarke Walling wrote:
> I am writing a lexer and parser for a small language I have created.
> The first part splitting some input into tokens was easy to write using
> "yield" (20 lines or so). I was actually a little overwhelmed how easy
> it was. The next part is to examine these tokens and parse them into
> language structures. I want to create a class with "current" and "next"
> methods to get the current token and fetch the next (advance the
> pointer). I have solved it by using the first lexer yielding tokens and
> collecting them in an array. Afterwards I can fetch tokens from the
> array. But could it be done in a smarter way?

You're not providing much context. I am assuming that you want the
current/next approach because your parser will pull tokens, presumably
because you're using recursive descent or another top-down parsing
method.

If that's what you are doing, and you want to stick with that (as
opposed to switching to a bottom-up parser), then you're dealing with a
classic "inversion of control" problem.

I don't really think making the first lexer yielding tokens buys you
much over just making the parser call methods in the lexer to tokenize
and return the tokens as a normal method call.  In other words, if
you're using a top-down parsing method, you really want to consider
making your parser  pull tokens from the lexer, instead of having the
lexer push tokens to the parser, which is what you are doing when you
use yield.

However, if you want to stick to using yield, you can use "Generator"
(see http://ruby-doc.org/core/classes/Generator.html) to invert the
control and let you "pull" tokens from your yield'ing lexer without
having to go via an array.

Vidar