From: "Luke A. Kanies" <luke@...>
Date: 2003-12-12T01:28:57+09:00
Subject: Re: problems with racc: $end token

On Fri, 12 Dec 2003, Jim Freeze wrote:

> On Friday, 12 December 2003 at  0:42:30 +0900, Luke A. Kanies wrote:
> > Hello,
> >
> > def next_token
> >     ....
> >     if @src.length == 0
> >         puts "returning end"
> >         #return [false, 0]
> >         #return [false, '$']
> >         return [false, false]
> >     end
> >     ....
> > end
> >
>
> I apologize for not being able to dig more into this, but for
> my next_token method I have:
>
>   def next_token
>     @q.shift
>   end
>
> Are there two different ways to setup the tokenizing (ie next_token and
> parse method)? I vaguely recall that there might be, but can't
> look it up right now.

Well, kind of, but we're both doing the same thing.  Most of the examples
I've seen preparse the entire source string into an array, and then just
pop tokens off the stack.  I'm using the 'next_token' routine to actually
collect the tokens and return them.

So what I'm doing is (theoretically) functionally equivalent, I'm just
doing the split-into-tokens inside next_token instead of inside parse,
which seems to make a bit more sense to me.

However, I'll try converting to stacking the text into an array of tokens
and see what happens.  I don't understand how that could solve the
problem, but that doesn't mean it won't.

Luke

-- 
A great many people think they are thinking when they are merely
rearranging their prejudices.
                -- William James