From: "Florian Groß" <florgro@...>
Date: 2005-04-26T09:06:42+09:00
Subject: Re: announcing RubyLexer 0.6.0

vikkous wrote:

>>Advisory tokens
> So you want to match the 'then' with it's owning 'if'? That's not
> something I've had to do yet, but it shouldn't be hard... How's this
> for an interface:
> I can add a new method to the Token class, let's call it match_id for
> now. Every time there's a token like 'if', '(', 'begin', that starts a
> nested context, the match_id of that token will be set to a unique
> value. When the corresponding 'end' or ')' comes along, it will have a
> match_id with the same value as the corresponding context opening
> token. We can easily have 'then' with a match_id corresponding to its
> 'if' as well. This should make it pretty easy to put the pieces
> together again afterward.

It is not so important to match the then to the if to me -- it is just 
important to get the part that comes between the if and the matching 
'then', ':', ';' or newline. I'm not sure if you even need to do it as 
you described -- I thought having a special mode / sub-class lexer which 
emits contextual tokens that are no real tokens would already do this 
fairly well while also being reasonably simple. So

   if condition then action end

would produce a token stream similar to

   # pardon me if my way of representing this is not at all compatible
   # with RubyLexer's design -- I need to get familiar with it soon
   [KeyWord['if'], IfConditionStart, VariableOrMethod['condition'],
    IfConditionEnd, KeyWord['then'], IfActionStart,
    VariableOrMethod['action'], IfActionEnd, KeyWord['end']]

And I think that that would be easier to analyze than the non-annotated 
token stream. Of course you would still have to do nesting counting to 
be able to extract the sections, but I think that would be reasonable 
for simplicity's sake.

>>most languages require you to keep some context
>>for actually tokenizing them.
> You can say that again. The amount of extra (non-lexical, strictly
> speaking) work to get RubyLexer working was phenomenal. You wouldn't
> believe all the squirrelly little cases. It makes the language easy to
> use, but hard to process programatically. Given the choice, I'd like to
> find a different way next time. If there could be one tool that does
> both at once... I don't know what that would look like. Reg might be
> able to do both, but in separate stages.

Hm, why is that? Could it not use the rules it uses for parsing for 
one-token-at-a-time-ahead lexing?

I'm not sure whether not having lexing and parsing more unified has 
benefits or downsides with your approach. I guess I will just have to 
write a Joy interpreter using all this. Do you think that that can 
already be done or is there features missing that would make it wise to 
delay this further?

> [Integrating the lexer with IRB]
> It's necessary because I want to. Because irb's lexer is sometimes
> wrong, and freaks like me who use irb to explore the syntax get fooled
> sometimes. Because irb could use it to colorize input and output.
> (Maybe it's current lexer would serve for the last purpose...)

Heh, you must have been reading old postings of mine. IRB doing syntax 
highlighting as you type has been on my wish list for a while.

That aside, I think I misunderstood you. I originally thought you wanted 
to integrate IRB's lexer with your tool chain, but it appears that you 
want to instead integrate your lexer with IRB.

I think such things are possible fairly easily with Ruby -- after all 
you just have to emulate the method interfaces of the part you want to 
replace and swap it out.

I have done similar things with ruby-breakpoint where I overwrite parts 
of IRB so that it can be split into a client and a server. The server 
part does not use STDIN/STDOUT which means I can then use IRB for 
debugging CGI applications and pretty much everything else as well.

> [pre.rb] 
> I got a little way through it... aside from the unique use of
> whitespace, my big problem so far is handling the dos-style newlines. I
> handle common cases of it now, but pre is anything but common. Are you
> a windows person, or did you do that just to be more deviant and make
> my life difficult? :)

Heh, I'm really one of them Windows users and mostly happy so far though 
I think I would not object against a free switch to Mac OS X if the 
opportunity ever turned up.

Had I wanted to make this yet more difficult I would have mixed multiple 
styles of newlines. ;)

Now I actually do wonder if using CRLF instead of LF does anything 
special to newline-delimited literals on any platforms.