From: Caleb Clausen <vikkous@...>
Date: 2009-05-31T01:41:04+09:00
Subject: Re: Pythonic indentation (or: beating a dead horse)

On 5/30/09, Tony Arcieri <tony@medioh.com> wrote:
> had to offer was some script he didn't write that did some very much
> informal and complicated regex munging to back up his claims.
>
> It has been interesting to see the alternantive formally decidable solutions
> proposed here for the lexer, and I'm interested in possibly incorporating

It's curious to me that you use phrases like 'formally decidable'. I'm
not sure this is really the best description. To my mind, the output
of pyrb.rb was perfectly decidable, and it was equally clear that it
had bugs. Fundamental bugs, that could be chased around but not really
fixed except by complete rewrite. It's a buggy approach because it
doesn't do what you're supposed to do in that kind of automatic
munger: actually tokenize the input stream.

Examples:

  begin # :
     foo
     bar
     baz

  begin p ":
    some more string here"
    foo
    bar

> them into Reia.  In fact I'd be interested in a better description of the
> lexer behavior, although I see it's been released so I suppose I could
> always check out the source myself.

RubyLexer is large and ugly, so I don't envy anyone trying to review
it. It's organized in a straightforward imperative manner, which is
the problem; hand-coded lexers are painful. Also, ruby syntax is very
complicated. If you want to know about something more specific about
its internals, please ask me.