From: daz <dooby@...10.karoo.co.uk>
Date: 2005-11-05T01:12:09+09:00
Subject: Re: YACC (Was: Re: ruby-dev summary 27393-27541)


Eric Mahurin wrote:
> On 11/3/05, nobu wrote:
> > Christian Neukirchen wrote in [ruby-talk:163903]:
> >
> > > I don't think the real problem is the real grammar, that part of
> > > parse.y looks like the easier one (and rather readable) to me. The
> > > problem is the lexer-parser communication, think heredocs, %q[] etc.
> > > There is no way to express that in BNF.
> >
> > Exactly. It's a headache.
> >
>
>
> I would love to see the ruby syntax refactored and simplified -
> especially with regards to the lexer state.


A-m-e-n.


Selection from thread:
  [Context sensitive scanner ?] - comp.compilers (1997)


  (1997-11-23) - Mikael Pettersson
  http://compilers.iecc.com/comparch/article/97-11-127

    "Transformation scanner"    [!]


  (1997-11-24) - Scott Stanchfield
  http://compilers.iecc.com/comparch/article/97-11-141


  (1997-11-28) - Chris F Clark
  http://compilers.iecc.com/comparch/article/97-11-150

   "Unfortunately, most parsing systems don't encourage
    thinking about parsing as sequences of transformations.
    Instead they promote the view of one lexer and one parser."


  (1997-11-28) - Henry Spencer
  http://compilers.iecc.com/comparch/article/97-11-151

   "... the screener ..."


  (( http://compilers.iecc.com/comparch/article/97-11-167 ))


Note:  In the next post, Chris mentions "Scannerless Parsing".
I read a lot of papers on this subject and there was always
a feeling that this wouldn't be suitable for a transitional
solution for Ruby. [IMHO]


  (1997-12-05) - Chris Clark USG
  http://compilers.iecc.com/comparch/article/97-12-016

   "The point has been clearly made that feedback from the
    parser to the lexer (using lexer states controlled from
    the parser) is not a good idea when avoidable."

   "However, at the same time using (manually controlled)
    state in a lexer to control which tokens are returned
    should also be avoided when possible."


  (( http://compilers.iecc.com/comparch/article/97-12-043  ))


I hope someone might be inspired.

daz