From: "James M. Lawrence" <quixoticsycophant@...>
Date: 2011-01-28T10:54:24+09:00
Subject: [ruby-core:34921] Re: Towards a standardized AST for Ruby code

Magnus Holm:
> * ParseTree. 1.8. Only parses 1.8 code (including
>   internal). UnifiedRuby AST.
> * RubyParser. Everywhere. Only parses 1.8 code. UnifiedRuby AST.
> * Ripper. 1.9 / MacRuby. Only parses 1.9 code. Event based.
> * Melbourne. C-extension. Only parses 1.8 code. Rubinius AST.
> * JRuby's parser. JRuby. Parses both 1.8 and most 1.9 code. JRuby
>   AST.
> * RedParse. Everywhere. Parses both 1.8 and some 1.9 code. RedParse
>   AST. Not used as much as the other tools.

Yes, this is vexing. I worked around it by creating parser
plug-ins. In retrospect it was an over-orchestrated solution to a
problem that should be fixed elsewhere.

> ## Suggested plan
>
> 1. Design and standardize an AST together with a test suite (based
> on current parsers' test suites)
> 2. Write converters for all the parsers above
> 3. Provide a gem with all of the converters which automatically
> chooses the best internal parser on each implementation.
> 4. Make sure everyone targets this meta-parser instead of custom AST
> provided by other parsers.
> 5. *If* this gets traction, each implementation can *then* provide
> their own Ruby::Parser class and we can slowly deprecate the gem.
>
> ## How should we proceed with this?

Converters seem somewhat impractical and a long way off, whereas
RubyParser is close--it just needs some 1.9 syntax.

A converter does make sense as an optimization. If a Ruby
implementation stores the internal AST of a method (presumably some
internal AST data exists before it gets converted into bytecode), it
would certainly be faster to translate a single method's AST than to
re-parse the file whence it came with RubyParser. Also remember the
load time for RubyParser is significant.

A converter makes the most sense when it is working on stored internal
AST data, preventing a needless re-parsing of the same code by an
external tool. But a converter would not be especially useful if no
internal AST is present--it's faster but it's still stupidly
unoptimized: code is parsed twice, and parsing is expensive. Might as
well use RubyParser.

It seems clear that RubyParser should come first. When it is fairly
complete, and when an implementation supports storing AST data, that
implementation can provide an optimized RubyParser (a converter).

Whether to rename RubyParser or where to put its hypothetical
optimized implementations are relatively small technical issues which
can be handled when the time comes, after RubyParser is working.

Regarding live ASTs, can't this be prototyped with RubyParser?

  require 'live_ast'
  require 'pp'

  class Greet
    def hello
      puts "hello, world"
    end
  end

  pp Greet.instance_method(:hello).ast

output:

  s(:defn,
   :hello,
   s(:args),
   s(:scope,
    s(:block, s(:call, nil, :puts, s(:arglist, s(:str, "hello, world"))))))

http://github.com/quix/live_ast

It also works with define_method and with definitions inside eval.
109 lines of code, comments and whitespace included.