[ruby-core:33687] Towards a standardized AST for Ruby code

Hey folks,

When it comes to working with Ruby files in Ruby (parsing, analyzing
etc), we definitely have a non-optimal solution right now. There's
plenty of parsers, but almost none work across implementations or are
compatible with each other:

* ParseTree. 1.8. Only parses 1.8 code (including internal). UnifiedRuby AST.
* RubyParser. Everywhere. Only parses 1.8 code. UnifiedRuby AST.
* Ripper. 1.9 / MacRuby. Only parses 1.9 code. Event based.
* Melbourne. C-extension. Only parses 1.8 code. Rubinius AST.
* JRuby's parser. JRuby. Parses both 1.8 and most 1.9 code. JRuby AST.
* RedParse. Everywhere. Parses both 1.8 and some 1.9 code. RedParse
AST. Not used as much as the other tools.

I'm proposing that we design and standardize a single AST which is
intended to be used by tools that works with Ruby syntax (Flog, Flay,
Roodi, Reek etc). Parsers can then produce this AST either directly
from the parser itself, or through a converter from the internal AST
to the standardized AST. Please be aware that this AST does not
mandate how the parsers/implementations should structure their
internal Ruby code; it's only a convention for how to work with Ruby
within Ruby in order to make it easier for all parts.

A long term goal is that every implementation should implement a
Ruby::Parser class which will parse the code with the internal parser,
and yet produce the same AST everywhere.

## Suggested plan

1. Design and standardize an AST together with a test suite (based on
current parsers' test suites)
2. Write converters for all the parsers above
3. Provide a gem with all of the converters which automatically
chooses the best internal parser on each implementation.
4. Make sure everyone targets this meta-parser instead of custom AST
provided by other parsers.
5. *If* this gets traction, each implementation can *then* provide
their own Ruby::Parser class and we can slowly deprecate the gem.

## How should we proceed with this?

First a few questions:

  Everyone, do you also see the value of this, or is it only me?
  Ruby core team, is there a way to make this an "official" AST?
  matz, what are your thoughts on a Ruby::Parser class?

Where should we continue discussing this AST? ruby-core? A new mailing
list? Somewhere else?

Also, if you know about someone who has either written a parser or a
tool that depends on a parser, could you please point them to this
discussion?

Thanks for your consideration,
Magnus Holm