[ruby-core:33696] Re: Towards a standardized AST for Ruby code

On Dec 12, 2010, at 08:46 , Charles Oliver Nutter wrote:

> For the record, JRuby will never freeze its AST. We have made changes
> to it (for various reasons) at roughly the same rate since I started
> working on the project in 2005. It's likely changes will continue to
> happen. So any "standard" AST should be produced by a separate offline
> parser.

It certainly doesn't have to be an offline parser. It just needs a translation layer like UnifiedRuby for ParseTree.

> I will cast my vote, whatever it's worth, for just enhancing and
> improving Ryan's RubyParser. It already parses 1.8 code very well, and
> being racc-based it's closest in structure to the canonical C Ruby
> parser, which will make enhancing and improving it far easier than
> one-offs. It also has an extensive test suite, backward-compatibility
> with ParseTree, and reasonable performance.
> 
> Converters will always suffer from changes that happen to each impl's
> AST, and will need constant maintenance. RubyParser needs no such
> maintenance beyond adding new syntax features as they are added to
> Ruby proper.

But if that converter ships with the impl, it is easy to keep in sync. It only took me an hour or so everytime MRI changed something and my test suite failed. Since the person implementing the change knows it best, it'd be trivial for them to change the converter at the same time (most of that hour was me figuring out what the change was and why... the code part is the easy part).

> Native parsers will be faster; but re-walking the native ASTs and
> converting them to a standard format could be nearly as "slow" as a
> pure-Ruby parser.

Totally disagree. "could" is speculative and I have plenty of proof that it is false. Manipulating an AST is nearly free compared to tracking all the BS involved with lexing and parsing ruby.

> The only down side at the moment is that RubyParser does not support
> 1.9 syntax. Ryan has a bounty on adding such support (or at least
> adding tests, which may encourage him to add such support) and I'll
> toss in a few bucks as well.
> http://blog.zenspider.com/2010/12/bounty-ruby-parser-needs-19-lo.html

And that bounty still stands... but I haven't gotten any nibbles yet. :(

> I'd also love to see someone implement a native racc backend for
> JRuby...but that's a separate discussion :)

I assume you're running the pure ruby racc runtime in jruby? How's the performance? I was talking to Alan at gemstone about optimizations to it that could speed it up a fair amount but I never got time to implement them before I left EY.