[ruby-core:33707] Re: Towards a standardized AST for Ruby code

On Sun, Dec 12, 2010 at 6:17 PM, Ryan Davis <[e-mail]> wrote:
> It certainly doesn't have to be an offline parser. It just needs a translation layer like UnifiedRuby for ParseTree.

It is not possible to translate from other formats into Unified's AST.
For example, Rubinius's current bytecode form, YARV's bytecode form,
and JRuby's upcoming IR form.

But an offline parser could do the same work as a live AST if the
source code were kept around. That's far easier to do than mapping AST
and keeping that mapping in sync.

> But if that converter ships with the impl, it is easy to keep in sync. It only took me an hour or so everytime MRI changed something and my test suite failed. Since the person implementing the change knows it best, it'd be trivial for them to change the converter at the same time (most of that hour was me figuring out what the change was and why... the code part is the easy part).

See above. JRuby's AST isn't too far removed from MRI's, but other
in-memory representations are.

>> Native parsers will be faster; but re-walking the native ASTs and
>> converting them to a standard format could be nearly as "slow" as a
>> pure-Ruby parser.
>
> Totally disagree. "could" is speculative and I have plenty of proof that it is false. Manipulating an AST is nearly free compared to tracking all the BS involved with lexing and parsing ruby.

"Could" is not speculative. It should be mathematically obvious that
it depends on the complexity of the mapping required. More mapping =
more complexity, and could easily exceed parsing overhead at some
point if ASTs diverge enough (or if AST is thrown away completely in
favor of a different format that takes a lot more massaging back into
standard form). Unified was based off MRI's AST (right?), so your
evidence is based on mapping from a slightly divergent AST to a
standard one that used to match more closely.

>> I'd also love to see someone implement a native racc backend for
>> JRuby...but that's a separate discussion :)
>
> I assume you're running the pure ruby racc runtime in jruby? How's the performance? I was talking to Alan at gemstone about optimizations to it that could speed it up a fair amount but I never got time to implement them before I left EY.

Pure Ruby, yes. Performance settles in around 50% slower than 1.9
*using the C extension*. I'm not sure if that says something about the
performance of racc (it's slow no matter what) or the performance of
JRuby (ZOMG only 50% slower than C).

- Charlie