From: "matz (Yukihiro Matsumoto)" <noreply@...>
Date: 2022-11-07T03:07:33+00:00
Subject: [ruby-core:110631] [Ruby master Feature#19070] Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods

Issue #19070 has been updated by matz (Yukihiro Matsumoto).


Sounds OK.

Matz.


----------------------------------------
Feature #19070: Enhance keep_tokens option for RubyVM::AbstractSyntaxTree parsing methods
https://bugs.ruby-lang.org/issues/19070#change-99967

* Author: yui-knk (Kaneko Yuichiro)
* Status: Open
* Priority: Normal
----------------------------------------
# Background

Implementation for Language Server Protocol (LSP) sometimes needs token information. For example both `m(1)` and `m(1, )` has same AST structure other than node locations then it's impossible to check the existence of `,` from AST. However in later case, it might be better to suggest variables list for the second argument. Token information is important for such case.

# Example

```ruby
require "pp"

node = RubyVM::AbstractSyntaxTree.parse(<<~STR, keep_tokens: true)
def m(a, b = 1, *rest, &block)
end

m(1, )
STR
defn  = node.children[2].children[0]
fcall = node.children[2].children[1]

puts "defn.tokens"
pp defn.tokens
puts "\n\n"

puts "fcall.tokens"
pp fcall.tokens
puts "\n\n"

puts defn.tokens.map{_1[2]}.join
puts fcall.tokens.map{_1[2]}.join
```

shows below, where token is `[sequence_id, token_type, token_string, [first_line, first_column, last_line, last_column]]`

```ruby
defn.tokens
[[0, :kw, "def", [1, 0, 1, 3]],
 [1, :sp, " ", [1, 3, 1, 4]],
 [2, :ident, "m", [1, 4, 1, 5]],
 [3, :lparen, "(", [1, 5, 1, 6]],
 [4, :ident, "a", [1, 6, 1, 7]],
 [5, :comma, ",", [1, 7, 1, 8]],
 [6, :sp, " ", [1, 8, 1, 9]],
 [7, :ident, "b", [1, 9, 1, 10]],
 [8, :sp, " ", [1, 10, 1, 11]],
 [9, :op, "=", [1, 11, 1, 12]],
 [10, :sp, " ", [1, 12, 1, 13]],
 [11, :int, "1", [1, 13, 1, 14]],
 [12, :comma, ",", [1, 14, 1, 15]],
 [13, :sp, " ", [1, 15, 1, 16]],
 [14, :op, "*", [1, 16, 1, 17]],
 [15, :ident, "rest", [1, 17, 1, 21]],
 [16, :comma, ",", [1, 21, 1, 22]],
 [17, :sp, " ", [1, 22, 1, 23]],
 [18, :op, "&", [1, 23, 1, 24]],
 [19, :ident, "block", [1, 24, 1, 29]],
 [20, :rparen, ")", [1, 29, 1, 30]],
 [21, :ignored_nl, "\n", [1, 30, 1, 31]],
 [22, :kw, "end", [2, 0, 2, 3]]]


fcall.tokens
[[25, :ident, "m", [4, 0, 4, 1]],
 [26, :lparen, "(", [4, 1, 4, 2]],
 [27, :int, "1", [4, 2, 4, 3]],
 [28, :comma, ",", [4, 3, 4, 4]],
 [29, :sp, " ", [4, 4, 4, 5]],
 [30, :rparen, ")", [4, 5, 4, 6]]]


def m(a, b = 1, *rest, &block)
end
m(1, )
```

# Interface

* Add `keep_tokens` option for `RubyVM::AbstractSyntaxTree.parse`, `.parse_file` and `.of`
* Add `RubyVM::AbstractSyntaxTree::Node#tokens` which returns tokens for the node including tokens for descendants nodes.
* Add `RubyVM::AbstractSyntaxTree::Node#all_tokens` which returns all tokens for the input script regardless the receiver node. 

# Implementation

https://github.com/yui-knk/ruby/tree/cst5




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>