[ruby-core:100946] [Ruby master Feature#17276] Ripper stops tokenizing after keyword as a method parameter

Issue #17276 has been updated by no6v (Nobuhiro IMAI).


> I would prefer `Ripper.{lex,tokenize}` returning fully parsed tokens.

pull request: https://github.com/ruby/ruby/pull/3791


----------------------------------------
Feature #17276: Ripper stops tokenizing after keyword as a method parameter
https://bugs.ruby-lang.org/issues/17276#change-88601

* Author: no6v (Nobuhiro IMAI)
* Status: Closed
* Priority: Normal
----------------------------------------
Although these are obviously syntax errors at this time, the following
codes cannot be tokenized correctly by `Ripper.tokenize`.

```
$ cat src.rb
def req(true) end
def opt(true=0) end
def rest(*true) end
def keyrest(**true) end
def block(&true) end
->true{}
->true=0{}
->*true{}
->**true{}
->&true{}
$ ruby -rripper -vlne 'p Ripper.tokenize($_)' src.rb
ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux]
["def", " ", "req", "(", "true", ")"]
["def", " ", "opt", "(", "true", "=", "0", ")"]
["def", " ", "rest", "(", "*", "true", ")"]
["def", " ", "keyrest", "(", "**", "true", ")"]
["def", " ", "block", "(", "&", "true", ")"]
["->", "true", "{"]
["->", "true", "=", "0", "{"]
["->", "*", "true", "{"]
["->", "**", "true", "{"]
["->", "&", "true", "{"]
```

`end` and `}` are not shown in result.

This seems to prevent `irb` from determining the continuity of the input.
See: https://github.com/ruby/irb/issues/38




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>