[ruby-core:100885] [Ruby master Feature#17276] Ripper stops tokenizing after keyword as a method parameter

Issue #17276 has been updated by jeremyevans0 (Jeremy Evans).

Backport deleted (2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN)
ruby -v deleted (ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux])
Tracker changed from Bug to Feature

Ripper records errors, but `Ripper.tokenize` and `Ripper.lex` cannot return them.  Here's how you can handle errors with Ripper (for tokenize, lex is similar):

```ruby
require 'ripper'
r = Ripper::Lexer.new('def req(true) end', 'a', 1)
p r.tokenize
# => ["def", " ", "req", "(", "true", ")"]
p r.errors
# => [#<Ripper::Lexer::Elem: on_parse_error@1:8:END: "true": syntax error, unexpected `true', expecting ')'>]
```

This is not a bug, it is a limitation of the API for `Ripper.tokenize` and `Ripper.lex`. Changing `Ripper.tokenize` and `Ripper.lex` to raise an exception is possible, but would break backwards compatibility.

Maybe we could support keyword arguments in `Ripper.lex` and `Ripper.tokenize` to raise SyntaxError for errors?  Here's a pull request for that approach: https://github.com/ruby/ruby/pull/3774

----------------------------------------
Feature #17276: Ripper stops tokenizing after keyword as a method parameter
https://bugs.ruby-lang.org/issues/17276#change-88533

* Author: no6v (Nobuhiro IMAI)
* Status: Open
* Priority: Normal
----------------------------------------
Although these are obviously syntax errors at this time, the following
codes cannot be tokenized correctly by `Ripper.tokenize`.

```
$ cat src.rb
def req(true) end
def opt(true=0) end
def rest(*true) end
def keyrest(**true) end
def block(&true) end
->true{}
->true=0{}
->*true{}
->**true{}
->&true{}
$ ruby -rripper -vlne 'p Ripper.tokenize($_)' src.rb
ruby 3.0.0dev (2020-10-21T00:24:47Z master da25affdac) [x86_64-linux]
["def", " ", "req", "(", "true", ")"]
["def", " ", "opt", "(", "true", "=", "0", ")"]
["def", " ", "rest", "(", "*", "true", ")"]
["def", " ", "keyrest", "(", "**", "true", ")"]
["def", " ", "block", "(", "&", "true", ")"]
["->", "true", "{"]
["->", "true", "=", "0", "{"]
["->", "*", "true", "{"]
["->", "**", "true", "{"]
["->", "&", "true", "{"]
```

`end` and `}` are not shown in result.

This seems to prevent `irb` from determining the continuity of the input.
See: https://github.com/ruby/irb/issues/38




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>