[ruby-core:93530] [Ruby master Bug#11485] Ripper parser generates nil tokens for ignored_nl event during parse errors

Issue #11485 has been updated by nobu (Nobuyoshi Nakada).


> `[ruby-core:<unknown>]`

This seems not having sent to the ML, and it happens sometimes, but I'm not sure why.

----------------------------------------
Bug #11485: Ripper parser generates nil tokens for ignored_nl event during parse errors
https://bugs.ruby-lang.org/issues/11485#change-79092

* Author: lsegal (Loren Segal)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
* ruby -v: 2.3.0-dev
* Backport: 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED
----------------------------------------
This seems to be a regression in the Ripper parser where the `on_ignored_nl` event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:

~~~ruby
require 'ripper'

class Parser < Ripper
  def on_ignored_nl(*args) p args end
  def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end

Parser.new(DATA, 'stdin').parse

__END__
something # comment
...
~~~

Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with `nil` as the argument. I'm not sure if it's specific to the `...` token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.

Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):

~~~
~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
ERROR! syntax error, unexpected ..., expecting end-of-input

~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input
~~~

I would not expect the token parameter to ever be `nil`, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.

(This behavior is present in 2.2.0p95 as well.)

---Files--------------------------------
ripper-ignored-nl-nil.patch (1.42 KB)


-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>