[ruby-core:70581] [Ruby trunk - Bug #11485] [Open] Ripper parser generates nil tokens for ignored_nl event during parse errors

Issue #11485 has been reported by Loren Segal.

----------------------------------------
Bug #11485: Ripper parser generates nil tokens for ignored_nl event during parse errors
https://bugs.ruby-lang.org/issues/11485

* Author: Loren Segal
* Status: Open
* Priority: Normal
* Assignee: 
* ruby -v: 2.3.0-dev
* Backport: 2.0.0: DONTNEED, 2.1: UNKNOWN, 2.2: UNKNOWN
----------------------------------------
This seems to be a regression in the Ripper parser where the `on_ignored_nl` event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:

~~~
require 'ripper'

class Parser < Ripper
  def on_ignored_nl(*args) p args end
  def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end

Parser.new(DATA, 'stdin').parse

__END__
something # comment
...
~~~

Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with `nil` as the argument. I'm not sure if it's specific to the `...` token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.

Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):

~~~
~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
ERROR! syntax error, unexpected ..., expecting end-of-input

~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input
~~~

I would not expect the token parameter to ever be `nil`, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.

(This behavior is present in 2.2.0p95 as well.)



-- 
https://bugs.ruby-lang.org/