From: merch-redmine@... Date: 2019-07-03T20:32:53+00:00 Subject: [ruby-core:93518] [Ruby master Bug#11485] Ripper parser generates nil tokens for ignored_nl event during parse errors Issue #11485 has been updated by jeremyevans0 (Jeremy Evans). File ripper-ignored-nl-nil.patch added This bug is still present on master. Attached is a patch that fixes it, by not calling `dispatch_delayed_token` when ripper is used if the dispatched value is `nil`. ---------------------------------------- Bug #11485: Ripper parser generates nil tokens for ignored_nl event during parse errors https://bugs.ruby-lang.org/issues/11485#change-79079 * Author: lsegal (Loren Segal) * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: 2.3.0-dev * Backport: 2.0.0: DONTNEED, 2.1: UNKNOWN, 2.2: UNKNOWN ---------------------------------------- This seems to be a regression in the Ripper parser where the `on_ignored_nl` event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce: ~~~ require 'ripper' class Parser < Ripper def on_ignored_nl(*args) p args end def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end end Parser.new(DATA, 'stdin').parse __END__ something # comment ... ~~~ Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with `nil` as the argument. I'm not sure if it's specific to the `...` token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those. Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective): ~~~ ~$ ruby -v ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14] ~$ ruby ripper-bug-2.2.0.rb ERROR! syntax error, unexpected ..., expecting end-of-input ~$ ruby -v ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14] ~$ ruby ripper-bug-2.2.0.rb [nil] ERROR! syntax error, unexpected ..., expecting end-of-input ~~~ I would not expect the token parameter to ever be `nil`, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all. (This behavior is present in 2.2.0p95 as well.) ---Files-------------------------------- ripper-ignored-nl-nil.patch (1.42 KB) -- https://bugs.ruby-lang.org/ Unsubscribe: