From: nobu@... Date: 2019-07-04T07:09:36+00:00 Subject: [ruby-core:93530] [Ruby master Bug#11485] Ripper parser generates nil tokens for ignored_nl event during parse errors Issue #11485 has been updated by nobu (Nobuyoshi Nakada). > `[ruby-core:]` This seems not having sent to the ML, and it happens sometimes, but I'm not sure why. ---------------------------------------- Bug #11485: Ripper parser generates nil tokens for ignored_nl event during parse errors https://bugs.ruby-lang.org/issues/11485#change-79092 * Author: lsegal (Loren Segal) * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: 2.3.0-dev * Backport: 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED ---------------------------------------- This seems to be a regression in the Ripper parser where the `on_ignored_nl` event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce: ~~~ruby require 'ripper' class Parser < Ripper def on_ignored_nl(*args) p args end def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end end Parser.new(DATA, 'stdin').parse __END__ something # comment ... ~~~ Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with `nil` as the argument. I'm not sure if it's specific to the `...` token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those. Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective): ~~~ ~$ ruby -v ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14] ~$ ruby ripper-bug-2.2.0.rb ERROR! syntax error, unexpected ..., expecting end-of-input ~$ ruby -v ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14] ~$ ruby ripper-bug-2.2.0.rb [nil] ERROR! syntax error, unexpected ..., expecting end-of-input ~~~ I would not expect the token parameter to ever be `nil`, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all. (This behavior is present in 2.2.0p95 as well.) ---Files-------------------------------- ripper-ignored-nl-nil.patch (1.42 KB) -- https://bugs.ruby-lang.org/ Unsubscribe: