From: "tompng (tomoya ishida) via ruby-core" <ruby-core@...>
Date: 2024-10-22T10:57:47+00:00
Subject: [ruby-core:119587] [Ruby master Bug#20809] Ripper fails to parse/tokenize nested heredocs within embexpr

Issue #20809 has been reported by tompng (tomoya ishida).

----------------------------------------
Bug #20809: Ripper fails to parse/tokenize nested heredocs within embexpr
https://bugs.ruby-lang.org/issues/20809

* Author: tompng (tomoya ishida)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-10-22T09:04:30Z master 54065f3b7b) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
In this example, Ripper wrongly concats `"s333\n"` and `"s4444"`.
~~~ruby
require 'ripper'

code1 = '<<HEREDOC1
s1
#{<<HEREDOC2}s333
s22
HEREDOC2
s4444#{55555}
HEREDOC1'

p Ripper.tokenize(code1)
# => ["<<HEREDOC1", "\n", "s1\n", "\#{", "<<HEREDOC2", "}", "s333\ns4444", "s22\n", "HEREDOC2\n", "\#{", "55555", "}", "\n", "HEREDOC1"]
~~~

In this example, result of `Ripper.sexp` is wrong. `"s333"` disappears.
~~~ruby
code2 = '<<HEREDOC1
s1
#{<<HEREDOC2}s333
s22
HEREDOC2
s4444
HEREDOC1'

p eval(code2) # => "s1\ns22\ns333\ns4444\n"
p Ripper.sexp(code2)
# => 
[:program,
 [[:string_literal,
   [:string_content,
    [:@tstring_content, "s1\n", [2, 0]],
    [:string_embexpr,
     [[:string_literal,
       [:string_content,
        [:@tstring_content, "s22\n", [4, 0]]]]]],
    [:@tstring_content, "s4444\n", [6, 0]]]]]]
~~~



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/