[ruby-core:108274] [Ruby master Bug#18738] IRB can't recognize heredoc after words

Issue #18738 has been updated by ccmywish (Aoran Zeng).


kaiquekandykoga (Kaíque Koga) wrote in #note-1:
> I think it can be interesting to open an issue at https://github.com/ruby/irb.

Now here: https://github.com/ruby/irb/issues/361

----------------------------------------
Bug #18738: IRB can't recognize heredoc after words
https://bugs.ruby-lang.org/issues/18738#change-97295

* Author: ccmywish (Aoran Zeng)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x86_64-linux]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
My irb_info
```ruby
irb(main):001:0> irb_info
=> 
Ruby version: 3.1.1                            
IRB version: irb 1.4.1 (2021-12-25)            
InputMethod: ReidlineInputMethod with Reline 0.3.1
RUBY_PLATFORM: x86_64-linux                    
LANG env: en_US.UTF-8                          
East Asian Ambiguous Width: 1  
```


See the code below please.

```ruby
a, b = <<EOF, %w[ hello
thank you
ruby devs
EOF
world
]
p a
p b
```
This works well if you save it to a file, and run with `ruby xxx.rb`. The results are here:
```ruby
"thank you\nruby devs\n"
["hello", "world"]
```


But when you type it to `irb`，the code will not end, and you will get:
```ruby
❯ irb
irb(main):001:0] a, b = <<EOF, %w[ hello
irb(main):002:0] thank you
irb(main):003:0] ruby devs
irb(main):004:0] EOF
irb(main):005:0] world
irb(main):006:0" ]
irb(main):007:-" 
irb(main):008:0" 
irb(main):009:0" 
irb(main):010:0" 
```

I found this issue when I read the mruby source code. in mruby, the token after the first line's `hello` should be `tHD_LITERAL_DELIM`. But in CRuby, there's no this token. I tried to dump CRuby's parser state, find that just after reading `<<EOF`，it will directly recognize the whole token 'thank you\nruby\devs'. So, I think this may not be the bug of Ripper, but how IRB called Ripper using its `ruby-lex` line by line.

For your convenience, you can see the parser state.

```
Stack now 0 2 82 341
Entering state 580
Next token is token "string literal" (1.7-1.12: )
Shifting token "string literal" (1.7-1.12: )
Entering state 60
Reducing stack by rule 613 (line 4830):
-> $$ = nterm string_contents (1.12-1.12: )
Stack now 0 2 82 341 580 60
Entering state 301
Reading a token: Next token is token "literal content" (1.12-1.12: "thank you\nruby devs\n")
Shifting token "literal content" (1.12-1.12: "thank you\nruby devs\n")
Entering state 507
Reducing stack by rule 619 (line 4926):
   $1 = token "literal content" (1.12-1.12: "thank you\nruby devs\n")
-> $$ = nterm string_content (1.12-1.12: )
Stack now 0 2 82 341 580 60 301
Entering state 511
Reducing stack by rule 614 (line 4840):
   $1 = nterm string_contents (1.12-1.12: )
   $2 = nterm string_content (1.12-1.12: )
-> $$ = nterm string_contents (1.12-1.12: )
Stack now 0 2 82 341 580 60
Entering state 301
Reading a token: 
lex_state: BEG -> END at line 7453
Next token is token "terminator" (1.12-1.12: )
Shifting token "terminator" (1.12-1.12: )
Entering state 512
Reducing stack by rule 596 (line 4693):
   $1 = token "string literal" (1.7-1.12: )
   $2 = nterm string_contents (1.12-1.12: )
   $3 = token "terminator" (1.12-1.12: )
-> $$ = nterm string1 (1.7-1.12: )
Stack now 0 2 82 341 580
Entering state 109
Reducing stack by rule 594 (line 4683):
   $1 = nterm string1 (1.7-1.12: )
-> $$ = nterm string (1.7-1.12: )
Stack now 0 2 82 341 580
Entering state 108
Reading a token: 
lex_state: END -> BEG|LABEL at line 9814
Next token is token ',' (1.12-1.13: )
```




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>