From: "ccmywish (Aoran Zeng) via ruby-core" <ruby-core@...>
Date: 2023-05-24T09:34:37+00:00
Subject: [ruby-core:113619] [Ruby master Bug#18738] IRB can't recognize heredoc after words

Issue #18738 has been updated by ccmywish (Aoran Zeng).





This has been fixed. Please close it.



----------------------------------------

Bug #18738: IRB can't recognize heredoc after words

https://bugs.ruby-lang.org/issues/18738#change-103263



* Author: ccmywish (Aoran Zeng)

* Status: Open

* Priority: Normal

* ruby -v: ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x86_64-linux]

* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

----------------------------------------

My irb_info

```ruby

irb(main):001:0> irb_info

=> 

Ruby version: 3.1.1                            

IRB version: irb 1.4.1 (2021-12-25)            

InputMethod: ReidlineInputMethod with Reline 0.3.1

RUBY_PLATFORM: x86_64-linux                    

LANG env: en_US.UTF-8                          

East Asian Ambiguous Width: 1  

```





See the code below please.



```ruby

a, b = <<EOF, %w[ hello

thank you

ruby devs

EOF

world

]

p a

p b

```

This works well if you save it to a file, and run with `ruby xxx.rb`. The results are here:

```ruby

"thank you\nruby devs\n"

["hello", "world"]

```





But when you type it to `irb`���the code will not end, and you will get:

```ruby

��� irb

irb(main):001:0] a, b = <<EOF, %w[ hello

irb(main):002:0] thank you

irb(main):003:0] ruby devs

irb(main):004:0] EOF

irb(main):005:0] world

irb(main):006:0" ]

irb(main):007:-" 

irb(main):008:0" 

irb(main):009:0" 

irb(main):010:0" 

```



I found this issue when I read the mruby source code. in mruby, the token after the first line's `hello` should be `tHD_LITERAL_DELIM`. But in CRuby, there's no this token. I tried to dump CRuby's parser state, find that just after reading `<<EOF`���it will directly recognize the whole token 'thank you\nruby\devs'. So, I think this may not be the bug of Ripper, but how IRB called Ripper using its `ruby-lex` line by line.



For your convenience, you can see the parser state.



```

Stack now 0 2 82 341

Entering state 580

Next token is token "string literal" (1.7-1.12: )

Shifting token "string literal" (1.7-1.12: )

Entering state 60

Reducing stack by rule 613 (line 4830):

-> $$ = nterm string_contents (1.12-1.12: )

Stack now 0 2 82 341 580 60

Entering state 301

Reading a token: Next token is token "literal content" (1.12-1.12: "thank you\nruby devs\n")

Shifting token "literal content" (1.12-1.12: "thank you\nruby devs\n")

Entering state 507

Reducing stack by rule 619 (line 4926):

   $1 = token "literal content" (1.12-1.12: "thank you\nruby devs\n")

-> $$ = nterm string_content (1.12-1.12: )

Stack now 0 2 82 341 580 60 301

Entering state 511

Reducing stack by rule 614 (line 4840):

   $1 = nterm string_contents (1.12-1.12: )

   $2 = nterm string_content (1.12-1.12: )

-> $$ = nterm string_contents (1.12-1.12: )

Stack now 0 2 82 341 580 60

Entering state 301

Reading a token: 

lex_state: BEG -> END at line 7453

Next token is token "terminator" (1.12-1.12: )

Shifting token "terminator" (1.12-1.12: )

Entering state 512

Reducing stack by rule 596 (line 4693):

   $1 = token "string literal" (1.7-1.12: )

   $2 = nterm string_contents (1.12-1.12: )

   $3 = token "terminator" (1.12-1.12: )

-> $$ = nterm string1 (1.7-1.12: )

Stack now 0 2 82 341 580

Entering state 109

Reducing stack by rule 594 (line 4683):

   $1 = nterm string1 (1.7-1.12: )

-> $$ = nterm string (1.7-1.12: )

Stack now 0 2 82 341 580

Entering state 108

Reading a token: 

lex_state: END -> BEG|LABEL at line 9814

Next token is token ',' (1.12-1.13: )

```









-- 

https://bugs.ruby-lang.org/

 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/