[ruby-core:18863] Re: tokenizing regular expressions when passed as method params

On Sep 24, 2008, at 09:07 , Seth Dillingham wrote:

>  title.gsub /some word "/, ' '


You'll want to look at ruby's parse.y either directly or indirectly.  
Also note that ruby -w complains about the ambiguity.


% parse_tree_show
title.gsub /some word "/, ' '
-:1: warning: ambiguous first argument; put parentheses or even spaces
s(:call,
  s(:vcall, :title),
  :gsub,
  s(:array, s(:lit, /some word "/), s(:str, " ")))

% ruby -y
s=''; s.gsub /some word "/, ' '
Starting parse
Entering state 0
Reducing stack by rule 1 (line 346):
-> $$ = nterm @1 ()
Stack now 0
Entering state 2
Reading a token: Next token is token tIDENTIFIER ()
Shifting token tIDENTIFIER ()
Entering state 34
Reading a token: Next token is token '=' ()
Reducing stack by rule 419 (line 2188):
    $1 = token tIDENTIFIER ()
-> $$ = nterm variable ()
Stack now 0 2
Entering state 90
Next token is token '=' ()
Reducing stack by rule 83 (line 842):
    $1 = nterm variable ()
-> $$ = nterm lhs ()
Stack now 0 2
Entering state 73
Next token is token '=' ()
Shifting token '=' ()
Entering state 315
Reading a token: Next token is token tSTRING_BEG ()
Shifting token tSTRING_BEG ()
Entering state 53
Reducing stack by rule 396 (line 2071):
-> $$ = nterm string_contents ()
Stack now 0 2 73 315 53
Entering state 287
Reading a token: Next token is token tSTRING_END ()
Shifting token tSTRING_END ()
Entering state 457
Reducing stack by rule 383 (line 1961):
    $1 = token tSTRING_BEG ()
    $2 = nterm string_contents ()
    $3 = token tSTRING_END ()
-> $$ = nterm string1 ()
Stack now 0 2 73 315
Entering state 82
Reducing stack by rule 381 (line 1954):
    $1 = nterm string1 ()
-> $$ = nterm string ()
Stack now 0 2 73 315
Entering state 81
Reading a token: Next token is token ';' ()
Reducing stack by rule 380 (line 1941):
    $1 = nterm string ()
-> $$ = nterm strings ()
Stack now 0 2 73 315
Entering state 80
Reducing stack by rule 269 (line 1455):
    $1 = nterm strings ()
-> $$ = nterm primary ()
Stack now 0 2 73 315
Entering state 75
Next token is token ';' ()
Reducing stack by rule 218 (line 1232):
    $1 = nterm primary ()
-> $$ = nterm arg ()
Stack now 0 2 73 315
Entering state 488
Next token is token ';' ()
Reducing stack by rule 174 (line 975):
    $1 = nterm lhs ()
    $2 = token '=' ()
    $3 = nterm arg ()
-> $$ = nterm arg ()
Stack now 0 2
Entering state 74
Next token is token ';' ()
Reducing stack by rule 40 (line 631):
    $1 = nterm arg ()
-> $$ = nterm expr ()
Stack now 0 2
Entering state 64
Next token is token ';' ()
Reducing stack by rule 34 (line 611):
    $1 = nterm expr ()
-> $$ = nterm stmt ()
Stack now 0 2
Entering state 63
Next token is token ';' ()
Reducing stack by rule 6 (line 399):
    $1 = nterm stmt ()
-> $$ = nterm stmts ()
Stack now 0 2
Entering state 62
Next token is token ';' ()
Shifting token ';' ()
Entering state 217
Reducing stack by rule 496 (line 2472):
    $1 = token ';' ()
-> $$ = nterm term ()
Stack now 0 2 62
Entering state 220
Reducing stack by rule 498 (line 2476):
    $1 = nterm term ()
-> $$ = nterm terms ()
Stack now 0 2 62
Entering state 300
Reading a token: Next token is token tIDENTIFIER ()
Shifting token tIDENTIFIER ()
Entering state 34
Reading a token: Next token is token '.' ()
Reducing stack by rule 419 (line 2188):
    $1 = token tIDENTIFIER ()
-> $$ = nterm variable ()
Stack now 0 2 62 300
Entering state 90
Next token is token '.' ()
Reducing stack by rule 430 (line 2201):
    $1 = nterm variable ()
-> $$ = nterm var_ref ()
Stack now 0 2 62 300
Entering state 91
Reducing stack by rule 274 (line 1460):
    $1 = nterm var_ref ()
-> $$ = nterm primary ()
Stack now 0 2 62 300
Entering state 75
Next token is token '.' ()
Reducing stack by rule 326 (line 1728):
    $1 = nterm primary ()
-> $$ = nterm primary_value ()
Stack now 0 2 62 300
Entering state 76
Next token is token '.' ()
Shifting token '.' ()
Entering state 344
Reading a token: Next token is token tIDENTIFIER ()
Shifting token tIDENTIFIER ()
Entering state 524
Reading a token: Next token is token tREGEXP_BEG ()
Reducing stack by rule 480 (line 2444):
    $1 = token tIDENTIFIER ()
-> $$ = nterm operation2 ()
Stack now 0 2 62 300 76 344
Entering state 528
Next token is token tREGEXP_BEG ()
Reducing stack by rule 253 (line 1392):
-> $$ = nterm @8 ()
Stack now 0 2 62 300 76 344 528
Entering state 246
Next token is token tREGEXP_BEG ()
Shifting token tREGEXP_BEG ()
Entering state 55
Reducing stack by rule 398 (line 2081):
-> $$ = nterm xstring_contents ()
Stack now 0 2 62 300 76 344 528 246 55
Entering state 289
Reading a token: Next token is token tSTRING_CONTENT ()
Shifting token tSTRING_CONTENT ()
Entering state 452
Reducing stack by rule 400 (line 2090):
    $1 = token tSTRING_CONTENT ()
-> $$ = nterm string_content ()
Stack now 0 2 62 300 76 344 528 246 55 289
Entering state 456
Reducing stack by rule 399 (line 2084):
    $1 = nterm xstring_contents ()
    $2 = nterm string_content ()
-> $$ = nterm xstring_contents ()
Stack now 0 2 62 300 76 344 528 246 55
Entering state 289
Reading a token: Next token is token tREGEXP_END ()
Shifting token tREGEXP_END ()
Entering state 460
Reducing stack by rule 385 (line 1990):
    $1 = token tREGEXP_BEG ()
    $2 = nterm xstring_contents ()
    $3 = token tREGEXP_END ()
-> $$ = nterm regexp ()
Stack now 0 2 62 300 76 344 528 246
Entering state 84
Reducing stack by rule 271 (line 1457):
    $1 = nterm regexp ()
-> $$ = nterm primary ()
Stack now 0 2 62 300 76 344 528 246
Entering state 75
Reading a token: Next token is token ',' ()
Reducing stack by rule 218 (line 1232):
    $1 = nterm primary ()
-> $$ = nterm arg ()
Stack now 0 2 62 300 76 344 528 246
Entering state 235
Next token is token ',' ()
Reducing stack by rule 219 (line 1238):
    $1 = nterm arg ()
-> $$ = nterm arg_value ()
Stack now 0 2 62 300 76 344 528 246
Entering state 236
Next token is token ',' ()
Reducing stack by rule 263 (line 1430):
    $1 = nterm arg_value ()
-> $$ = nterm args ()
Stack now 0 2 62 300 76 344 528 246
Entering state 239
Next token is token ',' ()
Shifting token ',' ()
Entering state 406
Reading a token: Next token is token tSTRING_BEG ()
Shifting token tSTRING_BEG ()
Entering state 53
Reducing stack by rule 396 (line 2071):
-> $$ = nterm string_contents ()
Stack now 0 2 62 300 76 344 528 246 239 406 53
Entering state 287
Reading a token: Next token is token tSTRING_CONTENT ()
Shifting token tSTRING_CONTENT ()
Entering state 452
Reducing stack by rule 400 (line 2090):
    $1 = token tSTRING_CONTENT ()
-> $$ = nterm string_content ()
Stack now 0 2 62 300 76 344 528 246 239 406 53 287
Entering state 458
Reducing stack by rule 397 (line 2074):
    $1 = nterm string_contents ()
    $2 = nterm string_content ()
-> $$ = nterm string_contents ()
Stack now 0 2 62 300 76 344 528 246 239 406 53
Entering state 287
Reading a token: Next token is token tSTRING_END ()
Shifting token tSTRING_END ()
Entering state 457
Reducing stack by rule 383 (line 1961):
    $1 = token tSTRING_BEG ()
    $2 = nterm string_contents ()
    $3 = token tSTRING_END ()
-> $$ = nterm string1 ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 82
Reducing stack by rule 381 (line 1954):
    $1 = nterm string1 ()
-> $$ = nterm string ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 81
Reading a token: Next token is token '\n' ()
Reducing stack by rule 380 (line 1941):
    $1 = nterm string ()
-> $$ = nterm strings ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 80
Reducing stack by rule 269 (line 1455):
    $1 = nterm strings ()
-> $$ = nterm primary ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 75
Next token is token '\n' ()
Reducing stack by rule 218 (line 1232):
    $1 = nterm primary ()
-> $$ = nterm arg ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 235
Next token is token '\n' ()
Reducing stack by rule 219 (line 1238):
    $1 = nterm arg ()
-> $$ = nterm arg_value ()
Stack now 0 2 62 300 76 344 528 246 239 406
Entering state 609
Next token is token '\n' ()
Reducing stack by rule 264 (line 1434):
    $1 = nterm args ()
    $2 = token ',' ()
    $3 = nterm arg_value ()
-> $$ = nterm args ()
Stack now 0 2 62 300 76 344 528 246
Entering state 239
Next token is token '\n' ()
Reducing stack by rule 500 (line 2480):
-> $$ = nterm none ()
Stack now 0 2 62 300 76 344 528 246 239
Entering state 408
Reducing stack by rule 262 (line 1427):
    $1 = nterm none ()
-> $$ = nterm opt_block_arg ()
Stack now 0 2 62 300 76 344 528 246 239
Entering state 407
Reducing stack by rule 233 (line 1300):
    $1 = nterm args ()
    $2 = nterm opt_block_arg ()
-> $$ = nterm call_args ()
Stack now 0 2 62 300 76 344 528 246
Entering state 414
Reducing stack by rule 255 (line 1404):
    $1 = nterm call_args ()
-> $$ = nterm open_args ()
Stack now 0 2 62 300 76 344 528 246
Entering state 415
Reducing stack by rule 254 (line 1392):
    $1 = nterm @8 ()
    $2 = nterm open_args ()
-> $$ = nterm command_args ()
Stack now 0 2 62 300 76 344 528
Entering state 670
Next token is token '\n' ()
Reducing stack by rule 55 (line 700):
    $1 = nterm primary_value ()
    $2 = token '.' ()
    $3 = nterm operation2 ()
    $4 = nterm command_args ()
-> $$ = nterm command ()
Stack now 0 2 62 300
Entering state 67
Next token is token '\n' ()
Reducing stack by rule 42 (line 641):
    $1 = nterm command ()
-> $$ = nterm command_call ()
Stack now 0 2 62 300
Entering state 65
Reducing stack by rule 35 (line 614):
    $1 = nterm command_call ()
-> $$ = nterm expr ()
Stack now 0 2 62 300
Entering state 64
Next token is token '\n' ()
Reducing stack by rule 34 (line 611):
    $1 = nterm expr ()
-> $$ = nterm stmt ()
Stack now 0 2 62 300
Entering state 470
Next token is token '\n' ()
Reducing stack by rule 7 (line 403):
    $1 = nterm stmts ()
    $2 = nterm terms ()
    $3 = nterm stmt ()
-> $$ = nterm stmts ()
Stack now 0 2
Entering state 62
Next token is token '\n' ()
Shifting token '\n' ()
Entering state 216
Reducing stack by rule 497 (line 2473):
    $1 = token '\n' ()
-> $$ = nterm term ()
Stack now 0 2 62
Entering state 220
Reducing stack by rule 498 (line 2476):
    $1 = nterm term ()
-> $$ = nterm terms ()
Stack now 0 2 62
Entering state 300
Reading a token: Now at end of input.
Reducing stack by rule 490 (line 2460):
    $1 = nterm terms ()
-> $$ = nterm opt_terms ()
Stack now 0 2 62
Entering state 299
Reducing stack by rule 4 (line 391):
    $1 = nterm stmts ()
    $2 = nterm opt_terms ()
-> $$ = nterm compstmt ()
Stack now 0 2
Entering state 61
Reducing stack by rule 2 (line 346):
    $1 = nterm @1 ()
    $2 = nterm compstmt ()
-> $$ = nterm program ()
Stack now 0
Entering state 1
Now at end of input.
Stack now 0 1
Cleanup: popping nterm program ()
%