[ruby-core:18520] Re: rubylex-analyzer?

On Sep 9, 2008, at 01:17 , C.E. Thornton wrote:

> I am trying to understand the "state equipped scanner"
> -- tough going !
>
> The hackers Guide mentions a program
> 'rubylex-analyzer' that should help make
> understanding it easier..
>
> Does it still exist and will it help??

That is probably '-y'. I don't find it helpful except when I'm  
debugging my ruby_parser and even then it is painful. Note, this  
output is both lexer and parser output:

% ruby -y -e 'p 1 + 1'

Starting parse
Entering state 0
Reducing stack by rule 1 (line 346):
-> $$ = nterm @1 ()
Stack now 0
Entering state 2
Reading a token: Next token is token tIDENTIFIER ()
Shifting token tIDENTIFIER ()
Entering state 34
Reading a token: Next token is token tINTEGER ()
Reducing stack by rule 477 (line 2439):
    $1 = token tIDENTIFIER ()
-> $$ = nterm operation ()
Stack now 0 2
Entering state 94
Next token is token tINTEGER ()
Reducing stack by rule 253 (line 1392):
-> $$ = nterm @8 ()
Stack now 0 2 94
Entering state 246
Next token is token tINTEGER ()
Shifting token tINTEGER ()
Entering state 40
Reducing stack by rule 415 (line 2176):
    $1 = token tINTEGER ()
-> $$ = nterm numeric ()
Stack now 0 2 94 246
Entering state 89
Reducing stack by rule 377 (line 1933):
    $1 = nterm numeric ()
-> $$ = nterm literal ()
Stack now 0 2 94 246
Entering state 79
Reducing stack by rule 268 (line 1454):
    $1 = nterm literal ()
-> $$ = nterm primary ()
Stack now 0 2 94 246
Entering state 75
Reading a token: Next token is token '+' ()
Reducing stack by rule 218 (line 1232):
    $1 = nterm primary ()
-> $$ = nterm arg ()
Stack now 0 2 94 246
Entering state 235
Next token is token '+' ()
Shifting token '+' ()
Entering state 337
Reading a token: Next token is token tINTEGER ()
Shifting token tINTEGER ()
Entering state 40
Reducing stack by rule 415 (line 2176):
    $1 = token tINTEGER ()
-> $$ = nterm numeric ()
Stack now 0 2 94 246 235 337
Entering state 89
Reducing stack by rule 377 (line 1933):
    $1 = nterm numeric ()
-> $$ = nterm literal ()
Stack now 0 2 94 246 235 337
Entering state 79
Reducing stack by rule 268 (line 1454):
    $1 = nterm literal ()
-> $$ = nterm primary ()
Stack now 0 2 94 246 235 337
Entering state 75
Reading a token: Next token is token '\n' ()
Reducing stack by rule 218 (line 1232):
    $1 = nterm primary ()
-> $$ = nterm arg ()
Stack now 0 2 94 246 235 337
Entering state 512
Next token is token '\n' ()
Reducing stack by rule 186 (line 1101):
    $1 = nterm arg ()
    $2 = token '+' ()
    $3 = nterm arg ()
-> $$ = nterm arg ()
Stack now 0 2 94 246
Entering state 235
Next token is token '\n' ()
Reducing stack by rule 219 (line 1238):
    $1 = nterm arg ()
-> $$ = nterm arg_value ()
Stack now 0 2 94 246
Entering state 236
Next token is token '\n' ()
Reducing stack by rule 263 (line 1430):
    $1 = nterm arg_value ()
-> $$ = nterm args ()
Stack now 0 2 94 246
Entering state 239
Next token is token '\n' ()
Reducing stack by rule 500 (line 2480):
-> $$ = nterm none ()
Stack now 0 2 94 246 239
Entering state 408
Reducing stack by rule 262 (line 1427):
    $1 = nterm none ()
-> $$ = nterm opt_block_arg ()
Stack now 0 2 94 246 239
Entering state 407
Reducing stack by rule 233 (line 1300):
    $1 = nterm args ()
    $2 = nterm opt_block_arg ()
-> $$ = nterm call_args ()
Stack now 0 2 94 246
Entering state 414
Reducing stack by rule 255 (line 1404):
    $1 = nterm call_args ()
-> $$ = nterm open_args ()
Stack now 0 2 94 246
Entering state 415
Reducing stack by rule 254 (line 1392):
    $1 = nterm @8 ()
    $2 = nterm open_args ()
-> $$ = nterm command_args ()
Stack now 0 2 94
Entering state 354
Next token is token '\n' ()
Reducing stack by rule 53 (line 683):
    $1 = nterm operation ()
    $2 = nterm command_args ()
-> $$ = nterm command ()
Stack now 0 2
Entering state 67
Next token is token '\n' ()
Reducing stack by rule 42 (line 641):
    $1 = nterm command ()
-> $$ = nterm command_call ()
Stack now 0 2
Entering state 65
Reducing stack by rule 35 (line 614):
    $1 = nterm command_call ()
-> $$ = nterm expr ()
Stack now 0 2
Entering state 64
Next token is token '\n' ()
Reducing stack by rule 34 (line 611):
    $1 = nterm expr ()
-> $$ = nterm stmt ()
Stack now 0 2
Entering state 63
Next token is token '\n' ()
Reducing stack by rule 6 (line 399):
    $1 = nterm stmt ()
-> $$ = nterm stmts ()
Stack now 0 2
Entering state 62
Next token is token '\n' ()
Shifting token '\n' ()
Entering state 216
Reducing stack by rule 497 (line 2473):
    $1 = token '\n' ()
-> $$ = nterm term ()
Stack now 0 2 62
Entering state 220
Reducing stack by rule 498 (line 2476):
    $1 = nterm term ()
-> $$ = nterm terms ()
Stack now 0 2 62
Entering state 300
Reading a token: Now at end of input.
Reducing stack by rule 490 (line 2460):
    $1 = nterm terms ()
-> $$ = nterm opt_terms ()
Stack now 0 2 62
Entering state 299
Reducing stack by rule 4 (line 391):
    $1 = nterm stmts ()
    $2 = nterm opt_terms ()
-> $$ = nterm compstmt ()
Stack now 0 2
Entering state 61
Reducing stack by rule 2 (line 346):
    $1 = nterm @1 ()
    $2 = nterm compstmt ()
-> $$ = nterm program ()
Stack now 0
Entering state 1
Now at end of input.
Stack now 0 1
Cleanup: popping nterm program ()
2