From: Michael Furr <furr@...>
Date: 2008-09-25T05:18:30+09:00
Subject: [ruby-core:18869] Re: tokenizing regular expressions when passed as method params



On Thu, 25 Sep 2008, Seth Dillingham wrote:

>  foo.bar /regex/, bat
>
> I can't figure out how to tell if that '/' is dividing foo.bar by
> whatever comes next, or if it's a regular expression.

[...]

> Is there any way to unambiguously identify regular expressions in this
> situation, or is this just a bug we'll have to live with?

Ruby's syntax can be disambiguated using just static information, but its 
rather complex.  The lexer/parser uses a state machine to determine how to 
lex the next character.  Here is my understanding of how this situation 
(lexing '/') is resolved.  There are two primary things that influence the 
state: whitespace and the location of the token in an expression (i.e., 
the beginning/middle/end).

Consider the following code:
   def f() 6 end
   i= 1

   f/3/i  # (1)
   f /3/i # (2)
   f=9
   f/3/i  # (3)
   f /3/i # (4)

When parsing line (1), Ruby first reads the letter 'f' and tokenizes it as 
a method name based on the fact no assignment has been seen to create the 
local variable 'f'.  This puts the lexer in the "middle of expression" 
state.  Here, when it reads the next character as '/', it is lexed as a 
binary operator (division).  Thus, line (1) is parsed as (f()/3)/i.

On line (2), 'f' is parsed as in (1), but it then sees a space, putting 
the lexer in the 'start of expression' state.  Here, '/' is parsed as a 
regexp delimiter (and +,-,*,& are parsed unary operators, etc...).  Thus, 
this line is parsed as f(/3/i).  Note, a space after the operator can also 
influence the state (as Caleb Clausen discussed).

Before line (3), an assignment has been seen**, and so on line (3), 'f' is 
lexed as a variable name and puts the lexer in state 'middle of 
expression'.  After this point, line (3) is parsed similarly to line (1): 
(f/3)/i.

On line (4), 'f' is again parsed as a variable name, but upon seeing the 
space, the lexer does *not* transition to the beginning state (since a 
variable can't take an argument, only methods can).  Thus the '/' 
character is still parsed as a binary operator, creating the same parse 
tree as (3): (f/3)/i.

Hope this helps,
-Mike


** This assignment doesn't have to happen dynamically, the static 
heuristic would also accept "f=9 if false" for instance.