From: William Djaja Tjokroaminata <billtj@...>
Date: 2002-09-23T23:44:18+09:00
Subject: Regexp: How to Find Legitimate Tokens?

Hi,

I am sure questions like these must have been posted before, but for some
reason I could not find a specific ones by searching through the web.

Basically, I want to find in a Ruby source code a token, such as ":",
which is really a token, i.e., it is not inside a comment nor inside a
string.  My questions are:

1) How do I do it using regexp in Ruby?

2) In Ruby, are the only things that can mask the significance of a token:
    - inside comment
    - inside string
   ?  Or anything else?

3) I have tried to read Ruby's "parse.y", but it is still too hard.  Is
all the Ruby syntax (but not semantics) completely defined by this single
file "parse.y"?

4) Are tools such as lex and yacc more powerful than Ruby in terms of
their lexing and parsing capabilities?  Without regard to efficiency, is
to use Ruby as easy as to use lex?  Is to use Ruby as easy as to use
yacc?  (Probably no, such as shown by the discussion on a parser tool in
Ruby?)

Regards,

Bill