From: "nicholasmabry@..." <nicholasmabry@...>
Date: 2008-06-24T10:58:19+09:00
Subject: Re: rex: howto use the lexer class?

Hey Fabrice,

The problem you're seeing is due to rex's assumption that you are
generating a parser in tandem with your lexer. The generated method
Lexer::scan_str looks like this:

def scan_str( str )
  scan_evaluate  str
  do_parse
end

While scan_evaluate(str) is the method generated by your token
definitions, do_parse() depends on a racc grammar having been defined
and initialized. The bad news is that the default scan_str() won't
work for your purposes. The good news is that scan_evaluate() will. If
you examine your generated test.rex.rb file, you'll see that
scan_evaluate() identifies your tokens and pushes them one by one into
a queue named @rex_tokens. To pull them out of the queue, simply call
next_token(). Here's a quick replacement for the bottom of your token
definition file:

lexer=Lexer.new
while 1
  str=$stdin.gets.strip
  puts "str=#{str}"

  # Here we're scanning the string for tokens
  lexer.scan_evaluate(str)

  # And then printing each one out to stdout
  while token = lexer.next_token
    p token
  end
  puts
"--------------------------------------------------------------------------"
end

The only other minor change I made was to "@{str}". The ruby string
interpolation escape sequence is actually "#{ }". Let us know if you
have more questions. Happy lexing!

-Nick