From: "Weirich, James" <James.Weirich@...>
Date: 2004-01-13T05:46:41+09:00
Subject: Re: Converting a string to an array of tokens

This is fairly general and easily extended ...

  Identifiers = %{[a-zA-Z_][a-zA-Z0-9_]*}
  Numbers     = %{[0-9]+}
  Punct       = %w{ << >> == += -= *= /= != [ ] }
  Single      = %{[-+*/<>{}=]}

  TokenList = []
  TokenList << Identifiers << Numbers 
  TokenList += Punct.collect{|op| Regexp.quote(op)}
  TokenList << Single
  TokenRe = Regexp.new(TokenList.join('|'))

  def tokenize(str)
    str.scan(TokenRe)
  end

-- 
-- Jim Weirich / Compuware
-- FWP Capture Services
-- Phone: 859-386-8855