From: sabbyxtabby@... (Sabby and Tabby)
Date: 2004-01-13T21:46:37+09:00
Subject: Re: Converting a string to an array of tokens

"John W. Long" <ng@johnwlong.com> wrote:

> Nice solution, but my idea of specifying your tokens is to have it separate
> before and after each token. I don't want to have to specify everything I'm
> looking for. Just the significant tokens. Sometimes whitespace is
> significant:
> 
> 'a = "    this is a string    "'.tokenize('"', "=")
> 
> should produce:
> 
> ["a", " ", "=", " ", "\"", "    this is a string    ", "\""]

class String
  def tokenize(*tokens)
    tokens.map! {|t|
      case t
      when Regexp
        s = t.to_s
        s.gsub!(/\\./m, '')       # Remove \(
        s.gsub!(/\(\?/, '')       # Remove (?
        s =~ /\(/ ? t : /(#{t})/  # Add capturing () if none.
      when Symbol
        /(\s*)(#{t})(\s*)/        # Significant whitespace.
      else
        /(#{t})/
      end
    }
    split(/#{tokens * '|'}/).reject {|x| x.empty?}
  end
end

str = 'a = "    this is a string    "'

p str.tokenize('"', '=')              # Whitespace not a token.
p str.tokenize('"', /(\s*)(=)(\s*)/)  # Significant whitespace.
p str.tokenize('"', :'=')             # Same but less typing.