From: Dan Doel <djd15@...>
Date: 2004-01-11T20:09:58+09:00
Subject: Re: Converting a string to an array of tokens

I believe this also works:

class String
    def tokenize(*tokens)
        regstr = ""
       
        regex = Regexp.new(tokens.map do |t|
            Regexp.escape(t)
        end.join("|"))
       
        do_tokenize(regex).delete_if { |str| str == "" }
    end
   
    def do_tokenize(regex)
        match = regex.match self
       
        if match
            [match.pre_match, match[0]] + 
match.post_match.do_tokenize(regex)
        else
            []
        end
    end
end

The recursion could cause problems if you have really long strings, in 
which case it'd probably
be wise to rewrite it as a loop (which is arguably somewhat uglier). You 
might also want to
make #do_tokenize private. I don't know if this is the best way, but 
it's a way.

- Dan