From: "John W. Long" <ng@...>
Date: 2004-01-13T14:25:12+09:00
Subject: Re: Converting a string to an array of tokens


"Dan Doel" wrote:
> I believe this also works:
> ..snip!..

This is almost exactly what I was looking for.

> The recursion could cause problems if you have really
> long strings, in which case it'd probably be wise to
> rewrite it as a loop (which is arguably somewhat
> uglier).

Depends on what you mean by ugly:

class String
    def tokenize(*tokens)
        regex = Regexp.new(tokens.map { |t| Regexp::escape(t) }.join("|"))
        string = self.dup
        array = []
        while match = regex.match(string)
            array += [match.pre_match, match[0]]
            string = match.post_match
        end
        array += [string]
        array.delete_if { |str| str == "" }
    end
    def each_token(*tokens, &b)
        tokenize(*tokens).each { |t| b.call(t) }
    end
end

Very nice. If only it would work with regular expressions as well.

I wonder what the odds are of getting this or something like this added to
the language. Seems like it would be a nice to have on the String class to
begin with and written in C for speed.

___________________
John Long
www.wiseheartdesign.com