From: Robert Klemme <bob.news@...>
Date: 2004-01-13T18:21:43+09:00
Subject: Re: Converting a string to an array of tokens


"John W. Long" <ng@johnwlong.com> schrieb im Newsbeitrag
news:010501c3d995$9f435390$6601a8c0@jwldesktop...
>
> "Dan Doel" wrote:
> > I believe this also works:
> > ..snip!..
>
> This is almost exactly what I was looking for.
>
> > The recursion could cause problems if you have really
> > long strings, in which case it'd probably be wise to
> > rewrite it as a loop (which is arguably somewhat
> > uglier).
>
> Depends on what you mean by ugly:
>
> class String
>     def tokenize(*tokens)
>         regex = Regexp.new(tokens.map { |t|
Regexp::escape(t) }.join("|"))
>         string = self.dup
>         array = []
>         while match = regex.match(string)
>             array += [match.pre_match, match[0]]
>             string = match.post_match
>         end
>         array += [string]
>         array.delete_if { |str| str == "" }
>     end
>     def each_token(*tokens, &b)
>         tokenize(*tokens).each { |t| b.call(t) }
>     end
> end
>
> Very nice. If only it would work with regular expressions as well.
>
> I wonder what the odds are of getting this or something like this added
to
> the language. Seems like it would be a nice to have on the String class
to
> begin with and written in C for speed.

Oh, we can still tweak the solution provided:

 - Use Array#push or Array#<< instead of "+=" which creates too much tmp
instances

 - implement the iteration in each_token and make tokenize depend on that,
   so that tokenizing of large strings via each_token is more efficient
   because no array is needed then.

 - Don't add empty strings to the array.

 - No need to dup.


That's what I'd do:

class String
    def tokenize(*tokens)
        array = []
        each_token(*tokens){|tk| array << tk}
        array
    end

    def each_token(*tokens)
        regex = Regexp.new(tokens.map { |t| t.kind_of?( Regexp ) ? t :
Regexp::escape(t) }.join("|"))
        string = self

        while( match = regex.match(string) )
            yield match.pre_match if match.pre_match.length > 0
            yield match[0] if match[0].length > 0

            string = match.post_match
        end

        yield string if string.length > 0
        self
    end
end

Kind regards

    robert