From: Robert Klemme <bob.news@...>
Date: 2004-01-13T23:56:36+09:00
Subject: Re: Converting a string to an array of tokens


"John W. Long" <ng@johnwlong.com> schrieb im Newsbeitrag
news:001f01c3d9da$28eeea70$6601a8c0@jwldesktop...
> "Robert Klemme" wrote:
> > That's what I'd do:
> >
> > class String
> >     def tokenize(*tokens)
> >         array = []
> >         each_token(*tokens){|tk| array << tk}
> >         array
> >     end
> >
> >     def each_token(*tokens)
> >         regex = Regexp.new(tokens.map { |t| t.kind_of?( Regexp ) ? t :
> > Regexp::escape(t) }.join("|"))
> >         string = self
> >
> >         while( match = regex.match(string) )
> >             yield match.pre_match if match.pre_match.length > 0
> >             yield match[0] if match[0].length > 0
> >
> >             string = match.post_match
> >         end
> >
> >         yield string if string.length > 0
> >         self
> >     end
> > end
>
> Good ideas. It would be fun to optimize this further for speed. Maybe
even
> do a little ruby golf with it. If I have time this evening I may work on
> creating a demanding timed test case for it.

Btw, this would be easier if Regexp#match would support additional
arguments for start and end like

def match(str, start = 0, end = str.length)
....
end

and MatchData would expose the the index of the start element and the
indes of the first elem after the match.  The loop above could then be
written as:

regex = ...
start = 0

while ( match = regex.match(string, start) )
  yield self[start, match.start_index - start] if match.start_index -
start > 0
  yield match[0] if match[0].length > 0
  start = match.end_index  # index of 1st element after match
end

yield self[start,self.length] if self.length - start > 0


What do others think, is this a reasonable extension?

Kind regards

    robert