From: Robert Klemme <bob.news@...>
Date: 2004-01-13T05:56:38+09:00
Subject: Re: Converting a string to an array of tokens


<nobu.nokada@softhome.net> schrieb im Newsbeitrag
news:200401121711.i0CHBhuw004600@sharui.nakada.kanuma.tochigi.jp...
> Hi,
>
> At Sun, 11 Jan 2004 23:06:39 +0900,
> Robert Klemme wrote:
> > irb(main):002:0> "a= c+ a".scan( /\w+|=|\+/ )
> > => ["a", "=", "c", "+", "a"]
> > irb(main):003:0> "a= c+ a".scan( /\w+|=|\+|\s+/ )
> > => ["a", "=", " ", "c", "+", " ", "a"]
>
> What about this?
>
>   "a= c+ a".split(/\b/) # => ["a", "= ", "c", "+ ", "a"]

I guess the OP won't like it because the whitespace is not separated
properly.  And he wanted to be able to provide the tokens conveniently.
Another solution, which can of course be tweaked and integrated into String:

irb(main):089:0> def tokenizer(*tokens)
irb(main):090:1>   Regexp.new( tokens.map{|tk| tk.kind_of?( Regexp ) ? tk :
Regexp.escape(tk)}.join('|') )
irb(main):091:1> end
=> nil
irb(main):092:0> def tokenize(str, *tokens)
irb(main):093:1>   str.scan tokenizer(*tokens)
irb(main):094:1> end
=> nil
irb(main):095:0> tokenize( "a= c+ a", "=", "+", /\w+/, /\s+/ )
=> ["a", "=", " ", "c", "+", " ", "a"]

That way one can reuse the tokenizer regexp for multiple passes.

Regards

    robert