From: Hal Fulton <hal9000@...>
Date: 2004-01-11T14:26:07+09:00
Subject: Re: Converting a string to an array of tokens

John W. Long wrote:
> Is there a fast way to convert a string into a list of tokens?
> 
> Something like:
> 
> "a= c+ a".tokenize(' ', '=', '+') #=> ['a', '=', ' ', 'c', '+', 'a']

irb comes with its own lexer. I've used that before.
(If it's Ruby you're tokenizing.)


Hal