From: Robert Klemme <bob.news@...>
Date: 2004-01-11T23:06:39+09:00
Subject: Re: Converting a string to an array of tokens


"John W. Long" <ng@johnwlong.com> schrieb im Newsbeitrag
news:00e401c3d804$93be7ff0$6601a8c0@jwldesktop...
> Hal Fulton wrote:
> > irb comes with its own lexer. I've used that before.
> > (If it's Ruby you're tokenizing.)
>
> Actually no, I'm not using it to parse Ruby. I'm just looking for
something
> general that works like String#scan, but leaves the tokens in place in the
> array. This would be useful for parsing almost any language.

I'm not sure that I understand what you mean by "leaves the tokens in place
in the array".  String#scan is usually the method of choice:

irb(main):002:0> "a= c+ a".scan( /\w+|=|\+/ )
=> ["a", "=", "c", "+", "a"]
irb(main):003:0> "a= c+ a".scan( /\w+|=|\+|\s+/ )
=> ["a", "=", " ", "c", "+", " ", "a"]

What are you missing?

Regards

    robert