From: Phrogz <phrogz@...>
Date: 2007-11-11T12:50:01+09:00
Subject: Re: Scan for Tokens

On Nov 10, 6:07 pm, Raul Parolari <raulparol...@gmail.com> wrote:
> I am looking for the best way to break an input string into individual
> tokens (I do not want to use a lexer library)

Look at the StringScanner library[1] included with Ruby. It's simple,
and it's fast. It's the basis of my TagTreeScanner library[2], which
is specialized for parsing arbitrary text and converting it into
hierarchically nested markup (e.g. XML).

[1] http://ruby-doc.org/stdlib/libdoc/strscan/rdoc/index.html
[2] http://phrogz.net/RubyLibs/OWLScribble/doc/tts.html