From: Raul Parolari <raulparolari@...>
Date: 2007-11-11T15:27:08+09:00
Subject: Re: Scan for Tokens

Gavin Kistner wrote:
> On Nov 10, 6:07 pm, Raul Parolari <raulparol...@gmail.com> wrote:
>> I am looking for the best way to break an input string into individual
>> tokens (I do not want to use a lexer library)
> 
> Look at the StringScanner library[1] included with Ruby. It's simple,
> and it's fast. It's the basis of my TagTreeScanner library[2], which
> is specialized for parsing arbitrary text and converting it into
> hierarchically nested markup (e.g. XML).
> 
> [1] http://ruby-doc.org/stdlib/libdoc/strscan/rdoc/index.html
> [2] http://phrogz.net/RubyLibs/OWLScribble/doc/tts.html

Gavin

   I was surprised at first that this basic capability was in a library, 
but
StringScanner works beautifully, and it is indeed extremely fast.

I will try your TagTreeScanner at the first chance

Thank you

Raul
-- 
Posted via http://www.ruby-forum.com/.