From: Hal Fulton <hal9000@...>
Date: 2003-10-05T10:35:22+09:00
Subject: Re: Breaking Ruby code into tokens

daz wrote:
> "Hal Fulton" <hal9000@hypermetrics.com> wrote:
> 
> 
>>I've been looking at lex.c and parse.y and parse.c, ...
> 
> 
> Pending a correction, lex.c is an unused remnant.
> parse.c is ignorable (generated by Yacc from parse.y).
> The real ruby lexer is in parse.y (function yylex).

Didn't know lex.c was a leftover.

I know parse.c is generated from parse.y, but I can
read C and can't read yacc. :)

> 
>>How might one simply break Ruby code into tokens?
>>
>>
>>Hal
>>
> 
> 
> While writing IRB, Keiju ISHITSUKA seems to have taken
> the trouble to expose his lexer to other callers.
> Thank you.

irb/ruby-lex is what I've settled on. It works nicely.
(Mauricio or batsman on IRC also pointed me that way.)

Thanks,
Hal