Breaking Ruby code into tokens

I've been looking at lex.c and parse.y and parse.c, but it's all rather
over my head.

How might one simply break Ruby code into tokens?

Maybe token isn't the proper term, but I think it is. I've written
very few real parsers in my life.

For example, obviously all keywords and identifiers and punctuation
(such as <<) would be treated as single entities. Strings and
regular expressions would also be treated as such.

I know that Ruby grammar is nontrivial -- for example, by looking at
the code I just now realized that "class <<" is treated as a special
case so that it won't look like a here-doc. Never thought of that
before.

But a full-fledged parser is overkill, too, isn't it? Surely this could
be done in 100 lines of Ruby or so?

Enlighten me...

Hal