From: Neil Hodgson <nhodgson@...>
Date: 2002-09-09T10:09:47+09:00
Subject: Re: SciTE/VIM Syntax highlighting

Gavin Sinclair:

> Do you happen to know any good (into-level) web resources on lexing?

   Not really anything introductory. I learned at university where the text
was the classic Aho and Ullman "Dragon" book "Principles of Compiler
Design". For medium level interest (a sizeable investment in learning for a
good chunk of capability) I'd recommend downloading Terence Parr's ANTLR and
experimenting with it.
http://www.antlr.org/
   There is an incomplete book with an interesting chapter "Building
Translators By Hand".
http://www.antlr.org/book/index.html
   Dr. Parr also wrote an earlier book that is available online:
http://www.antlr.org/papers/pcctsbk.pdf
   ANTLR includes functionality for writing lexers, parsers and tree
parsers.

> The
> current vim indenting rules for Ruby would really benefit from finding the
> beginning of the current statement (which may be before the current line,
of
> course).

   Statement recognition requires parse level knowledge rather than just
lexical knowledge. Lexers take an input stream of characters and break it up
into tokens such as strings, identifiers, numbers and whitespace. A parser
can take this token stream and recognise the structure of the code such as
statements, blocks and declarations.

   Neil