From: Gavin Kistner <gavin@...>
Date: 2005-06-20T10:10:54+09:00
Subject: Creating a new Syntax tokenizer

I want to write my own wiki markup language. Pure regexp fails me, as  
I need a proper parser to keep track of state.
I thought I'd give Syntax a try, but I'm a little confused as to some  
of the specifics.

1) What is a 'region', and how do I use the start_region method? It's  
not documented in the API, or the source. (I think this is what I  
want for nesting tags.)

2) Do I have to close_group and close_region, or do they  
automatically get invoked under certain circumstances? (Does starting  
one group close the previous one? Do repeated calls to open the same  
group cause them to be aggregated together (is that how accumulating  
text in :normal groups works?)

3) How do I keep track of state during successive calls to #step? I  
tried an instance variable, but that doesn't seem to exist across calls.

Following is my terrible, broken attempt at the basics of what I'm  
after. Am I totally misunderstanding how to use Syntax?


require 'rubygems'
require_gem 'syntax'

class OWLScribble < Syntax::Tokenizer
   def step
         if heading = scan( /^={1,6}/ )
             start_region "heading level #{heading.length}".intern
             $heading_end = Regexp.new( heading + "\\s*" )
         elsif $heading_end && ( heading = scan( $heading_end ) )
             end_region "heading level #{heading.length}".intern
             $heading_end = nil
         elsif char = scan( /^[\r\n]/ )
             start_group :paragraph, char
         elsif scan( /\*\*/ )
             if $inbold
                 end_region :bold
                 $inbold = nil
             else
                 start_region :bold
                 $inbold = true
             end
         elsif char = scan( /./ )
             start_group :normal, char
         else
             scan( /[\r\n]/ )
         end
   end
end

Syntax::SYNTAX[ 'owlscribble' ] = OWLScribble

str = <<END
Intro paragraph

= Heading 1 =
First **paragraph** under the heading.

== Second **Heading** = very yes ==
Another paragraph.
END

tokenizer = Syntax.load( "owlscribble" )
tokenizer.tokenize( str ) do |token|
   puts "#{token.group} (#{token.instruction}) #{token}"
end



--
(-, /\ \/ / /\/