From: Trans <transfire@...>
Date: 2005-08-17T10:16:15+09:00
Subject: Re: Generic Parsing Library

Hi Adam,

Actually I have written something similar to what you describe, though
it is token based. It may be adaptable to what you describe. Certainly
it could use some twaeking, more testing and any improvements you might
offer. Here's an example of parsing something like XML.

  require 'yaml'

  s = %Q{
  [p]
  This is plain paragraph.
  [t][b]This bold.[b.]This tee'd off.[t.]&tm;
  [p.]
  }

  tokens = []

  t = TokenParser::Token.new( :ONE )
  t.start = lambda { |match| %r{ \[ (.*?) \] }mx }
  t.stop = lambda { |match| %r{ \[ [ ]* (#{resc(match[1])}) (.*?) \. \]
}mx }
  tokens << t

  t = TokenParser::UnitToken.new( :TWO )
  t.start = lambda { |match| ; %r{ \& (.*?) \; }x }
  tokens << t

  cp = TokenParser.new( *tokens )
  d = cp.parse( s )
  y d

outputs (don't let this scare you, its easy to traverse the content)

  --- &id004 !ruby/array:TokenParser::Main
  - "
    "
  - &id002 !ruby/object:TokenParser::Marker
    content:
      - >

        This is plain paragraph.

      - &id001 !ruby/object:TokenParser::Marker
        content:
          - !ruby/object:TokenParser::Marker
            content:
              - This bold.
            inner_range: !ruby/range '36...46'
            match: !ruby/object:MatchData {}
            outer_range: !ruby/range '33...50'
            parent: *id001
            token: &id003 !ruby/object:TokenParser::Token
              key: :ONE
              parser:
              start: !ruby/object:Proc {}
              stop: !ruby/object:Proc {}
          - "This tee'd off."
        inner_range: !ruby/range '33...65'
        match: !ruby/object:MatchData {}
        outer_range: !ruby/range '30...69'
        parent: *id002
        token: *id003
      - !ruby/object:TokenParser::Marker
        content: []
        match: !ruby/object:MatchData {}
        outer_range: !ruby/range '69...73'
        parent: *id002
        token: !ruby/object:TokenParser::UnitToken
          key: :TWO
          parser:
          start: !ruby/object:Proc {}
    inner_range: !ruby/range '4...74'
    match: !ruby/object:MatchData {}
    outer_range: !ruby/range '1...78'
    parent: *id004
    token: *id003

Let me know if you'd like a copy to play with.

T.