From: james@...
Date: 2002-05-05T00:59:42+09:00
Subject: RE: REXML and streaming APIs

Sean wrote:

> Hello,
>
> I'd like to solicit comments from the people in this group that use
> streaming XML parser APIs.
>
> I recently received an email from a user who needed REXML to do
> something it didn't, and he provided a change to REXML that enabled
> this behavior.  Before I choose a direction REXML is going to go in
> with changes (additions, rather... I won't break backward
> compatibility) to the streaming API, I'd like to get feedback from
> people who actually use streaming APIs -- I don't use them, myself, so
> I don't feel confident about "laying down the law" about the best API.

You don't use a streaming API? Oh, you're missing out on some slick stuff!
I use the stream parser in REXML::Document on rubyxml.com to convert
database entries into blocks of HTML.  Very neat.  I wrote a class that
takes a DBI statement handle and wraps it with a stream source interface so
that REXML::Document.parse_stream thinks it's a string or a file.

>
>
> First, I'm going to add a StreamParser class.  It would have the
> parse_stream method from Document.  I can add methods to make it look
> more like a standard implementation of the listener pattern, where you
> can add and remove multiple listeners.  IE, you'd get two new methods:

<snip/>

Looks good.  I don't have an immediate use, but I can see that it would be
handy for creating, say, both an HTML and an RTF conversion from a single
XML source at the same time.  Or use it to pump XML out multiple sockets.

>
>
> I'm also considering a step() method, which would be used to step
> through the document at your leisure:
>
>  p = StreamParser.new( stream, listener )
>  step()    # Parses the first tag encountered, and halts at the next
>  step( true )   # Parses the next tag in its entirety
>  step( "e" )    # Parses everything until it hits tag "e"
>
> I'm not very keen on this API, because (1) I doubt it would be very
> useful, and (2) it seems like a lot of work to use.

I'm wondering where this came from.  It seems that many people use stream
parsers to process table rows, and like to work with row-chunks.  But I
don't see what the gain is from moving the logic out of the listener code,
where it would be in a typical SAX-like application.

>
> What I like more is adding a more intelligent listener interface,
> similar to what NQXML provides:
>
>  p = REXML::StreamParser.new(file)
>  p.handle(:tag_start, %w(root level1 level2)) {|e|
>   # do something with root, level1, and level2 tags
>  }
>  p.handle(:tag_start, %w(level3 level4)) { |e|
>   # do something else with level3 and level4 tags
>  p.handle(:text, %w(root level1 level2 level3)) {| e |
>   # do something with the text nodes
>  }
>  p.parse
>
> and/or
>
>  p = REXML::StreamParser.new(file)
>  p.handle( listener, %w( root level1 level2 ) )
>  p.handle( listener, %w( level3 level4 ) )
>  p.parse
>
> The second looks like less code, but would be more because of the
> listener class definitions.  However, I think they solve different
> problem domains; in the second case, you dedicate a class to handling
> all of the events within an element to a class, and in the first,
> you're only getting notified about a small domain of events.  I can
> implement both, but I don't want to implement anything that isn't
> going to be used.

This looks similar to what I use.  I have a listener class initialized with
the name of the root element, the name of the 'row' element, a template for
the root element, and a template for the row element.  (We're talking
task-specific database->XML code here.) You can optionally pass in a class
that knows to process the text of specific elements (helpful for fixing up
character codes or converting timestamps to nicer-looking dates).

When the listener encounters a 'row' tag, it knows to interpolate the row
data with the row template; the accumulated row text gets interpolated into
the root template.

>
> One last thing.  It seem petty of me to stick with "tag_start" and
> "tag_end" when SAX uses "start_element" and "end_element".  At the
> cost of a little bloat, I can change the listener API to conform more
> to SAX and /mostly/ maintain backwards compatibility -- or I can throw
> out the old API, break backwards compatibility, and put in heafty
> warnings for the next few releases.  Can I get a vote on this?

Go with the SAX API.

>
> Please CC me directly with your feedback, as I'm rarely in the
> newsgroup these days.
>
> Thanks!

Thank you for such a good piece of work.

James


>
> --- SER
>