From: TAKAHASHI Masayoshi <maki@...>
Date: 2001-12-04T20:01:20+09:00
Subject: [ruby-talk:27434] Re: ANN: REXML 1.1a7

Hi,

Sean Russell <ser@efn.org> wrote:
> > It's too difficult for Ruby to suport UTF-16, so we should
> > convert it into UTF-8 before parsing it.
> 
> Why would UTF-8 be any easier?

As I wrote in [ruby-talk:27350], Ruby's regex supports UTF-8,
but has some problems of handling UTF-16.
REXML uses regex heavily. So using UTF-8 is better choice
when REXML parses Unicode documents.

The problem to handle XML documents in UTF-8 with Ruby's regex
is that it doesn't support General Category Property(*).
It means when we represent NameChar or Name in regex, we should
specify all codepoints(ranges), like
([\xxx\xxx-\xxx\xxxx]|[\xxx\xxx-\xxx\xxx]|\xx\xxx|.....).
It likes large table in Appndix B of XML Rec.

Fortunately, other syntactical characters in XML, like '<>?-&#[];!',
are all in ASCII. So we can parse all well-formed XML documents
without knowing Name/NameChar's codepoints (but we cannot drop
not-well-formed documents containing illegal characters).

In addition, XML Blueberry Requirements(**) suggests that
XML next generation shall allows more characters as names.
Definitions of Name/NameChar are not eternal rule.

(*)   http://www.unicode.org/unicode/reports/tr18/tr18-6d2.html#Categories
(**)  http://www.w3.org/TR/2001/WD-xml-blueberry-req-20010921

> Thanks for the clarification.  I thought, from reading the posts, that you 
> had something to do with it, but I didn't see your name anywhere, so I 
> thought I'd err on the side of caution.  Could you add your name to the 
> comments in the source file?

Sorry, I added my name.


Regards,

TAKAHASHI Masayoshi (maki@inac.co.jp)