From: Bob Hutchison <hutch@...>
Date: 2001-11-12T09:59:16+09:00
Subject: [ruby-talk:24900] Re: XML libraries (Re:  Re: ruby and	webservices)

On 01/11/11 12:51 PM, "Sean Russell" <ser@efn.org> wrote:

> Bob Hutchison wrote:
> 
>> No I don't think so. How you represent the character stream internally is
>> entirely up to you (immediate *internal* conversion to UTF-8 by your
>> parser is OK). Restricting input to UTF-8 will place an impossible to live
>> with constraint on the use of your parser. Presumably having an XML parser
>> is to allow ruby programs to participate in a larger context -- and this
>> larger context isn't going to provide encoding conversions.
> 
> Then we have a problem.

Yes, but not quite as bad as it seems at first. All you have to do is *read*
UTF-8 and UTF-16, you do not have to use either character encoding
internally and you can output any encoding you like (though to be safe you
should cover one of UTF-8 or UTF-16). The Eiffel XML parsers have a similar
problem to Ruby -- most, if not all, Eiffel implementations have made
characters and bytes equivalent. There are two primary parsers in Eiffel
(that are being merged in the gobo library) that have taken different
approaches. One makes a special string type (so you don't have the support
of the built-in string class), the other just puts UTF-8 characters into a
standard Eiffel string (so on occasion you have some funny 'characters' but
as long as they stay adjacent things seem to work out). Though neither is
perfect they both seem to be workable in that people are able to do useful
work with both.

I think the way to go in the end will be for Ruby (and Eiffel and other
languages) to stop equating characters and bytes. The problem isn't just
with XML.

> 
> Until Ruby regular expressions (and Ruby Strings in general) support
> UTF-16, supporting UTF-16 is going to be problematic in any Ruby
> application.
> 
> I see two solutions:
> 
> 1) implement a byte-wise implementation of an XML parser in Ruby, and
> support UTF-16 explicitly, and
> 2) use a native XML parser, like XMLParser does, and support UTF-16 on the
> Ruby side explicitly.
> 
> (1) would be very slow.  REXML is as fast as it is because it uses the
> native Ruby REGEX code to do most of the grunt-work of the parsing.  Both
> would be troublesome for users.
> 
> The best solution, of course, is for UTF-16 support natively in Ruby.  Then
> all of the Ruby XML parsers would automatically become UTF-16 compliant --
> or, at least, with minimal code tweaking.
>