From: "Eric I." <rubytraining@...>
Date: 2008-08-23T05:46:06+09:00
Subject: Re: Change/ignore XML encoding?

On Aug 22, 12:32 pm, m...@tidbits.com (matt neuburg) wrote:
> > Now, with most of the XML results I get from movie-xml.com, the default
> > utf-8 is fine since there are no non-utf-8 characters. When searching
> > Sin City as an example, there are. Here's the response I get:
>
> > Input is not proper UTF-8, indicate encoding !
>
> > The source XML has an encoding declared as such:
>
> > <?xml version="1.0" encoding="ISO-8859-1"?>
>
> > So I should probably just decode as ISO-8859-1 as well. How the hell do
> > I do that? I have Googled the crap out of this and just can't seem to
> > find what I need here...
>
> Could this just be a bug in Libxml? REXML seems to do the right thing...

Clearly libxml is expecting UTF-8, even though the XML file specifies
that it's encoded in ISO-8859-1.  So that's a bug.

However, it appears that libxml is "correctly" rejecting data that is
not proper UTF-8 (independent of what it claims to be).  Twice in the
XML data the word "verg?enza" appears where the "?" has hex code 0xFC
that encodes a lower case "u" with umlaut in ISO-8859-1.  0xFC cannot
appear in UTF-8 data due to RFC-3629.

libxml should work with ISO-8859-1 data much of the time, as long as
it doesn't contain 13 specific bytes (0xC0, 0xC1, 0xF5..0xFF).

Eric

====

Are you interested in on-site Ruby or Ruby on Rails training
that uses well-designed, real-world, hands-on exercises?
http://LearnRuby.com