Re: Byte窶都tream parsing in Ruby

On Jul 23, 2009, at 12:02 PM, Brian Candler wrote:

> Where does it say that String#<< can now raise an exception, and under
> what circumstances?

Quoting from the page I linked to in my last message:
It's probably worth mentioning that it is possible for a transcoding =20
operation to fail with an error. For example:

$ cat transcode.rb
# encoding: UTF-8
utf8 =3D "R=E9sum=E9=85"
latin1 =3D utf8.encode("ISO-8859-1")
$ ruby transcode.rb
transcode.rb:3:in `encode': "\xE2\x80\xA6" from UTF-8 to ISO-8859-1 =20
(Encoding::UndefinedConversionError) from transcode.rb:3:in `<main>'
Naturally this fails because "=85" is not a valid character in Latin-1.

> Ah, I finally found it, right at the end of the
> *comments* at the bottom of section 8, added a month after initial
> publication. (+)

What does how long it took me to write the content have to do with =20
anything?  I added that comment to cover some items you had mentioned =20=

I had overlooked.  Now it's invalid because it took me a while???

> Where does it say that the encoding of a String can change when you
> concatenate another string onto it?

Quoting from the same page:

One thing that my help a little in normalizing your data is Ruby's =20
concept of compatibleEncodings. Here's an example of checking and =20
taking advantage of compatible Encodings:

# data in two different Encodings
p ascii_my # >> "My "
puts ascii_my.encoding.name # >> US-ASCII
p utf8_resume # >> "R=E9sum=E9"
puts utf8_resume.encoding.name # >> UTF-8
# check compatibility
p Encoding.compatible?(ascii_my, utf8_resume) # >> #<Encoding:UTF-8>
# combine compatible data
my_resume =3D ascii_my + utf8_resume
p my_resume # >> "My R=E9sum=E9"
puts my_resume.encoding.name # >> UTF-8
In this example I had data in two different Encodings, US-ASCII and =20
UTF-8. I asked Ruby if the two pieces of data were compatible?(). Ruby =20=

can respond to that question in one of two ways. If it returns false, =20=

the data is not compatible and you will probably need to transcode at =20=

least one piece of it to work with the other. If an Encoding is =20
returned, the data is compatible and can be concatenated resulting in =20=

data with the returned Encoding. You can see how that played out when =20=

I combined these Strings.

> (+) There is a warning in the string *comparisons* section saying =20
> that,
> basically, the rules are too complicated to understand, so you should
> always ensure that two strings are in the same encoding before =20
> comparing
> them. Arguably you could say the same applies to any other operation
> which takes two strings.
>
> But this to me shows the whole exercise is futile.

But you should be doing the exact same thing in Ruby 1.8, which I =20
understand you believe to be a superior system.  If you are going to =20
have two pieces of data interact, it just makes sense that they will =20
pretty much always need to be the same kinds of data.

> If, in order to write a valid program, you need to ensure that all =20
> strings are in the same encoding, then there should be a global flag =20=

> which sets the encoding.

Like -E and -U in Ruby 1.9?

> And in many apps, the encoding information is carried "out of band"
> anyway: for example: in HTTP or MIME, the encoding info is in a
> Content-Type: header.

Yeah, that's why a global switch won't really save you from doing your =20=

job.  You need to read that header, and treat the content accordingly.

James Edward Gray II