From: Brian Candler <b.candler@...>
Date: 2010-11-19T05:59:35+09:00
Subject: Re: Extended ASCII character handeling

Don Norcott wrote in post #962171:
> I have two questions
> 1) Should I ever have to worry about data being scraped from web pages
> not being handled correctly by ruby.

In ruby 1.9, you have to worry about this very much.

Strings in ruby 1.9 are two-dimensional: they have a sequence of bytes, 
and they have an encoding. There are additional 'dimensions' based on 
the string's content - empty, ascii_compatible, valid_encoding.

If your scraper library doesn't document how it choses the encodings to 
tag each string it returns, and doesn't document how it handles invalid 
encodings if it comes across them, then you have to test its behaviour 
for all the various edge cases.

You never have this issue with ruby 1.8, because a string is just a 
string of bytes. Of course, the "garbage in, garbage out" principle 
still applies; you just don't choke on the garbage.

> 2)How do I flag this data to allow me to manipulate it properly. That is
> load it into an array or write to a file.

That's a short question with a long answer, and I'm afraid my own 
attempt to answer it is incomplete:
https://github.com/candlerb/string19/blob/master/string19.rb

If you're reading stuff from a file or a socket yourself, you can 
control the process. If you're trusting a third-party library to fetch 
data from somewhere, then you have to trust that library to do the right 
thing in the situations you're interested in.

> Tried playing with the following but even if the code below is correct
> the extended ascii characters are lost by the time it gets to IRB

irb is not a good predictor of encoding behaviour for ruby 1.9, and 
you'd be better writing standalone .rb scripts that you run.

Note that it's one of the 1.9 language inconsistencies that transcoding 
is *not* done on output by default. So if you have a read a string from 
a file, and carefully tag it as say UTF-8, but your terminal is IBM437, 
then

    puts my_string

will just squirt the UTF-8 bytes to the terminal and they'll display 
wrongly. You can try something like this:

    STDOUT.set_encoding "IBM437"
or
    STDOUT.set_encoding "locale"

Regards,

Brian.

-- 
Posted via http://www.ruby-forum.com/.