From: Robert Klemme <shortcutter@...>
Date: 2010-11-18T02:15:17+09:00
Subject: Re: Extended ASCII character handeling

On 17.11.2010 17:01, Don Norcott wrote:
> "200 Millionen Jahre später                              # 17.39
> \n",
> "200 Millionen Jahre später                              #  9.87
> 3404211707    \n",
> "A l'assaut de l'invisible                         1977 #  4.91
> \n",
> "A l'assaut de l'invisible                         1990 #  5.18
> 226603779      \n",
>
> The above 4 lines are data I was attempting to load into an array to
> test some code.  I was getting what I thought were strange results until
> I realized not all characters were being loaded into the element
> resulting in column alignment problems.
>
> The data above was cut from a file that had been manipulated a dozen
> times in ruby arrays before being written to a file.  So it appears the
> default way ruby handles extended ASCII(?) is fine.
>
> I have two questions
> 1) Should I ever have to worry about data being scraped from web pages
> not being handled correctly by ruby.

Depends how you read the data from webpages.

> 2)How do I flag this data to allow me to manipulate it properly. That is
> load it into an array or write to a file.

You need to set encodings properly.  You can do that when opening the 
file.  Example:

irb(main):001:0> io = File.open "x","r"
=> #<File:x>
irb(main):002:0> io.external_encoding
=> #<Encoding:UTF-8>
irb(main):003:0> io.internal_encoding
=> nil
irb(main):004:0> io.read.encoding
=> #<Encoding:UTF-8>
irb(main):005:0> io.close
=> nil

irb(main):006:0> io = File.open "x","r:ASCII"
=> #<File:x>
irb(main):007:0> io.external_encoding
=> #<Encoding:US-ASCII>
irb(main):008:0> io.internal_encoding
=> nil
irb(main):009:0> io.read.encoding
=> #<Encoding:US-ASCII>
irb(main):010:0> io.close
=> nil

See http://blog.grayproductions.net/articles/understanding_m17n

> Tried playing with the following but even if the code below is correct
> the extended ascii characters are lost by the time it gets to IRB
>
> str = String.new
> str.encode(("US-ASCII")
> str = "Millionen Jahre später"

This won't work - ever.  You set the encoding for an instance and then 
you reassign str to point to another instance, so all your encoding 
settings are lost.  Also, there is no "ü" in ASCII which is 7bit!

irb(main):011:0> s="a"
=> "a"
irb(main):012:0> s.encoding
=> #<Encoding:UTF-8>
irb(main):013:0> t = s.encode "ASCII"
=> "a"
irb(main):014:0> t.encoding
=> #<Encoding:US-ASCII>

Now with "ü":

irb(main):015:0> s="ü"
=> "ü"
irb(main):016:0> s.encoding
=> #<Encoding:UTF-8>
irb(main):017:0> t = s.encode "ASCII"
Encoding::UndefinedConversionError: "\xC3\xBC" from UTF-8 to US-ASCII
         from (irb):17:in `encode'
         from (irb):17
         from /usr/local/bin/irb19:12:in `<main>'

Kind regards

	robert


-- 
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/