From: Don Norcott <dnorcott@...>
Date: 2010-11-18T07:02:19+09:00
Subject: Re: Extended ASCII character handeling

I am using nokogiri (with Mechanize) to scrape the data and the data I
am concerned with is  extracted only from displayable fields <table
class="result> .... </table>

The code set/language references I see are
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
Which is I believe, what I am calling Extended ASCII(8 bit 0 - 255)

AND

//<![CDATA[ var awsDomain = 'xxxxxxxx.xxx';
var surveyLink = "sm=93_2fjk6BaUHEqrn2qpdbknQ_3d_d"
var twoLetterISOCode = 'en'; //]]>

The scrapped data has never caused a problem within the ruby program
(would have been very obvious).  Can I safely assume that code sets will
never present a problem for this specific application as long as the
retrieval methods do not change????.
=========================
=========================
That being said when I open the file with io it reports
#<Encoding:IBM437> which would contain the characters giving problems
(but not there correct representation). That is to say the IBM437 for
character E4 is a Graphic character not the accented French 'a' in
"später". The graphic is what is also being displayed in the IRB
console.

I have gone through most of the Shades of Gray link and only thing that
I thought might have been of value is the LC_TYPE but either UTF-9 or
ISO-8859-1 both work identically in my situation.  I have removed
LC_TYPE since there is no problem with internal data and it might cause
a problem down the line when I have forgotten about it.

Also tried saving code & data to a file and running the file (ruby
xxx.rb) and still reports a multibyte error.

Played with ruby command line encoding settings (ruby -E XXX)and still
received errors regardless of code set I picked - may be related to
LC_TYPE as did not reboot so still valid??

Error is
CodeSet.rb:4: invalid multibyte char (US-ASCII) which is 7 bit.

Extended ASCII code sets ISO-8859 & IBM437 are 8 bit but can not seem to
set this.


=======================
=======================

I can edit the data file externally and read the data into an array
without problems.
So will assume no need to pursue the code set settings at this time.

Will not update unless I have a revelation.

By the recommended link was excellent, will save URL as a resource.

-- 
Posted via http://www.ruby-forum.com/.