From: Xavier Noria <fxn@...>
Date: 2008-05-02T23:48:55+09:00
Subject: Re: Searching for latin/umlaut characters in string

On Fri, May 2, 2008 at 4:38 PM, John Butler <johnnybutler7@gmail.com> wrote:
> Xavier Noria wrote:
>  > On Fri, May 2, 2008 at 2:52 PM, John Butler <johnnybutler7@gmail.com>
>  > wrote:
>  >>  >>
>
> >>  >>  No the string is coming in via a csv file.
>  >>  >
>  >>  > Is the CSV file in UTF8?
>  >>
>  >>  Its Ansii as far as i can tell.  Its actually coming from an excel
>  >>  spread sheet and the user then "saves as" comma separted csv.  Then its
>  >>  import to the application wher i am running into the problem.
>  >
>  > If the CSV is ANSII, and the Ruby source code is ANSII, where is the
>  > "é" coming from? :-)
>
>  Here is an example of the csv file with the word below and how it is
>  displayed.  They are identical. It looks like the file is utf-8 after
>  all.

No it is iso-8859-1. In that CSV "ü" is a single byte instead of two:

fxn@feynman:~/tmp$ od -a testcsv.csv
0000000    M   ?   l   l   e   r  cr  nl
0000010
fxn@feynman:~/tmp$ od -a testcsv_utf8.csv
0000000    M   ?   ?   l   l   e   r
0000007

You'd need to fetch data from the CSV, change its encoding in memory
with iconv, and work with the result.

In general, when doing I/O like that you need to control every
involved encoding to ensure everything matches.