From: Thomas Bednarz <lists@...>
Date: 2012-09-18T06:52:24+09:00
Subject: Encoding question

I am new to ruby and play around with it a little bit at the moment. I
have a large text file containing data with french accents and german
umlauts. The content of this file (some hundredthousend lines) should be
stored in a table in a postgres database. When I open the file on
windows with an editor called notepad++ it displays the data correctly.
When I look at the output from File.foreach(...) |line| puts line, I get
garbage for any non ASCII character. When I try to store the records to
postgres I get an error, as soon as data with non ASCII characters
should be inserted.

I use RubyMine as IDE and receive the following output with the
following code:

    File.foreach("somefile.txt") do |line|
      if counter > 0 then
        record = line.split(";")
        @az_addidnr = record[1]
        az_chnr = record[2]
        az_adr1 = record[6]
        puts "record data: #{@az_addidnr} | #{az_chnr} | #{az_adr1}"
        conn.exec_prepared('stmt1', [@az_addidnr, az_chnr, az_adr1])
      end

OUTPUT:

record data: 512999 | CH21702301867 | Garage de la Moli�re SA
Uncaught exception: FEHLER:  ungültige Byte-Sequenz für Kodierung
»UTF8«: 0xe87265

I also tried az_adr1 = record[6].encode("ISO-8859-1")

If I try az_adr1 = record[6].encode("ASCII") I get:
Uncaught exception: U+00DE to US-ASCII in conversion from CP850 to UTF-8
to US-ASCII

Could anybode please explain me the following:
How can I find, what kind of Encoding is used in a text file?
What kind of conversion to I need to a) get a correct output and b) to
be able to insert the record into postgresql.

Many thanks for your help.

Tom

-- 
Posted via http://www.ruby-forum.com/.