From: Jonathan Hudson <jh+ruby-lang@...>
Date: 2007-11-04T01:45:04+09:00
Subject: Re: Unicode illegal characters problem

On Sat, 3 Nov 2007 11:20:33 -0500
AEtzold@gmx.de wrote:

> 
> -------- Original-Nachricht --------
> > Datum: Sun, 4 Nov 2007 00:55:04 +0900
> > Von: Jonathan Hudson <jh+ruby-lang@daria.co.uk>
> > An: ruby-talk@ruby-lang.org
> > Betreff: Re: Unicode illegal characters problem
> 
> > On Sat, 3 Nov 2007 10:38:22 -0500
> > AEtzold@gmx.de wrote:
> > 
> > > Dear all,
> > > 
> > > when using Iconv, I am repeatedly running into
> > > problems.
> > > I tried to run this bit of code:
> > > 
> > > #!/usr/bin/env ruby
> > > $KCODE = 'u'
> > > require 'iconv'
> > > 
> > > s = 'caffè'
> > > 
> > > ic_ignore = Iconv.new('US-ASCII//IGNORE', 'UTF-8')
> > > puts ic_ignore.iconv(s) # => caff
> > > 
> > > ic_translit = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
> > > puts ic_translit.iconv(s) # => caff`e
> > > 
> > > (from here:
> > > http://www.ruby-forum.com/topic/70827), 
> > > but instead of the promised result in the comments above,
> > > I am getting:
> > > 
> > > corr_ebook.rb:29:in `iconv': "\351" (Iconv::InvalidCharacter)
> > >         from corr_ebook.rb:29
> > > 
> > > Why ?
> > > I am using ruby 1.8.6 (2007-03-13 patchlevel 0) [x86_64-linux] (OpenSuse
> > 10.2)
> > > 
> > > Thank you very much!
> > 
> > 
> > Your data is not UTF-8, from the error, most likely iso-8859-1 (or 15)
> > 
> > man iso_8859-1 shows octal 351 as expected.
> > 
> > 351   233   E9     é     LATIN SMALL LETTER E WITH ACUTE
> > 
> > -jh 
> 
> Dear Jonathan,
> 
> thanks for the hint. You are right. I corrected the encoding
> of the file I read the text in from,
> 
> $KCODE = 'u'
> require 'iconv'
> s=IO.readlines("/home/axel/text.txt").to_s
> p s # =>  'caffè'
> 
> ic_translit = Iconv.new('US-ASCII//TRANSLIT', 'UTF-8')
> puts ic_translit.iconv(s) # => caff`e
> 
> However, now I still get
> "caff?"  instead of "caff`e" as promised.
> 


I believe that's a "feature" of ruby iconv. 

$ echo  café | iconv -f UTF-8 -t ASCII//TRANSLIT
cafe

while

s="café"
ic_translit = Iconv.new('ASCII//TRANSLIT', 'UTF-8')
puts ic_translit.iconv(s) 
=> caf?

-jonathan