From: Xavier Noria <fxn@...>
Date: 2007-12-05T20:27:14+09:00
Subject: Re: iconv problems with different machines

On Dec 5, 2007, at 12:14 PM, Raymond O'Connor wrote:

> Actually I found some other posts about this same issue from awhile
> ago... Appears there's no solution.
>
> I stopped using the iconv library and instead switched to the iconv
> system command and that seems to work.  Not the best solution, but at
> least it works....

I have not been able to understand where is exactly the difference,  
but looks like depending on the system/version/something the  
transliteration tables are just different. At ASPgems we wrote this  
hand-crafted normalizer which we know is portable for sure (note that  
it uses Rails #chars and does a bit more stuff, but you see the idea):

   def self.normalize(str)
     return '' if str.nil?
     n = str.chars.downcase.strip.to_s
     n.gsub!(/[àáâãäåāă]/,    'a')
     n.gsub!(/æ/,            'ae')
     n.gsub!(/[ďđ]/,          'd')
     n.gsub!(/[çćčĉċ]/,       'c')
     n.gsub!(/[èéêëēęěĕė]/,   'e')
     n.gsub!(/ƒ/,             'f')
     n.gsub!(/[ĝğġģ]/,        'g')
     n.gsub!(/[ĥħ]/,          'h')
     n.gsub!(/[ììíîïīĩĭ]/,    'i')
     n.gsub!(/[įıijĵ]/,        'j')
     n.gsub!(/[ķĸ]/,          'k')
     n.gsub!(/[łľĺļŀ]/,       'l')
     n.gsub!(/[ñńňņʼnŋ]/,      'n')
     n.gsub!(/[òóôõöøōőŏŏ]/,  'o')
     n.gsub!(/œ/,            'oe')
     n.gsub!(/ą/,             'q')
     n.gsub!(/[ŕřŗ]/,         'r')
     n.gsub!(/[śšşŝș]/,       's')
     n.gsub!(/[ťţŧț]/,        't')
     n.gsub!(/[ùúûüūůűŭũų]/,  'u')
     n.gsub!(/ŵ/,             'w')
     n.gsub!(/[ýÿŷ]/,         'y')
     n.gsub!(/[žżź]/,         'z')
     n.gsub!(/\s+/,           ' ')
     n.delete!('^ a-z0-9_/\\-')
     n
   end

-- fxn