From: Robin Stocker <robin@...>
Date: 2006-02-21T22:53:54+09:00
Subject: Re: accents and String#tr

Xavier Noria wrote:
> I wrote this method
> 
>   def self.normalize_for_sorting(s)
>     return nil unless s
>     norm = s.downcase
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm.tr!('�����', 'aeiou')
>     norm
>   end
> 
> to normalize strings for sorting. This script is UTF-8, everything is 
> UTF-8 in my application, $KCODE is 'u'.
> 
> But it does not work, examples:
> 
>    Andr�s -> andruos
>    L�pez  -> luupez
>    P�rez  -> puorez
> 
> I tried to "force" it with Iconv.conv('UTF-8', 'ASCII', 'aeiou') to no 
> avail. Any ideas?
> 
> -- fxn

Hi,

My guess is that the "tr" method treats its arguments as a string of
bytes. And because characters with accents need more than 1 byte in
UTF-8, #tr doesn't do what you would expect it to. (It's not even tr's
fault, how is it supposed to know that two bytes actually represent a
single character?)

The solution is not to use #tr!, but #gsub!. It isn't as short, but at
least it's right ;)

   norm.gsub!('�', 'a')
   norm.gsub!('�', 'e')
   # and so on...

And because that is against DRY (Don't Repeat Yourself), I would
recommend storing the mapping as a hash:

   accents = { '�' => 'a', '�' => 'e', ... }
   accents.each do |accent, replacement|
     norm.gsub!(accent, replacement)
   end

Regards,
   Robin Stocker