From: Martin Duerst <duerst@...>
Date: 2008-12-19T16:22:40+09:00
Subject: [ruby-core:20688] Re: 1.9 character encoding (was:  encoding ofsymbols)

At 18:24 08/12/18, Brian Candler wrote:
>On Thu, Dec 18, 2008 at 08:43:45AM +0900, danielcavanagh@aanet.com.au wrote:
>> concatenation could be extended too. if one string is a superset of the
>> other then no actual conversion needs to be done
>
>Concatenation raises interesting issues. For example:
>
>   data = "".force_encoding("UTF-8")
>   while chunk = file.read(1024)
>     data << chunk
>   end
>   # what is data.encoding ?
>
>Here each chunk of 1024 bytes may have split multibyte characters at start
>or end. However it's OK to concatenate them, and as long as the file is read
>to the end, the result would be valid UTF-8.

As File#read works on bytes, the above is conceptually wrong.
It should be fixed to:

   data = ""
   while chunk = file.read(1024)
     data << chunk
   end
   data.force_encoding("UTF-8")
   # what is data.encoding ?

Another way to do it is to just read from a file and tell that you
want the result to be UTF-8.

With Ruby 1.9, we get a lot more power for internationalization/
multilingualization. But we also have to learn how to use that
power, the same way we had to learn (or are still learning :-)
other powerful Ruby features.

Regards,    Martin.



#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp