From: John Joyce <dangerwillrobinsondanger@...>
Date: 2007-09-30T08:50:37+09:00
Subject: Re: Unicode


On Sep 29, 2007, at 2:29 PM, James Edward Gray II wrote:

> On Sep 29, 2007, at 2:13 PM, John Joyce wrote:
>
>> The short version is that UTF-16 is basically wasteful.
>
> That's not always accurate:
>
> $ iconv -f utf-8 -t utf-16 japanese_prose_in_utf8.txt >  
> japanese_prose_in_utf16.txt
> Firefly:~/Desktop$ wc japanese_prose_in_utf8.txt
>       14      66    5921 japanese_prose_in_utf8.txt
> Firefly:~/Desktop$ wc japanese_prose_in_utf16.txt
>       16      45    3968 japanese_prose_in_utf16.txt
>
> James Edward Gray II
>
>
Scratch that! I must've gone cross-eyed!
My c++ code was indeed smaller file size in utf-8 than utf-16 as I  
expected!
Interestingly, *nix's apparently use utf-32 internally regardless of  
the source encoding... very interesting