From: John Joyce <dangerwillrobinsondanger@...>
Date: 2007-09-30T08:34:13+09:00
Subject: Re: Unicode


On Sep 29, 2007, at 2:29 PM, James Edward Gray II wrote:

> On Sep 29, 2007, at 2:13 PM, John Joyce wrote:
>
>> The short version is that UTF-16 is basically wasteful.
>
> That's not always accurate:
>
> $ iconv -f utf-8 -t utf-16 japanese_prose_in_utf8.txt >  
> japanese_prose_in_utf16.txt
> Firefly:~/Desktop$ wc japanese_prose_in_utf8.txt
>       14      66    5921 japanese_prose_in_utf8.txt
> Firefly:~/Desktop$ wc japanese_prose_in_utf16.txt
>       16      45    3968 japanese_prose_in_utf16.txt
>
> James Edward Gray II
>
>
interesting that you would generate more lines, fewer words, and  
fewer bytes (probably explained by fewer words..)
wc defines words as whitespace delimited, Extremely interesting  
considering that Japanese uses no whitespace except in page layout.  
Grammar does not dictate any whitespace at all. At most in Japanese  
prose you might have one whitespace between sentences, perhaps only  
between "paragraphs"

I don't know how iconv handles things. man iconv says it uses iswspace 
(3) which is in wctype.h but I always hate reading those headers.
I tried using iconv on a file in utf-8 to utf-16 and then back again.  
Results are similar, but interstingly, it's no indication of file  
size. Files are the same size
I then tried the same with some code in C++ and similar results occured.
It would seem to be a whitspace issue. I didn't realize this, but it  
does look like utf-8 is generating fewer whitespace characters while  
generating a bigger file...?
I'm curious what the deal is there.

In theory utf-8 should do better than utf-16 for characters in the  
ASCII range...
at least that was my understanding. And assuming code files are  
largely ASCII character sets...
hmm...!?