From: Daly <aeldaly@...>
Date: 2009-04-23T10:10:45+09:00
Subject: Unicode Question

Hello all,

I download a file from a website and one of the lines look like this:

row = "\0001\0002\0003\000\t\0002\0004\0008\0004\0005\000\t\000C\000o
\000m\000p\000U\000S\000A\000\t\000W
\0006\0003\0001\0000\0003\0000\0006\0000\0001\0000\0001\000\t
\0002\0000\0000\0009\000-\0000\0003\000-\0002\0007\000\t
\0001\0005\000:\0005\0001\000:\0000\0000\000\t\000U\000L\000T\000-\000D
\0002\000P\000K\000\t\0000\000.\0005\000\t\0001\000\t
\0000\000.\0000\0001\000\t\0002\0000\0000\0009\000-\0000\0003\000-
\0002\0008\000\t\0001\0003\000:\0003\0006\000:\0002\0004\000\r\000\n"

On my Mac, if I do:

Iconv.iconv("UTF8", "UCS-2", row)

I get:

["123\t24845\tCompUSA\tW63103060101\t2009-03-27\t15:51:00\tULT-D2PK
\t0.5\t1\t0.01\t2009-03-28\t13:36:24\r\n"]

Which is exactly right. On the production Linux box (Ubuntu 8.04),
doing the same thing yields:

 ["㄀㈀㌀ऀ㈀㐀㠀㐀㔀ऀ䌀漀洀瀀唀匀䄀ऀ圀㘀㌀㄀　㌀　㘀　㄀　㄀ऀ㈀　　㤀ⴀ　㌀ⴀ㈀㜀ऀ㄀㔀㨀㔀㄀㨀　　ऀ唀䰀吀ⴀ䐀㈀倀䬀ऀ　⸀㔀ऀ㄀ऀ
　⸀　㄀ऀ㈀　　㤀ⴀ　㌀ⴀ㈀㠀ऀ㄀㌀㨀㌀㘀㨀㈀㐀ഀ਀"]

I figured it out and fixed it by doing:

Iconv.iconv("UTF8", "UCS-2BE", row)

Which works on both environments. I fixed it by reading about encoding
and trial and error, so I'm left with a working solution, but not
knowing why it works on the Mac but not in Linux. Could someone please
explain?

Thanks,
Ahmed