From: Julian 'Julik' Tarkhanov <listbox@...>
Date: 2006-06-15T20:51:04+09:00
Subject: Re: Unicode roadmap?


On 15-jun-2006, at 13:21, Michal Suchanek wrote:

> On 6/15/06, Julian 'Julik' Tarkhanov <listbox@julik.nl> wrote:
>>
>
>> 5. Preferably separate (and strictly purposed) Bytestring that you
>> get out of Sockets and use in Servers etc. - or the ability to
>> "force" all strings recieved from external resources to be flagged
>> uniformly as being of a certain encoding in _your_ program, not
>> somewhere in someone's library. If flags have to be set by libraries,
>> they won't be set because most developers sadly don't care:
>>
>> http://www.zackvision.com/weblog/2005/11/mt-unicode-mysql.html
>> http://thraxil.org/users/anders/posts/2005/11/01/unicodification/
>
> Where else should the strings be flagged?
They should nog be flagged, because some strings will be flagged and  
some won't and exactly
in the wrong places at the wrong time. See _is_uf_8_ in Perl to  
witness the terrible ugliness of this.

> If you get a web page
> through http request, and the library parses the response for you, it
> should set enconding on the web page. You would never know since you
> only received the page, not the header.

That's why you should distinguish between a ByteArray and a String.

>
>>
>> 6. Unicode-aware strip dealing with weirdo whitespaces (hair space,
>> thin space etc.)
>> 7. And no, as I mentioned - it doesn't handle it properly because
>> the /i modifier is broken, and to deal without it you need to
>> downcase BOTH the regexp and the string itself. Closed circle - you
>> go and get the Unicode gem with tables.
>>
>> All of this can be controlled either per String (then 99 out of 100
>> libraries I use will be getting it wrong - see above) or by a global
>> setting such as $KCODE.
>
> I do not see why libraries should be always wrong. After all, you can
> always fix them. And setting the encoding globally is a bad thing. You
> cannot have strings encoded in different encodings in one process
> then. It looks quite limiting. For one, the web pages that you get
> from various servers (and even the same server) can be in varoius
> encodings.

Of course they can (and will). When I have to approach this I usually  
just snif the encoding of the strings I recieved and then feed them
to iconv and friends before doing any processing. A library that  
downloads stuff off the Internet should be (IMO) aware of
the charset madness and decode the strings for me.

Trust me, when multibyte/Unicode handling is optional, 80% of  
libraries do it wrong. Re-read the links above if you don't believe.

Actually it seems that the solution with an accessor is quite nice,  
but that I had to figure out the hard way after breaking the String  
class
with my hacks and seeing stuff collapse. Apparently the poster of a  
parallel thread finds it inspiring to repeat my experiment _in vitro_  
just for
the academic sake of it.

--
Julian 'Julik' Tarkhanov
please send all personal mail to
me at julik.nl