From: Yukihiro Matsumoto <matz@...>
Date: 2006-06-19T22:01:21+09:00
Subject: Re: Unicode roadmap?

Hi,

In message "Re: Unicode roadmap?"
    on Mon, 19 Jun 2006 21:39:33 +0900, "Michal Suchanek" <hramrach@centrum.cz> writes:

|> a), unless either of strings is "ascii" and the other is "ascii"
|> compatible.  This point is arguable.
|
|What is "ascii"? Specifically I would like string operations to suceed
|in cases when both strings are encoded as different subset of Unicode
|(or anything else). ie concatenating an ISO-8859-2 and an ISO-8859-1
|string sould result in UTF-* string, not an error.

Every encoding has an attribute named ascii_compat.  EUC_JP, SJIS,
ISO-8859-* and UTF-8 are declared ascii compatible, where EBCDIC,
UTF-16 and UTF-32 are not.  No other auto conversion shall be done,
since we don't particularly encourage mixed encoding model.

|> |- what to do with IO:
|> |  a) IO will return strings in "raw/bytes"?
|> |  b) IO can be tagged and will return Strings with given econding tag?
|> |  c) IO can be tagged and is by default tagged with global encoding tag?
|> |  d) IO can be tagged, but is not tagged by default, although methods
|> |returning strings (such as read, readlines) will use global encoding tag?
|> |  e) if IO is tagged and one tries to write to it a String with different
|> |encoding, what will happen?
|>
|> c), the global default shall be set from locale setting.
|
|I am not sure this is good for network IO as well. For diagnostics it
|might be useful to set the default to none, and have string raise an
|exception when such strings are combined with other strings.
|
|It is only obvious for STDIN and STDOUT that they should follow the
|locale setting.

Restricting default encoding from locale to STDIO may be a good idea.
There's still open issues, since default encoding from locale is not
covered by the prototype, so we need more experience.

							matz.