[ruby-core:33239] Re: not use system for default encoding

(2010/11/17 21:05), Brian Candler wrote:
> NARUSE, Yui wrote on 2010-11-15 11:07:
>> This is what Japanese people often say "Americans don't consider
>> non-ASCII
>
> Sure, many people want to handle non-ASCII text. But:
>
> * I would consider all non-Unicode character sets to be legacy, do you
> disagree?

Yes, they are legacy.
But it is different problem whether we can throw away them or not.

> * ruby 1.9's model doesn't handle stateful encodings like ISO-2022-JP,
> so these need transcoding at the edge anyway

Yes.

> * hence why not just transcode everything that's not Unicode into Unicode?

Conversion table from non Unicode to Unicode or back is not clear.
"XML Japanese Profile" describes about this confusion.
http://www.w3.org/Submission/japanese-xml/


> I would choose UTF-8 as the internal Unicode representation, since the
> majority of external Unicode data already UTF-8.  (*)

"already" or "more and more" is arguable, but I almost agree.

> Then you end up with the design used by both Python 3.0 and Erlang: you have
> two data types, one for binary strings, and one for UTF-8 text.  (I should
> add this to the document as an explicit alternative)

Python 3.0's internal representation is UTF-16/UTF-32.
I don't know Erlang.

How we walk around binary strings and Unicode strings is big design problem.
Before designing it I can't evaluate it.

> This would wipe out most of the complexity associated with ruby 1.9 at a
> stroke. What you would lose is:
>
> * the ability to handle things like EUC-JP or GB2312 "natively", that is,
> without transcoding them to UTF-8 and back
> * the ability to write ruby programs in non-UTF-8 character sets
>
> How big a loss is that?
>
> (*) There's an argument which says use UTF-16 or UTF-32 internally as it's
> better suited to character indexing.  I would say that this is outweighed by
> the extra RAM bandwidth used, and the fact that most data is UTF-8 so would
> have to be transcoded.

Why Rails3 still supports legacy encodings  may answer it.

>>> Have a universally-compatible "BINARY" encoding.
>>> Any operation between BINARY and FOO gives encoding BINARY,
>>> and transcoding between BINARY and any other encoding is a null operation.
>>
>> This will hide unexpectedly mixed BINARY string.
>> You'll realize hard to debug such strings.
>
> I would much rather have a program which outputs a plausible binary string
> from its inputs than one which crashes given unexpected data.  ruby 1.9
> hugely magnifies the number of unit test cases to achieve coverage of these
> edge cases.

Characters have huge number of edges.
Those edge will be still sharp even if the language only support Unicode.

>>> Treat invalid characters in the same way as String#[] does,
>>> i.e. never raise an exception. In particular, regexp matching always succeeds.
>>
>> This will raise security issue.
>
> In what way is it a security issue?  Why is it not a security issue that
> String#[] doesn't error?  Why is it not a security issue that 'sed' handles
> such files successfully?

See http://www.infoq.com/news/2009/09/rails-vulnerabilities

-- 
NARUSE, Yui  <[e-mail]>