From: Martin Duerst <duerst@...>
Date: 2007-12-18T16:31:57+09:00
Subject: Re: Another last-minute Ruby 1.9 question

Hello David,

At 15:34 07/12/18, David Flanagan wrote:
>Here's another last-minute question, probably for Matz or Nobu...
>
>Are the encoding supported by transcode.c going to be merged with the encodings in the enc/ directory?  Or will only iso-8859-1 be fully supported in 1.9.1?

We have had some discussions on the Japanese list. In terms of
actual functionality, the encodings truely supported by Ruby
internally (mostly equivalent to what you see in enc/) will
always be a subset of the encodings supported by transcode.c.
As a clear example, there is no plan to make Ruby work with EBCDIC
internally, but adding a few EBCDIC variants to transcode.c isn't
a big deal.

What Nobu proposed and implemented is to *label* each string with
a correct encoding. So assume there is a transcoding to an EBCDIC
variant, labeled EBCDIC-FOO. You should then be able to do
    'abc'.encode('EBCDIC-FOO').encoding
and get "EBCDIC-FOO", but this will be to some extent misleading
because EBCDIC-FOO will just be a replica of ASCII-8BIT, and won't
behave like an EBCDIC encoding at all.

But we are not yet 100% sure whether this is the right behavior,
or whether these should just be labeled with an encoding of ASCII-8BIT
or maybe even something more like BINARY.

>(Currently the encoding method of the string returned by encode("iso-8859-15") returns ASCII")

The very latest version I have behaves differently, please see
test/ruby/test_transcode.rb, around line 75:
    assert_equal("D\xFCrst".force_encoding('iso-8859-15'), "D\u00FCrst".encode('iso-8859-15'))

(It may be that you hit the bug that I described in the bottom half
of [ruby-dev:32591]. If yes, please send a test case, thanks.)

>Also, will the encode() method support Japanese encodings in 1.9.1, or only the iso-8859 encodings?

The plan is definitely to support at least Japanese encodings, too.
If everything goes well, at least Shift_JIS and EUC-JP should be in
in a day or two.

>If these are not finished in 1.9.1, is it safe to assume that they will be fixed in some later 1.9.x release?

Very much so. I don't think there will be any limit on the encodings
we can include, but I don't want to throw in everything at once, and
with the tight schedule up to 1.9.1, there is only so much time left :-(.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp