Re: Another last-minute Ruby 1.9 question

Martin Duerst wrote:
> Hello David,
> 
> At 15:34 07/12/18, David Flanagan wrote:
>> Here's another last-minute question, probably for Matz or Nobu...
>>
>> Are the encoding supported by transcode.c going to be merged with the encodings in the enc/ directory?  Or will only iso-8859-1 be fully supported in 1.9.1?

Thanks for the clarifications, Martin!

> We have had some discussions on the Japanese list. In terms of
> actual functionality, the encodings truely supported by Ruby
> internally (mostly equivalent to what you see in enc/) will
> always be a subset of the encodings supported by transcode.c.
> As a clear example, there is no plan to make Ruby work with EBCDIC
> internally, but adding a few EBCDIC variants to transcode.c isn't
> a big deal.

If encode can't *always* return a string whose encoding method specifies 
the encoding, then I suggest that it should *always* return a binary 
string.  That's what the name "encode" means, after all: converting 
characters to bytes.

> 
> What Nobu proposed and implemented is to *label* each string with
> a correct encoding. So assume there is a transcoding to an EBCDIC
> variant, labeled EBCDIC-FOO. You should then be able to do
>     'abc'.encode('EBCDIC-FOO').encoding
> and get "EBCDIC-FOO", but this will be to some extent misleading
> because EBCDIC-FOO will just be a replica of ASCII-8BIT, and won't
> behave like an EBCDIC encoding at all.

But what does "behave like" mean for encodings?  As far as I can tell 
from encoding.{c,h}, the only non-trivial feature of single-byte 
encodings is case folding.  It looks like Oniguruma encodings also 
support ctype testing, but I'm not sure that Ruby uses this feature.
I would rather have case-folding and ctype testing on on an EBCIDIC 
encoding abort with an error than apply incorrect ASCII case-folding to 
the string.  My point here is this: can't we trivially implement any 
single-byte encoding if we allow case folding to report an error? And 
isn't reporting an error better than pretending to be ASCII and doing 
case folding wrong?

Also, Oniguruma already defines the ISO-8859 encodings.  When I compile 
them, the striped object files are just 1780 bytes each, so it wouldn't 
be too heavy handed to actually import these encodings into Ruby as 
pre-defined ones, would it?

> But we are not yet 100% sure whether this is the right behavior,
> or whether these should just be labeled with an encoding of ASCII-8BIT
> or maybe even something more like BINARY.
> 
>> (Currently the encoding method of the string returned by encode("iso-8859-15") returns ASCII")
> 
> The very latest version I have behaves differently, please see
> test/ruby/test_transcode.rb, around line 75:
>     assert_equal("D\xFCrst".force_encoding('iso-8859-15'), "D\u00FCrst".encode('iso-8859-15'))
> 
> (It may be that you hit the bug that I described in the bottom half
> of [ruby-dev:32591]. If yes, please send a test case, thanks.)

I can't duplicate anymore.

(I did see one bug that I cannot reproduce: after messing around with 
the encode method and noticing that encodings are dynamically added to 
the array returned by Encoding.list, I saw <Encoding:iso-8859-1> change 
to <Encoding:black-diamond-graphic-character>  Not actually those words, 
but a character that looked like a diamond.  Somehow the name of these 
dynamically generated encoding objects got corrupted.  I can't reproduce 
it, however.  Sorry Nobu!)

> 
>> Also, will the encode() method support Japanese encodings in 1.9.1, or only the iso-8859 encodings?
> 
> The plan is definitely to support at least Japanese encodings, too.
> If everything goes well, at least Shift_JIS and EUC-JP should be in
> in a day or two.

Great!  Thanks!

>> If these are not finished in 1.9.1, is it safe to assume that they will be fixed in some later 1.9.x release?
> 
> Very much so. I don't think there will be any limit on the encodings
> we can include, but I don't want to throw in everything at once, and
> with the tight schedule up to 1.9.1, there is only so much time left :-(.
> 
> Regards,    Martin.
> 
> 
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
> #-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]     
> 
> 
>