From: Martin Duerst <duerst@...>
Date: 2008-09-25T10:16:57+09:00
Subject: [ruby-core:18874] Re: Character encodings - a less  radicalsuggestion

At 21:34 08/09/24, Michael Selig wrote:
>On Wed, 24 Sep 2008 21:02:14 +1000, Martin Duerst <duerst@it.aoyama.ac.jp>  
>wrote:

>> Anyhow, I had a look at the code, and it doesn't seem to be that
>> difficult. The function io_extract_encoding_option in io.c
>> seems to be central. I'm attaching a patch, which I hope is
>> a good start. I'm also writing to ruby-dev (in Japanese)
>> because that's where the real experts are.
>> The patch isn't as strict as your proposal with respect
>> to re-setting, but I'm fine either way.
>>
>> I have tested this patch with code like the following
>> (called with -Eutf-8, -Eshift_jis, -Eeuc-jp, and without -E
>> option, in all combinations)
>
>I am not sure if your patch also works correctly with IO#set_encoding.

I'm quite sure it doesn't.
 
>This is absolutely necessary for HTTP data to be transcoded correctly in  
>OpenURI, for example. Please see my previous post. That post also suggests  
>NOT implementing default_internal=, but rather extending the -E command  
>line flag to be -E "ext:int", and if not defined there to use the "source  
>encoding".

I have read that mail, but I disagree to use the "source encoding".
I think it should be possible to use e.g. UTF-8 as a source encoding
without default_internal being automatically set.

Also, I think it should be possible to set default_internal independent
of default_external. I can immagine writing an application where I
always want UTF-8 to be default_internal, but I want it to work with
all kinds of external encodings. In that case, with your proposal,
the alternatives would be to have the user write "ruby -E "ext:UTF-8"
(which means that the user has to figure out his/her external encoding,
which many may not be experts on), because I cannot use a #!.

Well, in a Japanese mail (ruby-dev:36523), matz made this work
with -E :utf-8. I guess that's why he is a language designer,
and I'm not :-).


>I guess your patch may "get the feature in" by the required date, but I  
>feel that it may require a little more thought to get it right.

Definitely it needs some more work and polishing. This is now being
discussed seriously on ruby-dev, but please keep your ideas and comments
comming.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp