[ruby-core:18876] Re: Character encodings - a less radical suggestion

----- Original Message ----- 
From: "Martin Duerst" <[e-mail]>
To: <[e-mail]>
Sent: Thursday, September 25, 2008 11:16 AM

> I have read that mail, but I disagree to use the "source encoding".
> I think it should be possible to use e.g. UTF-8 as a source encoding
> without default_internal being automatically set.

My reasoning is this: if a Ruby programmer puts "Encoding: XXX" at the top 
of his main program, it is saying that she will be using encoding XXX for 
the "constant" strings and regexps in her code, right? If the 
default_internal is different from this then they will get encoding 
compatibitlity problems unless they are careful. This is what 
"default_internal" is aiming to prevent, or at least reduce.

My feeling is that if they then really want to go on to read a different 
encoding without transcoding, they can always open that file with mode 
"r:ext:ext". Yes it's a bit ugly, but I think this is going to be the 
exception rather than the rule. I guess the mode could be extended to 
support something like "r:ext:-" where the "-" in the internal field 
indicates no transcoding. Perhaps that would be more tolerable.

My current thinking is to set default_internal to:
- The -E command line option
- The source encoding if not specified in -E
- Leave it nil (no transcoding) if niether is specified
- Also if "default_internal" is US-ASCII it is reset to UTF-8 automatically 
(can't do any harm and will cope better if they specify an external encoding 
but no internal encoding when opening a file, or their default_external is 
not US-ASCII)

Can you please outline a scenario where you feel that this would be 
unacceptable?

The other problem I have with "default_internal=" is that it's use may be 
confusing to the Ruby programmer. Does it immediately cause the next read of 
a file to use the new value or does it just apply to the next open? (I think 
any implementation would probably be the latter).

> Also, I think it should be possible to set default_internal independent
> of default_external. I can immagine writing an application where I
> always want UTF-8 to be default_internal, but I want it to work with
> all kinds of external encodings. In that case, with your proposal,
> the alternatives would be to have the user write "ruby -E "ext:UTF-8"
> (which means that the user has to figure out his/her external encoding,
> which many may not be experts on), because I cannot use a #!.
>
> Well, in a Japanese mail (ruby-dev:36523), matz made this work
> with -E :utf-8. I guess that's why he is a language designer,
> and I'm not :-).

Yes, I was assuming you could pass either "ext", "ext:int" or ":int" to the 
"-E" option. Sorry if I didn't make that clear.

> Definitely it needs some more work and polishing. This is now being
> discussed seriously on ruby-dev, but please keep your ideas and comments
> comming.
Shame I don't read Japanese!

I know it's after the freeze, but I'll have a go at producing a patch myself 
over this weekend.

Cheers
Mike.