[ruby-core:19562] Re: String literal encoding (Was: Default source encoding (Was: [Bug #680] csv.rb: CSV.parse is toolate when encoding is mismatch))

On Mon, 27 Oct 2008 23:07:16 +1100, Nobuyoshi Nakada <[e-mail]>  
wrote:

> Michael Selig wrote in [ruby-core:19540]:
>> But my original point is that 1.8 scripts exist which contain multibyte
>> characters (eg UTF-8) which work fine under 1.8 without-K, but will fail
>> under 1.9 unless a magic comment or -K is provided.
>
> It just seemed working by chance.

Actually I think that any ruby program which uses ASCII everywhere except  
in string literals will work without -K in 1.8. I think that this might be  
a reasonably common scenario, at least in countries which use UTF-8 or an  
ISO 8859 variant (outside the ASCII domain). These scripts will then fail  
in 1.9 without magic comment or -K.

>> > But it's very ambiguous and dangerous to imply encodings.  We
>> > can't trust locale for this purpose, at least.
>>
>> It's a trade-off between that and backward compatibility. I think the
>> "danger" is not high and it gives backward compatibility, so my vote  
>> would
>> be to use it.
>
> And it will suddenly crash or behave weirdly by moving other
> locales.

Indeed - that's the danger. But I think the cost of losing backward  
compatibility is worse.
How bad is this "danger" in reality? Simple scripts are often written just  
to be used on the same computer or LAN. There are probably lots of these,  
and there is almost certainly no danger there. I think that when people  
send out a script to other countries it would be perfectly reasonable to  
say that it must be run with -KU (or whatever) under 1.9 if their locale  
is not UTF-8 (or whatever). If they don't, then there is a high chance it  
will fail with an obvious error (eg: illegal multibyte char) - not a big  
deal to diagnose. There is a very small chance that it may run and behave  
weirdly. But then I would think that under 1.8 similar weird behaviour  
would have happened anyhow.

> Anyway, I think I understand the needs to specify source
> encoding without magic comments.  Is the option for that
> purpose an acceptable solution?

In my opinion it is an acceptable workaround to add -K to RUBYOPT  
environment variable, but it is far from ideal.
My preference is still to default to the locale encoding like -e does.
Another alternative may be to default to ASCII-8BIT (which would be even  
more 1.8 compatible :) ), but I haven't thought through if there are any  
other repercussions to this.

Losing backward compatibility for "simple" ruby scripts is a major  
negative IMHO.

Martin wrote "It may be somewhat suboptimal in the transition phase, but  
...."
If I may say, I think that you guys might be underestimating the grief  
caused when people upgrade to 1.9 or 2.0 and find that even simple scripts  
no longer work.

Cheers
Mike