From: Daniel Cavanagh <danielcavanagh@...>
Date: 2008-12-18T19:50:29+09:00
Subject: [ruby-core:20639] Re: 1.9 character encoding (was: encoding of symbols)

On 18/12/2008, at 7:59 PM, Brian Candler wrote:

> On Thu, Dec 18, 2008 at 06:22:28PM +0900, Daniel Cavanagh wrote:
>> and you're against early conversion (ie. at I/O)
>
> That's what ruby-1.9 does out-of-the-box. e.g. if you set the external
> encoding to ISO-8859-1, and the internal encoding to UTF-8, then your
> program will see the source as if it were a stream of UTF-8.

michael selig said "Ruby could have followed the Python route of  
converting everything to Unicode, but that was rejected for various  
good reasons. Also automatic transcoding to solve issues of  
incompatible encodings was also rejected because it causes a number of  
problems, in particular I believe that transcoding isn't necessarilly  
accurate, because for example there may be multiple or ambiguous  
representations of the same character."

i was taking that to mean that ruby will not be doing automatic  
conversion to one encoding. perhaps he just meant by default and that  
the option is there if wanted

>> why do i have to use /u
>> with regexes to match against unicode strings.
>
> Only in Ruby 1.8. Or do you have a counter-example? Here in irb19 from
> 1.9.1-preview2:
>
> irb(main):001:0> foo = "a�b"
> => "a�b"
> irb(main):002:0> foo =~ /b/
> => 2
> irb(main):003:0> foo =~ /�/
> => 1
>
>> why do i have to specify
>> the type of the regex at all?
>
> Only in 1.8

oops. sorry. for some reason i was under the impression that this was  
still necessary. i don't know why

>> why do i have to add in checks before i
>> have to be sure i can concatenate properly?
>
> Not sure what you mean. If you have decided to read in one string in  
> UTF-8,
> and a different string in Shift-JIS, and you really insist on  
> concatenating
> them, then clearly you'll end up with a binary muddle. But what else  
> do you
> expect? If you wrote
>
>    s = str1 + str2
>
> how does Ruby know whether s should take the encoding of str1, or  
> str2?
>
> But if you convert all your data to one encoding at input time, so  
> internal
> processing is consistent, then the problem goes away.
>
> At least, this is how I understand it at the moment :-)

if it's possible to convert everything to one encoding at input time,  
why is it not possible to do so at concatenation time? if neither  
encoding can represent both strings, ruby could pick a different,  
superior encoding. there should theoretically be less problems because  
ruby could always decide on a suitable encoding, but the programmer  
could pick a bad default encoding for input conversion (although why  
anyone would pick anything other than utf-8 is beyond me). and also  
it's possible that a converted string would have ended up not being  
used with strings of a different encodings, so it would have been a  
waste to convert. doing it at comparison would have saved this extra  
unnecessary conversion