From: Martin Duerst <duerst@...>
Date: 2007-12-13T11:24:19+09:00
Subject: Re: redefineable not operator

Hello Gary,

Here is my general understanding from discussions with Matz
and others:
- In general, Symbols are different if they have different
  encodings.
- The encodings that Ruby allows internally all have to be
  ASCII-based.
- If symbols are only ASCII, different encodings don't matter.

I think the consequences of this are:
- You can't use EBCDIC (but who cares?)
- You have to be a bit carefuly with non-ASCII identifiers
  such as π and friends. But I guess that's good advice
  anyway.

Please note that the last point would help to move to a
more streamlined (encoding-wise) treatment in the future
if we wanted, because writing programs relying on the same
symbol in different encodings being different would really
be silly.

>The whole situation would be simpler if you just dictated that Ruby
>scripts must be in UTF-8. :-)

That won't happen very soon :-(. But we definitely want to make it more
and more easy to use UTF-8.

Regards,   Martin.

At 02:16 07/12/13, Gary Wright wrote:
>
>On Dec 12, 2007, at 12:53 AM, Yukihiro Matsumoto wrote:
>> In message "Re: redefineable not operator"

>> OK, what do you think the best idea to "fix" this?  It's more
>> difficult than one might think at first sight.
>
>I'm trying to understand the string interning.  It seems like  
>rb_intern3()
>works hard to determine if the string to be interned is a lexical token
>of some sort so that an appropriate bit pattern can be selected for the
>symbol reference.
>
>This is difficult because you may be trying to flag lexical symbols
>from an arbitrary number of encodings.   Can the source encoding change
>from one file to the next? I guess the goal is to map the same lexical
>element in *any* encoding to a canonical symbol for that element,  
>correct?
>For example, you need to know that '$$' is the symbol for the special
>PID global regardless of how the '$$' is encoded in the source.  This
>leads to two (or more) byte sequences being mapped to a single symbol.
>
>I think it should be a two step process:
>
>     lexical processing:    (text, encoding)   --->  (canonical-byte- form)
>     symbol processing:     (arbitrary bytes)  ----> symbol
>
>The lexical processing is something internal to the interpreter.  The
>symbol processing is String#intern and Symbol#to_s.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp