Re: redefineable not operator

On Dec 12, 2007, at 12:53 AM, Yukihiro Matsumoto wrote:
> In message "Re: redefineable not operator"
>     on Wed, 12 Dec 2007 14:25:38 +0900, Dave Thomas  
> <[e-mail]> writes:
>
> |> This actually isn't a new situation.  Ruby 1.8 has
> |> this behavior also:
> |
> |OK, so the move to extend this behavior is wrong.. :)
>
> OK, what do you think the best idea to "fix" this?  It's more
> difficult than one might think at first sight.

I'm trying to understand the string interning.  It seems like  
rb_intern3()
works hard to determine if the string to be interned is a lexical token
of some sort so that an appropriate bit pattern can be selected for the
symbol reference.

This is difficult because you may be trying to flag lexical symbols
from an arbitrary number of encodings.   Can the source encoding change
from one file to the next? I guess the goal is to map the same lexical
element in *any* encoding to a canonical symbol for that element,  
correct?
For example, you need to know that '$$' is the symbol for the special
PID global regardless of how the '$$' is encoded in the source.  This
leads to two (or more) byte sequences being mapped to a single symbol.

I think it should be a two step process:

     lexical processing:    (text, encoding)   --->  (canonical-byte- 
form)
     symbol processing:     (arbitrary bytes)  ----> symbol

The lexical processing is something internal to the interpreter.  The
symbol processing is String#intern and Symbol#to_s.

The whole situation would be simpler if you just dictated that Ruby
scripts must be in UTF-8. :-)

Looking at rb_intern3():

I may be reading the code wrong but it seems there are any number of  
situations
where the first character of the text is compared in an ASCII context  
but that
the remainder of the text is examined relative to the actual encoding.

switch (*m) {
       case '$':
         id |= ID_GLOBAL;
         if ((mb = is_special_global_name(++m, e, enc)) != 0) {
             if (!--mb) enc = rb_enc_from_index(0);
             goto new_id;
         }

Later on the last character of the text is examined in an ASCII context
even though the text may have a non-ASCII encoding.

         if (m[last] == '=') {


The code is sufficiently intricate and new to me that I'm not sure if  
there
is a problem or if I'm just not reading the code carefully enough.


Gary Wright