From: Martin Duerst <duerst@...>
Date: 2007-09-27T13:09:35+09:00
Subject: Re: arbitrary Unicode characters in identifiers?

Hello David,

At 04:48 07/09/27, David Flanagan wrote:
>Content-Type: text/plain; charset=UTF-8; format=flowed
>X-MIME-Autoconverted: from 8bit to quoted-printable by scmailgw1.scop.aoyama.ac.jp id l8QJmpEu009680
>
>The following program is legal in Ruby 1.9 and prints 6:
>
># -*- coding: utf-8 -*-
>alias ホサ lambda
>テ��= ホサ{ |x,y| x*y}
>puts テ夕2,3]
>
>Is this a good thing?  Is this expected to remain legal, or is the parser likely to start enforcing rules about which characters are legal in identifiers?
>
>(If you can't see the special characters, I'm using the greek letter lambda as an alias for the "lambda" method.  Then I'm creating a lambda and assigning it to a variable that uses the Unicode multiplication sign.)

I have tried this. With 1.9 on cygwin, I get:
unicodeindents.rb:3: identifier x is not valid to set
unicodeindents.rb:3: identifier x is not valid to get

(the 'x' here is actually the Unicode multiplication sign).

So this seems to work halfway (it doesn't complain about lambda,
and it recognizes x as an identifier), but not fully.

I also tried it on 1.8 (WinXP). I get a long list of
Invalid char `\xyz' in expression
(where xyz is the octal number corresponding to the UTF-8 encoding bytes)
occasionally interspersed with a parse error. All this happens on the
fourth line only, again the lambda is okay.

Are these problems Windows-specific (difficult to immagine)?


As for "is this a good thing", from my experience with XML I can
definitely think about two areas where it can be helpful:

- Programming education. For hard-core programmers, English is an
  absolute must. For K12 programming education or so, being able
  to use e.g. Japanese method/variable names while still working
  with a top class programming language (rather than having to use
  one of the several Japanese-based programming languages that
  are mostly extremely simple).

- Work with terminologies where translation would be bothering a lot.
  This is definitely more relevant for a data format such as XML
  than for a programming language, but e.g. trying to translate
  all the buerocratic terms used in some area of the Japanese
  Government greatly increases the friction for everybody.
  See Makoto Murata's talk at the 2005 World Wide Web Conference
  Developer's day (http://www2005.org/keynotes/makoto.pdf)
  (p. 29 and following for some examples, p. 43,... for some
   discussion)


As for the use in examples like the above (mathematical notation),
the lambda seems cute, but the multiplication sign isn't what
most mathematicians would use (they'd just use a dot), for programming,
'*' is too established and 'x' can't work inline (at least not
in current Ruby), so it's not really that much of a benefit.

As for editors that don't support these chanacters, they are
by no way 'weird', and the situation is improving more and more.
Input can also be helped. In many cases, it may be a question of
personal preference. Some people like a graphically dense text,
those tend(ed) to use programming languages such as APL. Others
prefer something more uniform, they usually would go for Lisp.
(of course, preferences in terms of functionality usually also
count, and for more than the preferences in appearance, but
I think the importance of appearance is often underestimated).


>The program might not work for you if you cut-and-pasted it from this message.  I believe the encoding got munged when I cut and pasted it to my mail program.

I definitely know things got garbled on my side. But I don't think
it would have been necessary to use compression for the attachment.


Regards,    Martin.

>So I've also attached a gzipped version of the program to this message to preserve the utf-8 encoding.  You may or may not be able to display the program, but I you ought to be able to run it.
>
>       David
>
>
>


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:duerst@it.aoyama.ac.jp