Treating Unicode specially (was: Re: \u escapes in string literals: proof of concept implementation)

At 12:29 07/10/29, Nobuyoshi Nakada wrote:
>Hi,
>
>At Sat, 27 Oct 2007 05:18:33 +0900,
>David Flanagan wrote in [ruby-core:12980]:
>> In a related matter, Should String.inspect be modified to support \u 
>> escapes?  Perhaps when the primary encoding is not unicode it should do 
>> this?
>
>\x for other encodings?  It means rb_encoding has to know which
>escape to be used.

This is one possibility. If we want to support Unicode well, then
there may be more and more such cases. One way to deal with them
is to add more function pointer slots in the encoding data structure
(OnigEncodingType, as far as I understand).

One of the design goals of OnigEncodingType, and the work by Matz
that led to it, was to minimize the effort needed for implementing
yet another encoding. However, it is possible to reduce this implementation
effort with other means, in particular by providing default implementations.

For details, please see my paper given at the recent 31st
Internationalization and Unicode Conference
(http://www.unicodeconference.org/):
http://www.sw.it.aoyama.ac.jp/2007/pub/IUC31-ruby/Paper.html
(the presentation is also available, it looks most like a presentation
in Opera with full-screen (F11) mode:
http://www.sw.it.aoyama.ac.jp/2007/pub/IUC31-ruby/Presentation.html).
I'm sure I missed a few things when writing the paper; for any fixes,
please contact me directly.

For the point mentioned above, please search for "core primitive" and
"extended primitive".

Of course, for dealing with the fact that Unicode/UTF-8 is in many ways
a special encoding, other approaches are also possible, and in many
cases may be better suited. But we shouldn't avoid implementing something
for Unicode/UTF-8 just because it doesn't apply to other encodings.
(sorry to everybody for the triple negative; it's a typical Japanese
construct :-).

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]