\xDE instead of \336 (was: Re: \u escapes in string literals: proof of concept implementation)

At 05:51 07/11/07, Yukihiro Matsumoto wrote:

>    on Tue, 6 Nov 2007 14:58:24 +0900, Martin Duerst 
><[e-mail]> writes:

>Your proposal of using hexadecimal instead of octal in
>[ruby-core:13026] sounds interesting.

Thanks. Encouraged by this, I worked on a tiny patch, and a small
test file, which are attached below. Please take a look.

Please note that I have choosen upper-case hex escapes.
The other choice is lower-case hex escapes. I'm sure that
there will be opinions on this point, but I don't really
have one, so please just chose whichever you prefer.

Please also note that I wasn't as successful for regular expressions
as for Strings and Symbols. Regular expressions seem to keep their
literal representation internally, whereas strings get parsed.
Or I just haven't understood how they work at all yet.

I'm not at all sure whether this covers enough cases, or whether
this goes too far. Probably trying it out will show where things
break. I have run 'make test', and it stops at the same place with
and without the patch, which may not say much. For your reference,
the test that breaks is at line 49 of bootstraptest/test_knownbug.rb.
The following code:
    Regexp.union(
      "a",
      Regexp.new("\x80".force_encoding("euc-jp")),
      Regexp.new("\x80".force_encoding("utf-8")))
should obviously raise an exception, but doesn't.

Regards,    Martin.

P.S.: Just for the record.

>|>Rather, "\x{4366 4544} \x{3f2d 3159}" for both of Shift_JIS and
>|>EUC-JP
>|
>|Can you explain how you got these numbers?
>
>Perhaps JIS 0208 KU for higher byte, and TEN for lower byte, in other
>words, EUC-JP code point with each byte MSB masked.

The second half of your description is correct. But in my understanding,
Ku and Ten start at 1, which would be EUC-JP code point
with each byte masked *and then 32 substracted*.




#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]