Re: \u escapes in string literals: proof of concept implementation

At 04:19 07/10/23, David Flanagan wrote:
>Yukihiro Matsumoto wrote:
>> Hi,
>> In message "Re: \u escapes in string literals: proof of concept implementation"
>>     on Sat, 20 Oct 2007 09:50:58 +0900, David Flanagan <[e-mail]> writes:
>> |> If the current encoding is not based on the Unicode character
>> |> set, I think \u should:
>> |> |> a. raise compile error,
>> |> b. be converted from Unicode to the encoding, or
>> |> c. make the whole string UTF-8 encoding (and raise compile error
>> |>    if other non-ascii characters are there).
>> |
>> |The new patch attached here does c.  Any string with a \u in it is |forced to utf-8 encoding.  Unless the \u encodes a character in the |ASCII range, in which case it leaves the encoding alone.
>> I'd happy to merge the third patch, once I've been convinced by option
>> c.  Originally I was thinking option b, but option c is much simpler.
>
>Are you happy with \u escapes in Regexp literals?  Note that the \u is interpreted by the lexer rather than being passed on to the regexp engine.

This may be a somewhat controversial proposal, but while we are
adding \u to regular expressions, why not also add \x? Anybody know
the reason why currently \x isn't available in regular expressions?

>I don't have an argument against option b.  I didn't even consider it because I don't know how to that kind of conversion.  Are you still planning to add a method for transcoding of strings in 1.9?

I hope yes. Matz asked me and my student to work on it.
It may take a few more days or weeks.

>If we can't to transcoding explicitly, then I'd argue that making \u do transcoding doesn't make sense.  (Is "transcoding" the right word to use here?)

Yes, definitely.

>Option c was relatively simple. Note, though, that I don't currently implement the part in parentheses. That is, there is no check to see if a string has other multi-byte characters in it.  So with -Ke or -Ks you could start a Kanji string in one of those encodings and then throw in a \u escape. You'd probably end up with an string that was not legal utf-8 nor legal euc or sjis.  The \u escape could raise an error.  But what about the reverse case.  Say I start writing a string with a \u escape, then then add a Kanji character at the end of the string.  I don't believe that the lexer can currently detect that case and complain.  As I understand it, we can use \x to mess up any string, and I don't believe that the lexer applies any sanity checks to string literals.  So if we're not doing that, I'm not sure we need to apply those sanity checks to \u.

I think \x and \u are different. With \u, people are looking up a character
number and expect a character. With \x, it's a byte, and people should
get used to the fact that they really have to know what they are doing.


>On a related note, Martin submitted test cases that expected errors to be raised for string like "\uD800" because (I think) that codepoint is supposed to be part of a surrogate pair and it should be illegal to have it by itself in a string.  He's got a point, but I'm not sure that is something we want to enforce at the \u level.  If the lexer is going to complain about unpaired surrogates,

It should also complain about paired surrogates. There are no surrogate
pairs in UTF-8. \u designates a codepoint/character, but there is no
codepoint or character \uD800, so we should complain.

>I think it needs to catch them when encoded with \x or when they're just in the byte stream of the source file as well.

That would be nice, but I think it's on a different level.
As above, \x are bytes, at the user's disgression.

>Anyway, that is my "argument" for option c.

That things are simpler is quite clear. But option a wouldn't be
difficult to implement, either, I guess. My suggestion is to stay
with option a until we have a better idea of which of b and c is
really needed/implementable/... There may even be mixtures:
Transcode to current file encoding or primary encoding,...,
if it works (i.e. the actual characters can be converted),
otherwise transcode characters included literally to UTF-8.
Not that I'm saying that we need to go there, just that there
are other options that we might consider.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]