Re: \u escapes in string literals: proof of concept implementation

At 14:07 07/10/22, David Flanagan wrote:
>Martin,
>
>Thank you for the tests!

Glad they were of use.

>There were a few typos I had to correct.

Yes, I expected that, and should have said so. I was doing most
of the changes by hand, with an occasional 'replace' or 'replace all',
without actually testing the tests.

>One #u instead of \u.  A couple of \x instead of \u.  And a few places where you used the \uxxxx form with codepoints that had 5 digits.  I had to change those to use the \u{} form.
>
>I copied your test_basic method to make test_regexp, where I converted the strings to regexps to make sure those work, too.  I had to delete the tests that used \x since that is not supported in Regexp syntax, however.

I don't think that will be a big loss. My tests do double duty with
both actual UTF-8 and \x escape sequences, which in theory at least
shouldn't be necessary (we aren't testing \x, we are testing \u,
after all).

>(I don't know if \u is actually wanted in regexps.  I think it is a good idea, but I wouldn't be surprised if this is controversial.)

I definitely agree that it's a good idea. I can understand that \x
doesn't appear in regular expressions, after all, they are supposed
to work on characters, not on bytes, but then these days, strings
are also working on characters, so that doesn't apply anymore.

>Also, I had to comment out the test_fail method. I based my patch on the existing code for the \x escape. An invalid \x escape sequence (such as "\x") causes the interpreter to exit immediately.  No exception is raised.  This is a really low-level syntax error.  So I don't know how to automate testing of it.

Ah, I see. With something like:
> ruby -e 'eval %q{"\x"}'
I indeed get a SyntaxError, so maybe that's a way to do these tests.

>I'm pleased to say that after these edits, the 3rd version of the patch I posted passes all these tests.

Great!

>I've attached my edited version of your test cases.  Note that the surrogate pair tests you included in the (now commented out) test_fail method would not have passed.  But I think that is a deeper problem with Unicode handling in Ruby and is not really related to the \u escape...

I was under the impression that it's pretty tight. But it seems that
this impression was mainly due to the fact that unpack complained about
redundant UTF-8 sequences. It would be desirable if Ruby didn't allow
the creation of any byte sequences that are not well-formed UTF-8
as defined in the Unicode Standard (Version 5, p. 103, Section 3.9,
clause D92, if anybody is interested). For those who don't want to
look at the online version, or want something they can print,
well-formedness is virtually identical to the (Perl, sorry, I wrote
that quite a while ago) regular expression at the bottom of
http://www.w3.org/International/questions/qa-forms-utf-8,
with the only difference that the ASCII line has to be
simply [\x00-\x7E].

[Oh well, looking at a regular expression like this, I don't
understand why Ruby doesn't allow \x in regular expressions.]

For pack/unpack, I have found the relevant code in pack.c.
I can work on a patch for these unless somebody things there
is a good reason why this shouldn't be fixed.
My suspicion is that there is similar code in other places.
Any hints?


Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]