Re: Still exists: Problems with \M-x in utf-8 encoded strings

At 09:32 07/11/20, David Flanagan wrote:
>Wolfgang Nテ。dasi-Donner wrote:
>> I think the best way to solve this is not to allow "\M-."
>> constructs, if the resulting String object doesn't have an
>> encoding "<Encoding:ASCII-8BIT>", "<Encoding:ISO-8859-1>", or
>> "Binary", which isn't in the list of encodings in the actual
>> snapshot.
>> I hope I will bring better news tomorrow ;-)
>> Wolfgang Nテ。dasi-Donner
>> 
>
>The \M escape is a weird one, but it is really uncommon.  I'd be more worried about the \xxx octal escape an the \xhh hex escape, both of which allow the insertion of arbitrary bytes into a UTF-8 string.
>
>Right now when running with -Ku, any string with a byte that has the high-bit set is given UTF-8 encoding.  Perhaps it would be better to make those strings binary, and to raise an error if a string literal mixes multi-byte characters with single-byte characters with the high bit set.

One thing to consider is backwards compatibility. In 1.8, the only
escaping syntax for UTF-8 is byte-by-byte. Making such strings
binary just because they use bytewise escaping (rather than because
they use non-UTF-8 byte patterns) may be going too far.


>It is possible that I introduced this behavior with my various patches for \u escapes.  I thought I was just following the existing behvavior, however.

That's my guess, because that's behavior that came from 1.8.

Regards,    Martin.


#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[e-mail]