From: Martin Duerst Date: 2007-10-21T18:48:48+09:00 Subject: Re: \u escapes in string literals: proof of concept implementation --=====================_738519625==_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hello David and Nobu, At 18:36 07/10/19, Nobuyoshi Nakada wrote: >Hi, > >At Fri, 19 Oct 2007 16:59:17 +0900, >David Flanagan wrote in [ruby-core:12767]: Thanks a lot for working on this. I'm not familar enough with parse.y to actually judge the code, but I'll attach some tests, adapted from my charesc gem (http://rubyforge.org/projects/charesc). I'm not sure they're in the right form to go into the Ruby test suite directly, if anything needs changing, please tell me. I made assert_raise expect SyntaxErrors, but that's just a guess, maybe something different is needed. I don't think I have tested all corners, in particular because the syntax for charesc was a bit different than for the \u. >> But with -Ks or -Ke, it takes whatever codepoint you specify and >> translates it to the appropriate multibyte sequence for SJIS or EUC. (I >> don't know how to test this, however.) If you run the parser in ASCII >> mode, then \u1234 is equivalent to \x12\x34. Similarly for the \u{} >> form. I don't know if this is actually a good idea or not. There is a >> strong case to be made that \u should always create utf-8 bytes >> regardless of the encoding the parser is using. > >For raw byte data, you can use \x. > >If the current encoding is not based on the Unicode character >set, I think \u should: > >a. raise compile error, >b. be converted from Unicode to the encoding, or >c. make the whole string UTF-8 encoding (and raise compile error > if other non-ascii characters are there). I'm in full and complete agreement with Nobu here. I'm not exactly sure which one of b or c to choose, so for the moment, I'd probably go for a, we can always make something legal once we have figured out what the best way is to do so. b. would definitely be in line with how HTML/XML work, and is also what I implemented in charesc. Regards, Martin. #-#-# Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University #-#-# http://www.sw.it.aoyama.ac.jp mailto:duerst@it.aoyama.ac.jp --=====================_738519625==_ Content-Type: application/octet-stream; name="test_backslash_u.rb"; x-mac-type="54455854"; x-mac-creator="74747874" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="test_backslash_u.rb" IyAtKi0gY29kaW5nOiB1dGYtOCAtKi0NCg0KcmVxdWlyZSAndGVzdC91bml0Jw0KY2xhc3MgVGVz dEJhY2tzbGFzaFUgPCBUZXN0OjpVbml0OjpUZXN0Q2FzZQ0KICBkZWYgdGVzdF9iYXNpYw0KICAg IGFzc2VydF9lcXVhbCgnWXVraWhpcm8gTWF0c3Vtb3RvIC0g5p2+5pys6KGM5byYJywNCiAgICAg ICJZdWtpaGlybyBNYXRzdW1vdG8gLSBcdTY3N0VcdTY3MkNcdTg4NENcdTVGMTgiKQ0KICAgIGFz c2VydF9lcXVhbCgnTWF0eiAtIOOBvuOBpOOCguOBqCDjgobjgY3jgbLjgo0nLA0KICAgICAgIk1h dHogLSBcdTMwN0VcdTMwNjRcdTMwODJcdTMwNjggXHUzMDg2XHUzMDREXHUzMDcyXHUzMDhEIikN CiAgICBhc3NlcnRfZXF1YWwoIkFveWFtYSBHYWt1aW4gVW5pdmVyc2l0eSAtIFx4RTlceDlEXHg5 Mlx4RTVceEIxXHhCMVx4RTVceEFEXHhBNlx4RTlceDk5XHhBMlx4RTVceEE0XHhBN1x4RTVceEFE XHhBNiIsDQogICAgICAiQW95YW1hIEdha3VpbiBVbml2ZXJzaXR5IC0gXHU5NzUyXHU1QzcxXHU1 QjY2XHU5NjYyXHU1OTI3XHU1QjY2IikNCiAgICBhc3NlcnRfZXF1YWwoJ0FveWFtYSBHYWt1aW4g VW5pdmVyc2l0eSAtIOmdkuWxseWtpumZouWkp+WtpicsDQogICAgICAiQW95YW1hIEdha3VpbiBV bml2ZXJzaXR5IC0gXHU5NzUyXHU1QzcxXHU1QjY2XHU5NjYyXHU1OTI3XHU1QjY2IikNCiAgICBh c3NlcnRfZXF1YWwoJ+mdkuWxseWtpumZouWkp+WtpicsICJcdTk3NTJcdTVDNzFcdTVCNjZcdTk2 NjJcdTU5MjdcdTVCNjYiKQ0KICAgIGFzc2VydF9lcXVhbCgiTWFydGluIERceEMzXHhCQ3JzdCIs ICJNYXJ0aW4gRFx1MDBGQ3JzdCIpDQogICAgYXNzZXJ0X2VxdWFsKCdNYXJ0aW4gRMO8cnN0Jywg Ik1hcnRpbiBEXHUwMEZDcnN0IikNCiAgICBhc3NlcnRfZXF1YWwoJ8O8JywgIlx1MDBGQyIpDQog ICAgYXNzZXJ0X2VxdWFsKCJNYXJ0aW4gRFx4QzNceEJDcnN0IiwgIk1hcnRpbiBEXHV7RkN9cnN0 IikNCiAgICBhc3NlcnRfZXF1YWwoJ01hcnRpbiBEw7xyc3QnLCAiTWFydGluIERcdXtGQ31yc3Qi KQ0KICAgIGFzc2VydF9lcXVhbCgnw7wnLCAiXHV7RkN9IikNCiAgZW5kDQogIA0KICBkZWYgdGVz dF9zeW50YXhfdmFyaWFudHMNCiAgICAjIGFsbCBoZXggZGlnaXRzDQogICAgYXNzZXJ0X2VxdWFs KCJceEM0XHhBM1x4RTRceDk1XHhBN1x4RThceEE2XHhBQlx4RUNceEI3XHhBRiIsICJcdTAxMjNc dTQ1NjdcdTg5QUJcdUNERUYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhDNFx4QTNceEU0XHg5NVx4 QTdceEU4XHhBNlx4QUJceEVDXHhCN1x4QUYiLCAiXHUwMTIzXHU0NTY3XHU4OUFCXHVDREVGIikN CiAgICBhc3NlcnRfZXF1YWwoIlx4QzRceEEzXHhFNFx4OTVceEE3XHhFOFx4QTZceEFCXHhFQ1x4 QjdceEFGIiwgIlx1MDEyM1x1NDU2N1x1ODlhYlx1Y2RlZiIpDQogICAgYXNzZXJ0X2VxdWFsKCJc eEM0XHhBM1x4RTRceDk1XHhBN1x4RThceEE2XHhBQlx4RUNceEI3XHhBRiIsICJcdTAxMjNcdTQ1 NjdcdTg5YWJcdWNkZWYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhDNFx4QTNceEU0XHg5NVx4QTdc eEU4XHhBNlx4QUJceEVDXHhCN1x4QUYiLCAiXHUwMTIzXHU0NTY3XHU4OWFCXHVDZEVmIikNCiAg ICBhc3NlcnRfZXF1YWwoIlx4QzRceEEzXHhFNFx4OTVceEE3XHhFOFx4QTZceEFCXHhFQ1x4Qjdc eEFGIiwgIlx1MDEyM1x1NDU2N1x1ODlhQlx1Y0RFRiIpDQogIGVuZA0KICANCiAgZGVmIHRlc3Rf ZnVsdG9uDQogICAgIyBleGFtcGxlcyBmcm9tIEhhbCBGdWx0b24ncyBib29rIChzZWNvbmQgZWRp dGlvbiksIGNoYXB0ZXIgNA0KICAgICMgcHJlY29tcG9zZWQgZSdwZSdlDQogICAgYXNzZXJ0X2Vx dWFsKCfDqXDDqWUnLCAiXHUwMEU5XHUwMDcwXHUwMEU5XHUwMDY1IikNCiAgICBhc3NlcnRfZXF1 YWwoJ8OpcMOpZScsICJcdTAwRTlwI3UwMEU5ZSIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEMzXHhB OVx4NzBceEMzXHhBOVx4NjUiLCAiXHUwMEU5XHUwMDcwXHUwMEU5XHUwMDY1IikNCiAgICBhc3Nl cnRfZXF1YWwoIlx4QzNceEE5XHg3MFx4QzNceEE5XHg2NSIsICJcdTAwRTlwXHUwMEU5ZSIpDQog ICAgIyBkZWNvbXBvc2VkIGUncGUnZQ0KICAgIGFzc2VydF9lcXVhbCgnZcyBcGXMgWUnLCAiXHUw MDY1XHUwMzAxXHUwMDcwXHUwMDY1XHUwMzAxXHUwMDY1IikNCiAgICBhc3NlcnRfZXF1YWwoJ2XM gXBlzIFlJywgImVcdTAzMDFwZVx1MDMwMWUiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHg2NVx4Q0Nc eDgxXHg3MFx4NjVceENDXHg4MVx4NjUiLCAiXHUwMDY1XHUwMzAxXHUwMDcwXHUwMDY1XHUwMzAx XHUwMDY1IikNCiAgICBhc3NlcnRfZXF1YWwoIlx4NjVceENDXHg4MVx4NzBceDY1XHhDQ1x4ODFc eDY1IiwgImVcdTAzMDFwZVx1MDMwMWUiKQ0KICAgICMgY29tYmluYXRpb25zIG9mIE5GQy9ELCBO RktDL0QNCiAgICBhc3NlcnRfZXF1YWwoJ8O2ZmZuZW4nLCAiXHUwMEY2XHUwMDY2XHUwMDY2XHUw MDZFXHUwMDY1XHUwMDZFIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4QzNceEI2ZmZuZW4iLCAiXHUw MEY2XHUwMDY2XHUwMDY2XHUwMDZFXHUwMDY1XHUwMDZFIikNCiAgICBhc3NlcnRfZXF1YWwoJ8O2 ZmZuZW4nLCAiXHUwMEY2ZmZuZW4iKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhDM1x4QjZmZm5lbiIs ICJcdTAwRjZmZm5lbiIpDQogICAgYXNzZXJ0X2VxdWFsKCdvzIhmZm5lbicsICJcdTAwNkZcdTAz MDhcdTAwNjZcdTAwNjZcdTAwNkVcdTAwNjVcdTAwNkUiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHg2 Rlx4Q0NceDg4ZmZuZW4iLCAiXHUwMDZGXHUwMzA4XHUwMDY2XHUwMDY2XHUwMDZFXHUwMDY1XHUw MDZFIikNCiAgICBhc3NlcnRfZXF1YWwoJ2/MiGZmbmVuJywgIm9cdTAzMDhmZm5lbiIpDQogICAg YXNzZXJ0X2VxdWFsKCJceDZGXHhDQ1x4ODhmZm5lbiIsICJvXHUwMzA4ZmZuZW4iKQ0KICAgIGFz c2VydF9lcXVhbCgnw7bvrIBuZW4nLCAiXHUwMEY2XHVGQjAwXHUwMDZFXHUwMDY1XHUwMDZFIikN CiAgICBhc3NlcnRfZXF1YWwoIlx4QzNceEI2XHhFRlx4QUNceDgwbmVuIiwgIlx1MDBGNlx1RkIw MFx1MDA2RVx1MDA2NVx1MDA2RSIpDQogICAgYXNzZXJ0X2VxdWFsKCfDtu+sgG5lbicsICJcdTAw RjZcdUZCMDBuZW4iKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhDM1x4QjZceEVGXHhBQ1x4ODBuZW4i LCAiXHUwMEY2XHVGQjAwbmVuIikNCiAgICBhc3NlcnRfZXF1YWwoJ2/MiO+sgG5lbicsICJcdTAw NkZcdTAzMDhcdUZCMDBcdTAwNkVcdTAwNjVcdTAwNkUiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHg2 Rlx4Q0NceDg4XHhFRlx4QUNceDgwbmVuIiwgIlx1MDA2Rlx1MDMwOFx1RkIwMFx1MDA2RVx1MDA2 NVx1MDA2RSIpDQogICAgYXNzZXJ0X2VxdWFsKCdvzIjvrIBuZW4nLCAib1x1MDMwOFx1RkIwMG5l biIpDQogICAgYXNzZXJ0X2VxdWFsKCJceDZGXHhDQ1x4ODhceEVGXHhBQ1x4ODBuZW4iLCAib1x1 MDMwOFx1RkIwMG5lbiIpDQogICAgIyBHZXJtYW4gc2hhcnAgcyAoc3opDQogICAgYXNzZXJ0X2Vx dWFsKCdTdHJhw59lJywgIlx1MDA1M1x1MDA3NFx1MDA3Mlx1MDA2MVx1MDBERlx1MDA2NSIpDQog ICAgYXNzZXJ0X2VxdWFsKCJceDUzXHg3NFx4NzJceDYxXHhDM1x4OUZceDY1IiwgIlx1MDA1M1x1 MDA3NFx1MDA3Mlx1MDA2MVx1MDBERlx1MDA2NSIpDQogICAgYXNzZXJ0X2VxdWFsKCdTdHJhw59l JywgIlN0cmFcdTAwREZlIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4NTNceDc0XHg3Mlx4NjFceEMz XHg5Rlx4NjUiLCAiU3RyYVx1MDBERmUiKQ0KICAgIGFzc2VydF9lcXVhbCgnU3RyYcOfZScsICJc dXs1M31cdXs3NH1cdXs3Mn1cdXs2MX1cdXtERn1cdXs2NX0iKQ0KICAgIGFzc2VydF9lcXVhbCgi XHg1M1x4NzRceDcyXHg2MVx4QzNceDlGXHg2NSIsICJcdXs1M31cdXs3NH1cdXs3Mn1cdXs2MX1c dXtERn1cdXs2NX0iKQ0KICAgIGFzc2VydF9lcXVhbCgnU3RyYcOfZScsICJTdHJhXHV7REZ9ZSIp DQogICAgYXNzZXJ0X2VxdWFsKCJceDUzXHg3NFx4NzJceDYxXHhDM1x4OUZceDY1IiwgIlN0cmFc dXtERn1lIikNCiAgZW5kDQogIA0KICBkZWYgdGVzdF9lZGdlX2Nhc2VzDQogICAgIyBzdGFydCBh bmQgZW5kIG9mIGVhY2ggb3V0ZXIgcGxhbmUNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjRceDhGXHhC Rlx4QkYiLCAiXHUxMEZGRkYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGNFx4ODBceDgwXHg4MCIs ICJcdTEwMDAwMCIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYzXHhCRlx4QkZceEJGIiwgIlx1RkZG RkYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGM1x4QjBceDgwXHg4MCIsICJcdUYwMDAwIikNCiAg ICBhc3NlcnRfZXF1YWwoIlx4RjNceEFGXHhCRlx4QkYiLCAiXHVFRkZGRiIpDQogICAgYXNzZXJ0 X2VxdWFsKCJceEYzXHhBMFx4ODBceDgwIiwgIlx1RTAwMDAiKQ0KICAgIGFzc2VydF9lcXVhbCgi XHhGM1x4OUZceEJGXHhCRiIsICJcdURGRkZGIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjNceDkw XHg4MFx4ODAiLCAiXHVEMDAwMCIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYzXHg4Rlx4QkZceEJG IiwgIlx1Q0ZGRkYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGM1x4ODBceDgwXHg4MCIsICJcdUMw MDAwIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjJceEJGXHhCRlx4QkYiLCAiXHVCRkZGRiIpDQog ICAgYXNzZXJ0X2VxdWFsKCJceEYyXHhCMFx4ODBceDgwIiwgIlx1QjAwMDAiKQ0KICAgIGFzc2Vy dF9lcXVhbCgiXHhGMlx4QUZceEJGXHhCRiIsICJcdUFGRkZGIikNCiAgICBhc3NlcnRfZXF1YWwo Ilx4RjJceEEwXHg4MFx4ODAiLCAiXHVBMDAwMCIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYyXHg5 Rlx4QkZceEJGIiwgIlx1OUZGRkYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGMlx4OTBceDgwXHg4 MCIsICJcdTkwMDAwIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjJceDhGXHhCRlx4QkYiLCAiXHU4 RkZGRiIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYyXHg4MFx4ODBceDgwIiwgIlx1ODAwMDAiKQ0K ICAgIGFzc2VydF9lcXVhbCgiXHhGMVx4QkZceEJGXHhCRiIsICJcdTdGRkZGIikNCiAgICBhc3Nl cnRfZXF1YWwoIlx4RjFceEIwXHg4MFx4ODAiLCAiXHU3MDAwMCIpDQogICAgYXNzZXJ0X2VxdWFs KCJceEYxXHhBRlx4QkZceEJGIiwgIlx1NkZGRkYiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGMVx4 QTBceDgwXHg4MCIsICJcdTYwMDAwIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjFceDlGXHhCRlx4 QkYiLCAiXHU1RkZGRiIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYxXHg5MFx4ODBceDgwIiwgIlx1 NTAwMDAiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGMVx4OEZceEJGXHhCRiIsICJcdTRGRkZGIikN CiAgICBhc3NlcnRfZXF1YWwoIlx4RjFceDgwXHg4MFx4ODAiLCAiXHU0MDAwMCIpDQogICAgYXNz ZXJ0X2VxdWFsKCJceEYwXHhCRlx4QkZceEJGIiwgIlx1M0ZGRkYiKQ0KICAgIGFzc2VydF9lcXVh bCgiXHhGMFx4QjBceDgwXHg4MCIsICJcdTMwMDAwIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjBc eEFGXHhCRlx4QkYiLCAiXHUyRkZGRiIpDQogICAgYXNzZXJ0X2VxdWFsKCJceEYwXHhBMFx4ODBc eDgwIiwgIlx1MjAwMDAiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhGMFx4OUZceEJGXHhCRiIsICJc dTFGRkZGIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RjBceDkwXHg4MFx4ODAiLCAiXHUxMDAwMCIp DQogICAgIyBCTVANCiAgICBhc3NlcnRfZXF1YWwoIlx4RUZceEJGXHhCRiIsICJcdUZGRkYiKQ0K ICAgIGFzc2VydF9lcXVhbCgiXHhFRVx4ODBceDgwIiwgIlx1RTAwMCIpDQogICAgYXNzZXJ0X2Vx dWFsKCJceEVEXHg5Rlx4QkYiLCAiXHVEN0ZGIikNCiAgICBhc3NlcnRfZXF1YWwoIlx4RTBceEEw XHg4MCIsICJcdTA4MDAiKQ0KICAgIGFzc2VydF9lcXVhbCgiXHhERlx4QkYiLCAiXHUwN0ZGIikN CiAgICBhc3NlcnRfZXF1YWwoIlx4QzJceDgwIiwgIlx1MDA4MCIpDQogICAgYXNzZXJ0X2VxdWFs KCJceDdGIiwgIlx1MDA3RiIpDQogICAgYXNzZXJ0X2VxdWFsKCJceDAwIiwgIlx1MDAwMCIpDQog IGVuZA0KDQogICMgVGVzdHMgdG8gbWFrZSBzdXJlIHRoYXQgZGlzYWxsb3dlZCBjYXNlcyBmYWls DQogIGRlZiB0ZXN0X2ZhaWwNCiAgICBhc3NlcnRfcmFpc2UoU3ludGF4RXJyb3IpIHsgIlx4RDgw MCIgfSAgICAgICAjIHN1cnJvZ2F0ZSBibG9jaw0KICAgIGFzc2VydF9yYWlzZShTeW50YXhFcnJv cikgeyAiXHhEQ0JBIiB9ICAgICAgICMgc3Vycm9nYXRlIGJsb2NrDQogICAgYXNzZXJ0X3JhaXNl KFN5bnRheEVycm9yKSB7ICJceERGRkYiIH0gICAgICAgIyBzdXJyb2dhdGUgYmxvY2sNCiAgICBh c3NlcnRfcmFpc2UoU3ludGF4RXJyb3IpIHsgIlx1RDg0N1x1REQ5QSIgfSAjIHN1cnJvZ2F0ZSBw YWlyDQogICAgYXNzZXJ0X3JhaXNlKFN5bnRheEVycm9yKSB7ICJcdWFiYyIgfSAgICAgICAgIyB0 b28gc2hvcnQNCiAgICBhc3NlcnRfcmFpc2UoU3ludGF4RXJyb3IpIHsgIlx1YWIiIH0gICAgICAg ICAjIHRvbyBzaG9ydA0KICAgIGFzc2VydF9yYWlzZShTeW50YXhFcnJvcikgeyAiXHVhIiB9ICAg ICAgICAgICMgdG9vIHNob3J0DQogICAgYXNzZXJ0X3JhaXNlKFN5bnRheEVycm9yKSB7ICJcdSIg fSAgICAgICAgICAgIyB0b28gc2hvcnQNCiAgICBhc3NlcnRfcmFpc2UoU3ludGF4RXJyb3IpIHsg Ilx1ezAwQUJDRH0iIH0gICAjIGxlYWRpbmcgemVyb3MNCiAgICBhc3NlcnRfcmFpc2UoU3ludGF4 RXJyb3IpIHsgIlx1e2FiY2RlZn0iIH0gICAjIHRvbyBsb25nDQogICAgYXNzZXJ0X3JhaXNlKFN5 bnRheEVycm9yKSB7ICJcdXsxMTAwMDB9IiB9ICAgIyB0b28gaGlnaA0KICBlbmQNCmVuZA0K --=====================_738519625==_--