[ruby-core:118607] [Ruby master Bug#20517] `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string.
From:
"nagachika (Tomoyuki Chikanaga) via ruby-core" <ruby-core@...>
Date:
2024-07-15 13:02:56 UTC
List:
ruby-core #118607
SXNzdWUgIzIwNTE3IGhhcyBiZWVuIHVwZGF0ZWQgYnkgbmFnYWNoaWthIChUb21veXVraSBDaGlr YW5hZ2EpLg0KDQpCYWNrcG9ydCBjaGFuZ2VkIGZyb20gMy4xOiBSRVFVSVJFRCwgMy4yOiBSRVFV SVJFRCwgMy4zOiBET05FIHRvIDMuMTogUkVRVUlSRUQsIDMuMjogRE9ORSwgMy4zOiBET05FDQoN CnJ1YnlfM18yIGNvbW1pdDo5NmE4MjQxOGIyZWZlOThhOTJmMjM5YTlhMWNiZjMwZGQzOTZkMzM1 IG1lcmdlZCByZXZpc2lvbihzKSBjb21taXQ6MDU1NTNjZjIyZDQzZGQ3OGI4ZjMwY2M0NTkxMjMw YjVjMDAwYzUzOC4NCg0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K QnVnICMyMDUxNzogYFJpcHBlci50b2tlbml6ZSgnIlxcTS3jgYIiJylgIHNlcGFyYXRlcyBlbmNv ZGluZyB2YWxpZCBzdHJpbmcgdG8gZW5jb2RpbmcgaW52YWxpZCBzdHJpbmcuDQpodHRwczovL2J1 Z3MucnVieS1sYW5nLm9yZy9pc3N1ZXMvMjA1MTcjY2hhbmdlLTEwOTEzNw0KDQoqIEF1dGhvcjog dG9tcG5nICh0b21veWEgaXNoaWRhKQ0KKiBTdGF0dXM6IENsb3NlZA0KKiBydWJ5IC12OiBydWJ5 IDMuNC4wZGV2ICgyMDI0LTA0LTExVDA4OjU3OjUyWiBtYXN0ZXIgZTdmOGRiOTA3OSkgW3g4Nl82 NC1saW51eF0NCiogQmFja3BvcnQ6IDMuMTogUkVRVUlSRUQsIDMuMjogRE9ORSwgMy4zOiBET05F DQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQp+fn5ydWJ5DQpSaXBw ZXIudG9rZW5pemUgJyJcXE0t44GCIicNCj0+IFsiXCIiLCAiXFxNLVx4RTMiLCAiXHg4MSIsICJc eDgyIiwgIlwiIl0NCn5+fg0KSSBleHBlY3QgYWxsIHRva2VucyB0byBiZSB2YWxpZF9lbmNvZGlu ZyBpZiB0aGUgc291cmNlIHN0cmluZyBpcyB2YWxpZF9lbmNvZGluZy4NClNpbWlsYXIgdG8gaHR0 cHM6Ly9idWdzLnJ1YnktbGFuZy5vcmcvaXNzdWVzLzIwMDMwDQoNCg0KDQotLSANCmh0dHBzOi8v YnVncy5ydWJ5LWxhbmcub3JnLw0KIF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18KIHJ1YnktY29yZSBtYWlsaW5nIGxpc3QgLS0gcnVieS1jb3JlQG1sLnJ1Ynkt bGFuZy5vcmcKIFRvIHVuc3Vic2NyaWJlIHNlbmQgYW4gZW1haWwgdG8gcnVieS1jb3JlLWxlYXZl QG1sLnJ1YnktbGFuZy5vcmcKIHJ1YnktY29yZSBpbmZvIC0tIGh0dHBzOi8vbWwucnVieS1sYW5n Lm9yZy9tYWlsbWFuMy9saXN0cy9ydWJ5LWNvcmUubWwucnVieS1sYW5nLm9yZy8=