From: duerst@...
Date: 2021-06-25T09:39:55+00:00
Subject: [ruby-dev:51069] [Ruby master Bug#12052] String#encode with xml option returns wrong result for totally non-ASCII-compatible encodings

Issue #12052 has been updated by duerst (Martin Dürst).

Status changed from Rejected to Open
Subject changed from String#encode with xml option returns wrong result to String#encode with xml option returns wrong result for totally non-ASCII-compatible encodings

Sorry to @jeremyevans0, but I have to disagree. This is a bug. We can disagree about how important it is to fix this bug, but it's a bug nevertheless. First, xml: :text works correctly in other encodings even if the source and destination encodings match.
```Ruby
"<q&".force_encoding("shift_JIS").encode("shift_JIS", xml: :text)
=> "&lt;q&amp;"
```

The bug is that we process UTF-16LE as if it consisted of 1-byte ASCII-based code units. I still have to identify exactly where and when that happens.

I have changed the subject to indicate what I understand is the extent of the problem. By using "totally", I want to distinguish this from encodings such as Shift_JIS which are also not as ASCII-compatible as say UTF-8, but still more so than UTF-16 (in its various variants).

----------------------------------------
Bug #12052: String#encode with xml option returns wrong result for totally non-ASCII-compatible encodings
https://bugs.ruby-lang.org/issues/12052#change-92645

* Author: nobu (Nobuyoshi Nakada)
* Status: Open
* Priority: Normal
* Assignee: akr (Akira Tanaka)
* Backport: 2.0.0: REQUIRED, 2.1: REQUIRED, 2.2: REQUIRED, 2.3: REQUIRED
----------------------------------------
`String#encode`をASCII非互換エンコーディングから同じエンコーディングへ、`xml:`オプション付きで呼ぶとおかしな結果を返します。
バイナリとして変換してしまっているようです。

```ruby
p "<\0>\0".encode("utf-16le", "utf-16le", xml: :text)
#=> "\u6C26\u3B74\u2600\u7467;"
```



-- 
https://bugs.ruby-lang.org/