From: "jhawthorn (John Hawthorn) via ruby-core" <ruby-core@...>
Date: 2026-11-12T22:16:32+00:00
Subject: [ruby-core:127455] [Ruby Bug#22542] ASCII-incompatible default_internal mislabels ENV and Dir strings without converting them

Issue #22542 has been reported by jhawthorn (John Hawthorn).

----------------------------------------
Bug #22542: ASCII-incompatible default_internal mislabels ENV and Dir strings without converting them
https://bugs.ruby-lang.org/issues/22542

* Author: jhawthorn (John Hawthorn)
* Status: Open
* ruby -v: ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [x86_64-linux]
* Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN
----------------------------------------
`Encoding.default_internal` accepts ASCII-incompatible encodings like UTF-16LE. IO transcodes into it correctly, but ENV and Dir return the original UTF-8 bytes relabeled as UTF-16LE.

```shell
$ LC_ALL=C.UTF-8 FOO=hello ruby -E UTF-8:UTF-16LE --disable-gems -e 'str = ENV["FOO"]; p str.bytes; p str.encoding; p str.valid_encoding?'
[104, 101, 108, 108, 111]
#<Encoding:UTF-16LE>
false
```

The cause is the shortcut in `rb_external_str_new_with_enc` for ASCII-only input:

``` c
    /* ASCII compatible, and ASCII only string, no conversion in
     * default_internal */
    if ((eidx == rb_ascii8bit_encindex()) ||
        (eidx == rb_usascii_encindex()) ||
        (rb_enc_asciicompat(eenc) && !search_nonascii(ptr, ptr + len))) {
        return rb_enc_str_new(ptr, len, ienc);
    }
```

This skips the conversion and tags the bytes with `ienc`, which is only valid if `ienc` is also ASCII-compatible. IO uses its own transcoding path, so it isn't affected.

Related to #18338, which is the same kind of mislabeling caused by an ASCII-incompatible `default_external` (via the filesystem encoding). 

Should we even support non-ascii-compatible internal and external encodings? It doesn't seem like they've ever worked and so I can't imagine a demand for using them.



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/