[ruby-core:109150] [Ruby master Bug#18898] IO#set_encoding with invalid arguments leads to a segfault

Issue #18898 has been reported by javanthropus (Jeremy Bopp).

----------------------------------------
Bug #18898: IO#set_encoding with invalid arguments leads to a segfault
https://bugs.ruby-lang.org/issues/18898

* Author: javanthropus (Jeremy Bopp)
* Status: Open
* Priority: Normal
* ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Save the following to a file and run it:
```yaml
#!/usr/bin/env ruby

Encoding.default_external = 'utf-8'
File.open(__FILE__) do |f|
  f.set_encoding('utf-8', 'invalid')

  printf(
    "default external: %p\ndefault internal: %p\nexternal:         %p\ninternal:         %p\n\n",
    Encoding.default_external,
    Encoding.default_internal,
    f.external_encoding,
    f.internal_encoding
  )

  f.read
end
```

The above script will result in a segfault at `f.read`.  This seems to happen because the call to `#set_encoding` results in the internal encoding of the IO object being set to follow `Encoding.default_external` while also setting the external encoding of the IO object to match.  Ovbiously, there shouldn't be a segfault, but I actually expected the IO object's internal encoding to be set to nil due to the invalid encoding being specified for it.

I was able to reproduce this on all versions of Ruby from 2.7.0 to 3.0.2.



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>