[ruby-core:109153] [Ruby master Bug#18898] IO#set_encoding with invalid arguments leads to a segfault

Issue #18898 has been updated by javanthropus (Jeremy Bopp).


Thank you for working on this.  While the patch prevents the crash, it does not address the other odd behavior that was reported, namely that the *internal* encoding of the IO instance is set to the default *external* encoding.  Shouldn't it set the internal encoding to either `nil` or `Encoding.default_internal`?

----------------------------------------
Bug #18898: IO#set_encoding with invalid arguments leads to a segfault
https://bugs.ruby-lang.org/issues/18898#change-98294

* Author: javanthropus (Jeremy Bopp)
* Status: Closed
* Priority: Normal
* ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
Save the following to a file and run it:
```ruby
#!/usr/bin/env ruby

Encoding.default_external = 'utf-8'
File.open(__FILE__) do |f|
  f.set_encoding('utf-8', 'invalid')

  printf(
    "default external: %p\ndefault internal: %p\nexternal:         %p\ninternal:         %p\n\n",
    Encoding.default_external,
    Encoding.default_internal,
    f.external_encoding,
    f.internal_encoding
  )

  f.read
end
```

The above script will result in a segfault at `f.read`.  This seems to happen because the call to `#set_encoding` results in the internal encoding of the IO object being set to follow `Encoding.default_external` while also setting the external encoding of the IO object to match.  Ovbiously, there shouldn't be a segfault, but I actually expected the IO object's internal encoding to be set to nil due to the invalid encoding being specified for it.

I was able to reproduce this on all versions of Ruby from 2.7.0 to 3.0.2.



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>