[ruby-core:120162] [Ruby master Bug#20937] "can't set length of shared string" error when using OpenSSL::Cipher#update with buffer

Issue #20937 has been updated by rhenium (Kazuki Yamaguchi).


The exception from `rb_str_set_len()` is after the fact, indicating it has illegally written to `RSTRING_PTR(str)` of a shared string.

`Cipher#update` uses `rb_str_resize()` to allocate enough room for the output, but apparently it won't (and never did in previous versions of Ruby) make it independent when the String happens to have the expected length already.

I made an alternative PR: https://github.com/ruby/openssl/pull/824

----------------------------------------
Bug #20937: "can't set length of shared string" error when using OpenSSL::Cipher#update with buffer
https://bugs.ruby-lang.org/issues/20937#change-110917

* Author: akiellor (Andrew Kiellor)
* Status: Open
* Assignee: rhenium (Kazuki Yamaguchi)
* ruby -v: ruby 3.3.6 (2024-11-05 revision 75015d4c1f) +YJIT [arm64-darwin23]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
We've observed a recurring "can't set length of shared string" error in production emerging from the `aws-sdk-s3` library when using it's client encryption features. The sdk in this mode uses OpenSSL::Cipher in decrypt mode with a String buffer. It appears that under some circumstances the buffer becomes a "shared string" and is no longer compatible with the requirements of OpenSSL::Cipher#update.

I've attached a reproduction scenario using only the ruby standard library.

``` shell
$ ruby -v
ruby 3.3.6 (2024-11-05 revision 75015d4c1f) +YJIT [arm64-darwin23]
$ ruby scripty.rb
scripty.rb:32:in `update': can't set length of shared string (RuntimeError)
	from scripty.rb:32:in `block (2 levels) in <main>'
	from scripty.rb:31:in `each'
	from scripty.rb:31:in `block in <main>'
	from /nix/store/fhmbmmzr4h5yax66nl2x44rrdf039b3s-ruby-3.3.6/lib/ruby/3.3.0/tempfile.rb:447:in `create'
	from scripty.rb:26:in `<main>'
```

The attached script has a few different read patterns, some fail and some do not.

Credit for isolating this issue goes to [@simoleone](https://bugs.ruby-lang.org/users/55425).

---Files--------------------------------
scripty.rb (1.15 KB)


-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/