From: "bannable (Joe Truba)" <noreply@...>
Date: 2022-11-18T20:00:03+00:00
Subject: [ruby-core:110818] [Ruby master Bug#19136] OpenSSL::PKey::EC.check_key is useless when linked against OpenSSL 3

Issue #19136 has been reported by bannable (Joe Truba).

----------------------------------------
Bug #19136: OpenSSL::PKey::EC.check_key is useless when linked against OpenSSL 3
https://bugs.ruby-lang.org/issues/19136

* Author: bannable (Joe Truba)
* Status: Open
* Priority: Normal
* ruby -v: 3.1.2, 2.7.2
* Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
When calling `#check_key` against an `OpenSSL::PKey::EC` instance representing an invalid point for the group, the method always returns true.

I believe this is because OpenSSL 3 deprecated `EC_KEY_check_key`, and the underlying call is swapped out for `EVP_PKEY_public_check` in `ruby/openssl` 3+.

However, the `EVP_PKEY_public_check` does not serve the same purpose as `EC_KEY_check_key`.

`EVP_PKEY_public_check` validates only the resulting public component, and does not validate the private component.

**Reproducer**

```ruby
# check.rb
ver = ARGV[0]
gem 'openssl', ver
require 'openssl'

# ECDSA secp384r1 encoded key where the point is not on the curve
pem = <<~INVALID_KEY
-----BEGIN EC PRIVATE KEY-----
MIGkAgEBBDDA1Tm0m7YhkfeVpFuarAJYVlHp2tQj+1fOBiLa10t9E8TiQO/hVfxB
vGaVEQwOheWgBwYFK4EEACKhZANiAASyGqmryZGqdpsq5gEDIfNvgC3AwSJxiBCL
XKHBTFRp+tCezLDOK/6V8KK/vVGBJlGFW6/I7ahyXprxS7xs7hPA9iz5YiuqXlu+
lbrIpZOz7b73hyQQCkvbBO/Avg+hPAk=
-----END EC PRIVATE KEY-----
INVALID_KEY

begin
  result = OpenSSL::PKey::EC.new(pem).check_key
rescue => e
  result = e.message
end

puts format('%25s: %s','RUBY_VERSION', RUBY_VERSION)
puts format('%25s: %s','OPENSSL_LIBRARY_VERSION', OpenSSL::OPENSSL_LIBRARY_VERSION)
puts format('%25s: %s','OPENSSL_VERSION', OpenSSL::VERSION)
puts format('%25s: %s','result', result)
```

**OpenSSL 1.1.1**
```
$ rvm 2.7 do ruby check.rb 2.1.2
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 2.1.2
                   result: EC_KEY_check_key: invalid private key
$ rvm 2.7 do ruby check.rb 3.0.1
             RUBY_VERSION: 2.7.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key

$ rvm 3.1.2 do ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 1.1.1f  31 Mar 2020
          OPENSSL_VERSION: 3.0.1
                   result: EVP_PKEY_public_check: invalid private key
```

**OpenSSL 3.0.2**
```ruby
$ ruby check.rb
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.1
                   result: true
$ ruby check.rb 3.0.0
             RUBY_VERSION: 3.1.2
  OPENSSL_LIBRARY_VERSION: OpenSSL 3.0.2 15 Mar 2022
          OPENSSL_VERSION: 3.0.0
                   result: true
```




-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>