[ruby-dev:49671] [Ruby trunk Bug#12504][Assigned] test failure of openssl with ubuntu 16.04

Issue #12504 has been updated by Kazuki Yamaguchi.

Status changed from Open to Assigned
Assignee set to Kazuki Yamaguchi

I reproduced it, and can avoid it by applying this.

~~~diff
diff --git a/test/openssl/test_pkey_ec.rb b/test/openssl/test_pkey_ec.rb
index e05b70a..e15d875 100644
--- a/test/openssl/test_pkey_ec.rb
+++ b/test/openssl/test_pkey_ec.rb
@@ -14,12 +14,12 @@ def setup
     OpenSSL::PKey::EC.builtin_curves.each do |curve, comment|
       group = OpenSSL::PKey::EC::Group.new(curve)
 
-      key = OpenSSL::PKey::EC.new(group)
-      key.generate_key!
-
       # Oakley curves and X25519 are not suitable for signing
       next if ["Oakley", "X25519"].any? { |n| curve.start_with?(n) }
 
+      key = OpenSSL::PKey::EC.new(group)
+      key.generate_key!
+
       @groups << group
       @keys << key
     end
~~~

The following code sometimes causes the crash (also with Ruby 2.3.0), I didn't dig any deeper but I suspect there is a bug in Ubuntu's patched OpenSSL.

~~~ruby
require "openssl"

begin
  OpenSSL::PKey::EC.new("Oakley-EC2N-4").generate_key
rescue
  p $!
end
OpenSSL::PKey::RSA.new(512)
~~~

~~~
#<OpenSSL::PKey::ECError: EC_KEY_generate_key: pairwise test failed>
fips.c(139): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE
Aborted
~~~

----------------------------------------
Bug #12504: test failure of openssl with ubuntu 16.04
https://bugs.ruby-lang.org/issues/12504#change-59266

* Author: Hiroshi SHIBATA
* Status: Assigned
* Priority: Normal
* Assignee: Kazuki Yamaguchi
* ruby -v: 
* Backport: 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN
----------------------------------------
CI is broken at OpenSSL Ubuntu 16.04(xenial). It shows following log.

```
[ 1071/16956] OpenSSL::TestEOF1#test_eof_0fips.c(139): OpenSSL internal error, assertion failed: FATAL FIPS SELFTEST FAILURE
```

see full of log. http://rubyci.s3.amazonaws.com/ubuntu/ruby-trunk/log/20160618T063004Z.fail.html.gz

version of openssl on above instance is here:

```
Package: openssl
Version: 1.0.2g-1ubuntu4.1
(snip)
```





-- 
https://bugs.ruby-lang.org/