[ruby-core:37670] [Ruby 1.9 - Bug #4944][Open] crash in FIPS mode after unchecked EVP_DigestInit_ex failure

Issue #4944 has been reported by Jared Jennings.

----------------------------------------
Bug #4944: crash in FIPS mode after unchecked EVP_DigestInit_ex failure
http://redmine.ruby-lang.org/issues/4944

Author: Jared Jennings
Status: Open
Priority: Normal
Assignee: 
Category: ext
Target version: 
ruby -v: ruby 1.9.3dev (2011-06-28 trunk 32273) [i686-linux]


=begin
I've got a host configured to be compliant with ((<U.S. Federal Information Processing Standard 140-2|URL:http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>)) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved. Any attempt to do an MD5 checksum using Ruby's openssl module (OpenSSL::Digest::MD5) presently results in the interpreter quitting with either a SIGSEGV or SIGABRT. This exists both in Ruby 1.8.7 as packaged in Red Hat Enterprise Linux 6.1, and in the nightly snapshot whose (({ruby -v})) you see below.

Here is a script which causes such a crash under FIPS mode:

 require 'openssl'
 md5 = OpenSSL::Digest::MD5.new
 md5 << 'hi'
 puts md5.hexdigest

The problem progresses like this: At source:/ext/openssl/ossl_digest.c#L36, GetDigestPtr fetches the MD5 algorithm using EVP_get_digestbyname or EVP_get_digestbyobj; this goes fine. At line 71, line 125 or line 162, we attempt to initialize the digest with EVP_DigestInit_ex. This returns 0 instead of 1, to indicate failure. The return value is presently ignored. (Even the example usage in my man page for EVP_DigestInit_ex doesn't check the return value!) Later on, either a SIGSEGV happens when a null function pointer is called, or some part of OpenSSL says on stderr,

 digest.c(149): OpenSSL internal error, assertion failed: Digest init previous FIPS forbidden algorithm error ignored

Then it calls abort(), resulting in a SIGABRT. I haven't teased out exactly what leads to each outcome: both seem bad to me.

If the EVP_DigestInit_ex failure is tested for, the openssl module can throw an exception instead of causing an interpreter crash. The attached patch applies against the snapshot and does this.

Earlier discussion of this issue in the Puppet redmine is at ((<URL:http://projects.puppetlabs.com/issues/8120>)) (see note 2 particularly); a patch against 1.8.7, which is the same except for whitespace, is in the ruby-talk message ((<URL:http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/384989>)).
=end



-- 
http://redmine.ruby-lang.org