[ruby-core:43270] [ruby-trunk - Bug #6137] openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode

Issue #6137 has been updated by Martin Bosslet.


Oh, thanks for the hint! Yes, now it makes sense why MD5 was used. I still think we don't need a crypto-strength hash just to simply bound a value by length, especially if it's not critical to security otherwise. We could use Object#hash for that instead, which could be turned into a binary string for this purpose and will not conflict with any FIPS restrictions.

I need to make sure, however, that the value will not enter any security-critical phase during the actual connection verification. I'll check how the value is used within OpenSSL.
----------------------------------------
Bug #6137: openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode
https://bugs.ruby-lang.org/issues/6137

Author: Jared Jennings
Status: Assigned
Priority: Normal
Assignee: Martin Bosslet
Category: ext
Target version: 
ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]


=begin
I've got a host configured to be compliant with ((<U.S. Federal Information Processing Standard 140-2|URL:http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>)) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved.

When I try to run Puppet's master subcommand, it sets up a secure HTTP server using WEBrick, which in turn uses the openssl module. But in the OpenSSL::SSL::SSLServer class, at source:ext/openssl/lib/openssl/ssl.rb@33695#L149, the MD5 digest is used to make a session ID from a context. On my host this fails as follows:

    /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize': Digest initialization failed.: unknown cipher (OpenSSL::Digest::DigestError)
        from /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize'
        from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
        from /usr/lib/ruby/1.8/openssl/ssl-internal.rb:143:in `initialize'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `new'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `listen'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `collect!'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `listen'
        from /usr/lib/ruby/1.8/webrick/server.rb:63:in `initialize'
        from /usr/lib/ruby/1.8/webrick/httpserver.rb:24:in `initialize'
        from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `new'
        from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `listen'
        [...]

I'm not sure exactly how, but ext/openssl/lib/openssl/ssl.rb from the source tree appears to be installed as /usr/lib/ruby/1.8/openssl/ssl-internal.rb on the system.

I replaced the instantiation of OpenSSL::Digest::MD5 with OpenSSL::Digest::SHA256 on my own system. The puppet master command worked, and no other bad things happened. Accordingly I suggest this change for Ruby in general. - Reasons to make the change:
* Anyone trying to use OpenSSL::SSL::SSLServer who is in the U.S. government, a company contracting with the U.S. government, or possibly a bank, will appreciate if it works. (That's who cares about FIPS 140-2.)
* I haven't seen any migration issues.
* According to my reading of the code, any cryptographic hash will do.

Possible reasons not to make the change:
* SHA256 takes more time than MD5. I haven't checked how often the hash is called. Embedded servers that use OpenSSL::SSL::SSLServer may slow down.
* SHA256 hash values are longer than those of MD5. I don't think the hash values are stored in any variables with fixed size, but I haven't exhaustively confirmed it. 

=end



-- 
http://bugs.ruby-lang.org/