[ruby-core:43268] [ruby-trunk - Bug #6137][Assigned] openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode

Issue #6137 has been updated by Martin Bosslet.

Status changed from Open to Assigned
Assignee set to Martin Bosslet

Thank you for your detailed analysis! Judging from OpenSSL's documentation [1] the parameter can potentially be any binary data and should not be critical for the overall security, so it might be possible to just use no hash at all. I'll try to find out what the original motivation for using a hash was.

One issue with mandating SHA-256 is that unfortunately there are also users that run Ruby on top of old versions of OpenSSL that do not yet support SHA-2 type hashes, so I'd prefer the hashless version if possible.

I could also find nothing in the codebase that would indicate migration issues in case we changed the value to something else. 

[1] http://www.openssl.org/docs/ssl/SSL_CTX_set_session_id_context.html
----------------------------------------
Bug #6137: openssl: hardcoded MD5 use leads to SSL server failure in FIPS mode
https://bugs.ruby-lang.org/issues/6137

Author: Jared Jennings
Status: Assigned
Priority: Normal
Assignee: Martin Bosslet
Category: ext
Target version: 
ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [i386-linux]


=begin
I've got a host configured to be compliant with ((<U.S. Federal Information Processing Standard 140-2|URL:http://csrc.nist.gov/publications/fips/fips140-2/fips1402.pdf>)) (FIPS 140-2). On this host, the OpenSSL library refuses to do an MD5 checksum, because the MD5 algorithm is not FIPS Approved.

When I try to run Puppet's master subcommand, it sets up a secure HTTP server using WEBrick, which in turn uses the openssl module. But in the OpenSSL::SSL::SSLServer class, at source:ext/openssl/lib/openssl/ssl.rb@33695#L149, the MD5 digest is used to make a session ID from a context. On my host this fails as follows:

    /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize': Digest initialization failed.: unknown cipher (OpenSSL::Digest::DigestError)
        from /usr/lib/ruby/1.8/openssl/digest.rb:55:in `initialize'
        from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:30:in `digest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
        from /usr/lib/ruby/1.8/openssl/digest.rb:46:in `hexdigest'
        from /usr/lib/ruby/1.8/openssl/ssl-internal.rb:143:in `initialize'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `new'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:94:in `listen'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `collect!'
        from /usr/lib/ruby/1.8/webrick/ssl.rb:93:in `listen'
        from /usr/lib/ruby/1.8/webrick/server.rb:63:in `initialize'
        from /usr/lib/ruby/1.8/webrick/httpserver.rb:24:in `initialize'
        from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `new'
        from /usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:33:in `listen'
        [...]

I'm not sure exactly how, but ext/openssl/lib/openssl/ssl.rb from the source tree appears to be installed as /usr/lib/ruby/1.8/openssl/ssl-internal.rb on the system.

I replaced the instantiation of OpenSSL::Digest::MD5 with OpenSSL::Digest::SHA256 on my own system. The puppet master command worked, and no other bad things happened. Accordingly I suggest this change for Ruby in general. - Reasons to make the change:
* Anyone trying to use OpenSSL::SSL::SSLServer who is in the U.S. government, a company contracting with the U.S. government, or possibly a bank, will appreciate if it works. (That's who cares about FIPS 140-2.)
* I haven't seen any migration issues.
* According to my reading of the code, any cryptographic hash will do.

Possible reasons not to make the change:
* SHA256 takes more time than MD5. I haven't checked how often the hash is called. Embedded servers that use OpenSSL::SSL::SSLServer may slow down.
* SHA256 hash values are longer than those of MD5. I don't think the hash values are stored in any variables with fixed size, but I haven't exhaustively confirmed it. 

=end



-- 
http://bugs.ruby-lang.org/