From: jared.jennings.ctr@...
Date: 2014-10-21T20:31:36+00:00
Subject: [ruby-core:65833] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

Issue #9659 has been updated by Jared Jennings.


Nobuyoshi Nakada wrote:
> Is EVP API necessary?

The EVP API has been recommended over the old digest-specific API for [almost fifteen years](http://git.openssl.org/gitweb/?p=openssl.git;a=commitdiff;h=4facdbb5fa9d791fc72dc78b9c3512ea1384df33#patch3). It seems that EVP might [automatically use hardware acceleration](http://stackoverflow.com/a/25978145) where possible. And if EVP is not used, Ruby crashes on the secure systems used by banks and governments, with no indication of which Ruby code caused the problem. 

Nobuyoshi Nakada wrote:
> I've reverted it because of segfaults on many platforms.

Since EVP is so old already, any problem is likely due somehow to my code. I'd like to fix this. Can you share any further details?

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-49573

* Author: Jared Jennings
* Status: Feedback
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: DONTNEED, 2.1: DONTNEED
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627

---Files--------------------------------
002-builtin-indicate-digest-failure.patch (10.4 KB)
001-detect-digest-failure.patch (2.12 KB)
003-digest-openssl-md5-use-evp-api.patch (1.8 KB)


-- 
https://bugs.ruby-lang.org/