From: jared.jennings.ctr@...
Date: 2014-03-24T23:00:01+00:00
Subject: [ruby-core:61657] [ruby-trunk - Bug #9659] crash in FIPS mode after unchecked algo->init_func failure

Issue #9659 has been updated by Jared Jennings.


I changed the `rb_digest_hash_init_func` typedef from a return type of `void` to `int`, so that the return value of `MD5_Init` could be checked. I changed digest.c to check the return value of `algo->init_func`, which at the time of the crash seems to point at `MD5_Init`, and raise an exception if the function returns 0.

The interpreter still crashes. Running with `gdb` reveals that in my version of OpenSSL the `MD5_Init` function goes sort of like, `{ if (FIPS_mode() ...) { OpenSSLDie(..., "Digest MD5 forbidden in FIPS mode!"); } return private_MD5_Init(...); }`. `OpenSSLDie` goes on to call `abort`. There's no returning 0 for failure in this case.

On a further look at `md5(3)`, I see: "Applications should use the higher level functions `EVP_DigestInit(3)` etc. instead of calling the hash functions directly." Those functions should return a value to indicate failure rather than raising a signal: the `openssl` module was successfully modified to check their return value in #4944, to good effect.

----------------------------------------
Bug #9659: crash in FIPS mode after unchecked algo->init_func failure
https://bugs.ruby-lang.org/issues/9659#change-45916

* Author: Jared Jennings
* Status: Open
* Priority: Normal
* Assignee: 
* Category: ext
* Target version: current: 2.2.0
* ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
This is just like #4944, but in the `digest` extension instead of the `openssl` extension.

On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes:

~~~
  $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')"
  md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode!
  Aborted (core dumped)
~~~

The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure.

I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places:

 * source:ext/digest/digest.c@43668#L551
 * source:ext/digest/digest.c@43668#L589
 * source:ext/digest/digest.c@43668#L627



-- 
https://bugs.ruby-lang.org/