From: "jeremyevans0 (Jeremy Evans)" Date: 2021-08-26T22:41:22+00:00 Subject: [ruby-core:105075] [Ruby master Bug#9659] crash in FIPS mode after unchecked algo->init_func failure Issue #9659 has been updated by jeremyevans0 (Jeremy Evans). Status changed from Feedback to Closed The crash in FIPS mode was fixed 7 years ago. The reason this issue was reopened 7 years ago was because of digest/md5 using the low-level OpenSSL API instead of the EVP API. The digest extension removed the OpenSSL engine in commit:2e601c284c9b61c286aa031d91e5198c17b44f00, so the low-level OpenSSL API is no longer used (openssl/ossl_digest.c uses the EVP API). So this can be closed. ---------------------------------------- Bug #9659: crash in FIPS mode after unchecked algo->init_func failure https://bugs.ruby-lang.org/issues/9659#change-93481 * Author: jared.jennings.ctr (Jared Jennings) * Status: Closed * Priority: Normal * ruby -v: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux] * Backport: 2.0.0: DONTNEED, 2.1: DONTNEED ---------------------------------------- This is just like #4944, but in the `digest` extension instead of the `openssl` extension. On my host, which is configured for FIPS 140-2 compliance (this is a U.S. Government security standard), OpenSSL refuses to perform an MD5 checksum. It indicates this refusal when the digest algorithm initialization function is called: this function returns a 0 indicating failure instead of a 1 indicating success. But it's just a bunch of arithmetic; how can it fail? So the return code is ignored. But if the initialization fails, and we go on trying to use the algorithm, the Ruby interpreter crashes: ~~~ $ OPENSSL_FORCE_FIPS_MODE= ruby -rdigest -e "puts Digest::MD5.hexdigest('hi')" md5_dgst.c(78): OpenSSL internal error, assertion failed: Digest MD5 forbidden in FIPS mode! Aborted (core dumped) ~~~ The digest extension, in the `rb_digest_base_alloc`, `rb_digest_base_reset`, and `rb_digest_base_finish` functions, is ignoring the return code of `algo->init_func`. If OpenSSL is present at build time, `algo->init_func` works out to be the `MD5_Init` function from OpenSSL. This function, according to its man page, returns a 1 for success or 0 for failure. I see the problem under Ruby 1.8.7 as patched by Red Hat; I can't easily build the trunk on my system, but it looks like in r43668 the return value still isn't being checked in these three places: * source:ext/digest/digest.c@43668#L551 * source:ext/digest/digest.c@43668#L589 * source:ext/digest/digest.c@43668#L627 ---Files-------------------------------- 002-builtin-indicate-digest-failure.patch (10.4 KB) 001-detect-digest-failure.patch (2.12 KB) 003-digest-openssl-md5-use-evp-api.patch (1.8 KB) -- https://bugs.ruby-lang.org/ Unsubscribe: