[ruby-core:67069] [ruby-trunk - Bug #10633] OpenSSL RangeError on update

Issue #10633 has been updated by Danny Guinther.


I don't have much experience with building Ruby by hand, so part of my issue may be there, or perhaps the example provided by Kenneth should never work, but should blow up in a different way than what he ran into, but I pulled down and built ruby-trunk and I still fail out of the example that Kenneth provided. It doesn't fail in the same way, which makes me think my build is good, but the error is pretty nondescript:

ossl_large.rb:8:in `update': OpenSSL::Cipher::CipherError

That's it. Line 8 is consistent with Kenneth's line 8: 

cipher.update(contents)

Should the provided script work now or is this indicative of something else going on? I tried connecting to the script process w/ strace and there wasn't much informative there. I also played around with gdb, but I don't have enough experience with gdb to actually dig into which of the "ossl_raise(eCipherError, NULL)" calls the script was hitting.

Thoughts?

----------------------------------------
Bug #10633: OpenSSL RangeError on update
https://bugs.ruby-lang.org/issues/10633#change-50585

* Author: Kenneth Hoffmann
* Status: Closed
* Priority: Normal
* Assignee: openssl
* Category: ext/openssl
* Target version: current: 2.2.0
* ruby -v: 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
OpenSSL Cipher throws a RangeError on the update function when given a very large string. I've verified that it happens with a file of 2 or 3GB in size. 1 GB works fine. This has been tested with 2.1.2, 2.1.3, and 2.2rc-1.

```ruby
require 'openssl'
cipher = OpenSSL::Cipher::Cipher.new('aes-256-cbc')
cipher.decrypt
cipher.iv = "fooooooooooooooooooooooooooooooooo"
cipher.key = "barrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr"
f = File.open("<path to a 2 GB file generated with dd>", "r")
contents = f.read;
cipher.update(contents)
```

Running this will return:

```
RangeError: integer 2147483648 too big to convert to `int'
```

File generated with dd:

```
dd if=/dev/zero of=2g.img bs=1 count=0 seek=2G
```




-- 
https://bugs.ruby-lang.org/