From: Ben Nagy <ben@...>
Date: 2006-05-08T19:57:44+09:00
Subject: OpenSSL weird padding

Hi,

I've been using OpenSSL::Cipher to try and talk to an IPSec implementation
and ran into something weird. I read what documentation there is and
consulted google and the archives with little success. I'll try and explain
via irb.

irb(main):669:0> ctext
=>
"Y\322\325T8As\226\360\t\335\3329\257\366\263WW\024<A\2632vg\026\a\353G\217\
237\004"

That is what I receive on the wire

irb(main):670:0> plain
=>
"\010\000\000\f\001\000\000\000\254\020#\200\000\000\000\024\254\2749\350\22
6t\235k\203\375]\200\226\316\210@"

That is the correct decryption, once I trick the Cipher module.

irb(main):671:0> key
=> ":I.\354\246\022O\e"
irb(main):672:0> iv
=> "\234W\211\213k\231+\217"
irb(main):673:0> c=Cipher.new("DES-CBC")
irb(main):674:0> c.key=key
irb(main):675:0> c.iv=iv
irb(main):676:0> c.encrypt                   
irb(main):677:0> wackjob_ctext=c.update(plain)+c.final
=>
"Y\322\325T8As\226\360\t\335\3329\257\366\263WW\024<A\2632vg\026\a\353G\217\
237\004\264\t\263\353H\305\351$"

Note that this is the same as what I am receiving except for the trailing
\264\t\263\353H\305\351$

irb(main):678:0> c.iv=iv
irb(main):679:0> c.decrypt
irb(main):680:0> c.update(wackjob_ctext)+c.final == plain
=> true
irb(main):681:0> c.iv=iv

The ctext I am getting has no wacky last byte, nor should it as far as I can
tell, since it's 8 byte aligned and DES has an 8 byte blocksize. Let's try
it...

irb(main):682:0> c.update(ctext)
=>
"\010\000\000\f\001\000\000\000\254\020#\200\000\000\000\024\254\2749\350\22
6t\235k"

Well that's mostly right, but you stole my last block! Give it up!

irb(main):683:0> c.final
(irb):683: warning: error on stack: error:06065064:digital envelope
routines:EVP_DecryptFinal:bad decrypt
OpenSSL::CipherError: error:06065064:digital envelope
routines:EVP_DecryptFinal:bad decrypt
        from (irb):683:in `final'
        from (irb):683
        from :0

Ow, something blew up. :(

Try again...

irb(main):684:0> c.iv=iv

Let's use any extra 8 bytes, just so we can extract our plaintext...

irb(main):685:0> c.update(ctext+"WHATEVER")==plain
=> true

Hm. Yay, I guess. Just don't call the final method, or...

irb(main):686:0> c.final
(irb):686: warning: error on stack: error:06065064:digital envelope
routines:EVP_DecryptFinal:bad decrypt
OpenSSL::CipherError: error:06065064:digital envelope
routines:EVP_DecryptFinal:bad decrypt
        from (irb):686:in `final'
        from (irb):686
        from :0

Boom, again.

So, using the key, IV and plaintext above, if anyone can get OpenSSL to
produce and decrypt the correct ciphertext without the mystery extra byte I
would be extremely grateful. I was hoping for a "just don't mess with this"
flag.

Oh, one more thing, I also tried to use the 'c.encrypt(key,
iv).update(plain) + c.final' syntax. It produces something, but it's
completely different ciphertext to the output using the explicit calls to
c.key and c.iv as used above. Since there is no documentation it's hard to
work out why. Anyone know?

Thanks for any insight, but I post this mainly in case someone else has the
same problem sometime in the future.

Cheers,

ben