[ruby-core:50292] [ruby-trunk - Bug #7463][Open] OpenSSL::TestSSL#test_npn_* fail with "string contains null byte" on big-endian machines

Issue #7463 has been reported by ngoto (Naohisa Goto).

----------------------------------------
Bug #7463: OpenSSL::TestSSL#test_npn_* fail with "string contains null byte" on big-endian machines
https://bugs.ruby-lang.org/issues/7463

Author: ngoto (Naohisa Goto)
Status: Open
Priority: Normal
Assignee: MartinBosslet (Martin Bosslet)
Category: ext
Target version: 
ruby -v: ruby 2.0.0dev (2012-11-28) [sparc64-solaris2.10]


On Solaris10 running in sparc64 CPU, during make test-all, the following 4 tests raise "ArgumentError: string contains null byte". Patch attached fix the bug.

 OpenSSL::TestSSL#test_npn_protocol_selection_ary = 0.05 s = E
 OpenSSL::TestSSL#test_npn_protocol_selection_cancel = 0.05 s = E
 OpenSSL::TestSSL#test_npn_protocol_selection_enum = 0.05 s = E
 OpenSSL::TestSSL#test_npn_selected_protocol_too_long = 0.05 s = E

  49) Error:
 test_npn_protocol_selection_cancel(OpenSSL::TestSSL):
 ArgumentError: string contains null byte
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `initialize'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `new'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `accept'
     /HOMEDIR/test/openssl/utils.rb:242:in `block in server_loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `server_loop'
     /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'
 
  50) Error:
 test_npn_protocol_selection_ary(OpenSSL::TestSSL):
 ArgumentError: string contains null byte
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `initialize'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `new'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `accept'
     /HOMEDIR/test/openssl/utils.rb:242:in `block in server_loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `server_loop'
     /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'
 
  51) Error:
 test_npn_protocol_selection_enum(OpenSSL::TestSSL):
 ArgumentError: string contains null byte
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `initialize'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `new'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `accept'
     /HOMEDIR/test/openssl/utils.rb:242:in `block in server_loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `server_loop'
     /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'
 
  53) Error:
 test_npn_selected_protocol_too_long(OpenSSL::TestSSL):
 ArgumentError: string contains null byte
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `initialize'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `new'
     /HOMEDIR/.ext/common/openssl/ssl.rb:172:in `accept'
     /HOMEDIR/test/openssl/utils.rb:242:in `block in server_loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `loop'
     /HOMEDIR/test/openssl/utils.rb:251:in `server_loop'
     /HOMEDIR/test/openssl/utils.rb:288:in `block in start_server'

The cause of the bug is the following code in ssl_npn_encode_protocol_i in ext/openssl/ossl_ssl.c.

     int len = RSTRING_LENINT(cur);                                             
     if (len < 1 || len > 255)                                                  
        ossl_raise(eSSLError, "Advertised protocol must have length 1..255");   
     /* Encode the length byte */                                               
     rb_str_buf_cat(encoded, (const char *) &len, 1);                           

The above code treats "int len" as byte sequence that differs between little-endian and big-endian. In big-endian machines, when 1<=len<=255, the first byte of "len" is always 0.



-- 
http://bugs.ruby-lang.org/