From: Nathaniel Talbott <nathaniel@...>
Date: 2003-08-10T02:38:49+09:00
Subject: Ruby/OpenSSL... bug? [LONG]

Well, there are three variables in this problem now (there were four, but I
eliminated DRb), and one of them is broken:

  1. OpenSSL
  2. Ruby binding to OpenSSL
  3. My head

I'm betting the problem isn't #1, since it's so widely used. My hope is that
the problem is #3, but I'm betting on #2. Here's the code:

s.rb:

  require 'socket'
  require 'openssl'

  #require 'patch'

  key = OpenSSL::PKey::RSA.new(512)
  cert = OpenSSL::X509::Certificate.new
  cert.not_before = Time.now
  cert.not_after = Time.now + 3600
  cert.public_key = key.public_key
  cert.sign(key, OpenSSL::Digest::SHA1.new)

  ctx = OpenSSL::SSL::SSLContext.new
  ctx.key = key
  ctx.cert = cert

  svr = TCPServer.new('localhost', 5777)
  STDOUT.sync = true
  loop do
    ns = svr.accept
    ssl = OpenSSL::SSL::SSLSocket.new(ns, ctx)
    ssl.accept
    ssl.sync = true
    loop do
      IO.select([ssl], nil, nil)
      size = ssl.read(5)
      if(size)
        size = size.to_i
      else
        break
      end
      print(".")
      ssl.write("a" * size)
    end
    ssl.close
    ns.close
  end


c.rb:

  require 'socket'
  require 'openssl'
  require 'timeout'

  STDOUT.sync = true

  def ssl_open
    s = TCPSocket.new('localhost', 5777)
    ssl = OpenSSL::SSL::SSLSocket.new(s, OpenSSL::SSL::SSLContext.new)
    ssl.sync = true
    ssl.connect
    yield(ssl)
    ssl.close
    s.close
  end

  def try(ssl, i)
      ssl.write("%05d" % i)
      begin
        timeout(5) do
          ssl.read(i)
        end
      rescue TimeoutError
        return false
      end
      return true
  end

  def try_out(*sizes)
    ssl_open do |ssl|
      sizes.each do |size|
        print("#{size}... ")
        puts(try(ssl, size) ? "OK" : "FAILED")
      end
    end
  end


  puts("BLOCK_SIZE is #{Buffering::BLOCK_SIZE}.")
  try_out(2, 1000, 50000, 2000)


  Buffering::BLOCK_SIZE = 4096
  puts("BLOCK_SIZE is #{Buffering::BLOCK_SIZE}; note that 2000 is OK now.")
  try_out(2, 2000, 50000, 5000)


  Buffering::BLOCK_SIZE = 16384
  puts("BLOCK_SIZE is #{Buffering::BLOCK_SIZE}, which is the SSL window
size. Everything works now.")
  try_out(2, 2000, 5000, 17000, 50000, 99999)

Here's the (c.rb) output:

  ntalbott@proxytest:~$ ruby -v c.rb
  ruby 1.8.0 (2003-08-04) [i386-linux]
  BLOCK_SIZE is 1024.
  2... OK
  1000... OK
  50000... OK
  2000... FAILED
  c3.rb:44: warning: already initialized constant BLOCK_SIZE
  BLOCK_SIZE is 4096; note that 2000 is OK now.
  2... OK
  2000... OK
  50000... OK
  5000... FAILED
  c3.rb:49: warning: already initialized constant BLOCK_SIZE
  BLOCK_SIZE is 16384, which is the SSL window size. Everything works now.
  2... OK
  2000... OK
  5000... OK
  17000... OK
  50000... OK
  99999... OK

The problem clearly revolves around BLOCK_SIZE... and as soon as BLOCK_SIZE
is increased to 16k, which according to
http://www.openssl.org/docs/ssl/SSL_read.html is the maximum record size for
SSLv3/TLSv1, everything always works. For now, I'm going to just use a
BLOCK_SIZE of 16k, but it seems that the underlying problem should be hunted
down and fixed.

Thanks,


Nathaniel

P.S. Oh, and I don't know why 50k always works... your guess is as good as
mine.

<:((><