[ruby-core:57071] [ruby-trunk - Bug #8875] Select is not usable with SSLSocket

Issue #8875 has been updated by headius (Charles Nutter).


akr (Akira Tanaka) wrote:
> 2013/9/8 headius (Charles Nutter) <[e-mail]>:
>  
>  > In JRuby, my first fix was to make IO.select aware of SSLSocket's native buffers by adding a method to query if SSLSocket had buffered data itself. This adds the socket to the list of pending read streams and does not attempt to do a blocking select on it. This fixes the simple issue of sysread/sysread_nonblock reading more data than requested and potentially draining the stream and allows select to work correctly.
>  
>  I think the right fix is that application uses read_nonblock before select.
>  See the document of IO#read_nonblock and OpenSSL::Buffering#read_nonblock.
>  I wrote this problem in [ruby-core:38681].

I would agree, except that users are shown, through examples online and in source, that SSLSocket is "IO-like" and can be used anywhere an IO can be used. IO can do buffered IO and still be selectable. SSLSocket cannot.

This is a shame because without the buffering in buffering.rb, it *would* be feasible to make select work with SSLSocket, as I have done in JRuby.

>  > However, the buffering issue is harder to fix. I believe buffering.rb needs to go away entirely, or at least needs to not buffer data on its own.
>  
>  OpenSSL::Buffering#gets may be difficult to implement efficiently
>  without buffering.

Difficult but not impossible :-) If the buffer in buffering.rb was known to SSLSocket proper and SSLSocket included it when reporting to IO.select that it has data buffered, all would be fine. This could either mean moving the implementations of gets and friends into C code to work directly against the SSLSocket buffers, or moving @rbuffer into a native location so it can be efficiently queried for data.

I may attempt to implement the latter proposal for JRuby, and then we could translate it for MRI. The bottom line is that there are too many buffers and not enough visibility into them. So we need to fix one thing or the other.

- Charlie
----------------------------------------
Bug #8875: Select is not usable with SSLSocket
https://bugs.ruby-lang.org/issues/8875#change-41683

Author: headius (Charles Nutter)
Status: Open
Priority: Normal
Assignee: 
Category: ext/openssl
Target version: 
ruby -v: all
Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN


Because of the various levels of buffering SSLSocket employs, it is not possible to reliably use IO.select to check when it has data available.

SSLSocket wraps a normal IO that it uses for reading and writing unencrypted data. This IO has its own buffers, at the OS/libc level.

Select normally operates against IO, checking whether data has been buffered or is available on the wire. However, in order to decrypt data on the wire, SSLSocket often needs to read more data than it needs, potentially draining the stream. This is problem #1.

This problem can be mitigated by making IO.select know that it's an SSLSocket and that it may have its own buffers.

However, there's another layer of buffering that happens in openssl/buffering.rb, where read, readpartial, read_nonblock, and methods that call them eventually hit fill_rbuf, which can easily drain both the IO buffers and the SSLSocket buffers into a Ruby-land buffer IO.select does not know about.

An example script is here: https://gist.github.com/headius/6477345

In investigating why this hangs on JRuby (under the original assumption that it was a JRuby issue) I realized that fill_rbuff is reading 16k bytes at a time to try to fill its internal buffer. This effectively drains all data in all buffers visible to IO.select, causing select to hang after the first read.

ruby-head (a few months old), Ruby 1.9.3p253, Ruby 1.8.7p358, JRuby (all versions), and Rubinius (all versions) are affected, because we all share buffering.rb which is where the problem lies.

This may be a known issue, but we continue to get bug reports from Ruby users claiming JRuby is failing to support select + SSLSocket correctly. I'd like to figure out if there's anything we as a community can do to fix this.


-- 
http://bugs.ruby-lang.org/