From: Young Hyun <youngh@...>
Date: 2009-07-01T02:58:18+09:00
Subject: [ruby-core:24098] Re: [Bug #1696] http downloads are unuseably slow


On Jun 29, 2009, at 2:12 PM, James Gray wrote:

> On Jun 29, 2009, at 1:38 PM, Eero Saynatkari wrote:
>
>> Excerpts from Tanaka Akira's message of Mon Jun 29 21:17:58 +0300  
>> 2009:
>>> In article <4a464441bf3f7_13bd3907d016634@redmine.ruby-lang.org>,
>>> Steven Hartland <redmine@ruby-lang.org> writes:
>>>
>>>> This I've tracked down to the net/protocol module and the  
>>>> Net::BufferIO::rbuf_fill method which is using a timeout block to  
>>>> wrap the @io.sysread(1024) call. This is clearly an extremely bad  
>>>> way to do this, totally crippling the performance.
>>>
>>> There is a reson to use timeout().  It is used to fix
>>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=265429
>>>
>>> Before the fix, IO.select was used in rbuf_fill.
>>
>> On the face of it, using timeout does not seem like the
>> optimal fix to the problem...speaking of, can anyone
>> reproduce this issue currently, or has it gotten fixed
>> otherwise in the 5 intervening years?
>
> Well, this definitely sounds like the kind of thing I am seeing.

For what it's worth, I've been fighting a similar problem for a while  
now, trying Ruby 1.8.5, 1.8.6, and 1.8.7.  The circumstances are  
similar: TCP connection with SSL, Ruby hanging on sysread, and very  
rare failure (once every 3 months with persistent connections).

One possibility is that IO.select is returning a false readiness  
notification, so a subsequent sysread hangs.  Linux manpages note that  
select(2) "may report a socket file descriptor as 'ready for reading',  
while nevertheless a subsequent read blocks" [1], and the recommended  
solution is to work with nonblocking sockets.  (Incidentally, my  
failures were under FreeBSD.)  However, until Ruby 1.9, SSL sockets  
didn't support nonblocking I/O, so we were always susceptible to this  
problem; we just lucked out because of the rarity of this failure mode.

Another possibility, and the one I have some suspicion of, is that  
there's some pathological interaction with SSL; that is, IO.select  
indicates readiness (and there actually are some bytes available to  
read), the client invokes SSLSocket#sysread, and the SSL library does  
some low-level SSL re-negotiations (which according to the SSL  
protocol can happen at any time) and then blocks waiting on actual  
user-level data.       Or instead of a low-level SSL re-negotiation,  
perhaps the connection with the peer is lost (say, the remote box  
loses power so that there's no notification of connection loss) in an  
inopportune moment in SSLSocket#sysread (for example, in the middle of  
reading an SSL protocol message), and SSLSocket#sysread hangs since it  
has no way of detecting the connection loss (assuming SSLSocket isn't  
using keepalives).

Anyway, this is all speculation, but I've had multiple failures over  
the past 18 months, and a perusal of the Ruby core dump shows the  
exact same pattern of a hang on select(2) called underneath in the  
implementation of rb_io_wait_readable() and similar internal functions  
(I'm not talking about a hang in select(2) as a direct result of a  
user-level code invoking IO.select()).

I'm going to try using the nonblocking SSL operations under Ruby 1.9  
to see whether it prevents my particular problem.  In general, though,  
it seems safest to always use nonblocking sockets with Ruby if you're  
going to use IO.select.

  --Young

   [1] http://linux.die.net/man/2/select