From: Eric Wong <normalperson@...>
Date: 2011-07-29T13:09:03+09:00
Subject: Re: How to overcome network IO block (lost network) ?

Peter Fitzgibbons <peter.fitzgibbons@gmail.com> wrote:
> I would like to know of any suggestions you have on working around lost
> network connection on an ActiveRecord object.
> The AR Connection is open, then network is lost, and the desired effect is
> short (5sec) timeout on the subsequent request.

It depends /heavily/ on the underlying C libraries for you're using
(both the Ruby C extension and client library).  Things like mysql2 and
pg should be pretty good with interruptability/network failures.

> I have not found a secret sauce on MRI (1.8.7 or 1.9.2) that will perform an
> timeout when the thread (main thread) is blocked on network IO.
> 
> Example code is here : https://gist.github.com/gists/1111340
> 
> Any advice is greatly appreciated.

To have half a chance of working with the standard timeout.rb library,
it should use TRAP_BEG/TRAP_END under MRI 1.8 and
rb_thread_blocking_region() with 1.9 if the underlying DB library relies
on blocking I/O (like mysql2), or
rb_thread_select()/rb_thread_fd_select() if non-blocking I/O is possible
(pg).

I needed to look recently at the "tiny_tds" gem you seem to be using for
an unrelated bug, but tiny_tds didn''t seem to be using either of the
things mysql2/pg used to be interruptable/thread-non-blocking.

As a last resorts (if you're unwilling/unable to hack on tiny_tds),
terminator[1] should be effective, as is the Unicorn timeout
mechanism[2] as they both use SIGKILL to nuke the process (Unicorn will
restart the killed process).  But tiny_tds should really be fixable by
its author(s).

[1] http://www.rubyinside.com/terminator-hardcore-timeout-for-ruby-code-1189.html
[2] http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout
    (disclaimer: I'm the BDFL of Unicorn)

-- 
Eric Wong