From: Matthew Bloch <mattbee@...>
Date: 2002-07-17T19:31:34+09:00
Subject: Patch to stop TCPSocket.new blocking on DNS lookups

Hello;

I'm using Ruby for a multi-threaded (50 or 60 threads at once) web spider and 
found that every time I started a fetch in a thread, the whole process got 
blocked on getaddrinfo, which of course knackers the throughput since all the 
already-connected threads have to wait for a single DNS lookup to happen.

So after a bit of experimenting I've made a patch which calls getaddrinfo_a 
(a GNU extension) and lets the other threads run while a lookup is going on. 
I'll post it today after I've tested it on my spider.  I'm a bit uneasy 
about the implementation though:

in ext/socket/socket.c

*) changed getaddrinfo calls to ruby_nonblock_getaddrinfo;
*) implmeneted ruby_nonblock_getaddrinfo:
      sets up a call to getaddrinfo_a for the query and asks to be notified
        by SIGUSR2;
      while (request is in progress) calls rb_thread_wait_signal("USR2")
      returns with result

in eval.c:

*) added WAIT_SIGNAL constant and a 'char* signal' field in struct thread to 
     indicate which signal a thread is waiting for;
*) added static wakeup_threads_sleeping_on_signal which is called from 
     rb_signal_raise every time a signal comes in to wake any threads that
     are interested in that signal;
*) implemented ruby_thread_wait_signal: just sets the thread status to stopped
     and calls rb_thread_schedule().
   
Okay, for one being notified by a signal makes me a little nervous; I'm not 
sure there aren't race conditions in waiting for the signal until a request 
is done.  Also, fencing off a hard-coded signal for our use like that is not 
very flexible when there are other calls in socket.c which could benefit from 
being made asychronous in the same manner.  And of course there's the fact 
that there's no Win32 implementation (or does Win32 sockets do asychronous 
DNS already?).

getaddrinfo_a can also notify us by calling a function in another kernel 
thread; would this be preferable? 

Any ideas appreciated.

cheers,

-- 
Matthew Bloch         Bytemark Computer Consulting Limited
                                http://www.bytemark.co.uk/
                                  tel. +44 (0) 8707 455026