From: E F van de Laar <emiel@...>
Date: 2003-02-26T22:50:47+09:00
Subject: Re: Ruby in Performance Testing

* Lennon Day-Reynolds (lennon@day-reynolds.com) wrote:
> On Wed, 26 Feb 2003 17:27:17 +0900
> E F van de Laar <emiel@il.fontys.nl> wrote:
> 
> > Rubyists,
> > 
> > I'm trying to setup a performance/load test to bombard a server with
> > a ton of requests.  The idea is to generate as many parallel requests
> > as possible to really stress the server. However my code already bombs
> > out at around 300 threads with:
> > 
> > naft_s.rb:140: [BUG] rb_sys_fail() - errno == 0
> > ruby 1.7.3 (2002-11-17) [i386-mswin32]
> 
> Is this the error message from the server, or the client? From the high 
> source line number, I can only assume the error is showing up on the 
> server, in which case it's difficult to determine exactly what might be 
> causing the problem without the source. However, I'll assume that the
> basic issue is symmetrical -- that is, both the client and the server 
> begin to show the same symtoms after a high number of concurrent 
> connections are open.

Error occurs in the client side. The server is a java implementation
that I have very little control over. It does use a pool of native
threads however.
 
> > I also get "Bad file descriptor" errors at times.
> > 
> > Does anyone have any other approaches for this kind of test? Is ruby
> > not up to this sort of task? Would fork be a better approach?
> > How do I raise this limit? Any tips are welcome.
> 
> I'm not sure how one would fix the problem under Windows, but I know
> that under most Linux and BSD systems, there's a kernel-controlled
> limit on the number of file descriptors a single process can have open at
> any one time. Either a kernel recompile or a simple configuration option
> usually fixes this on the open source OSes, but obviously, under Windows,
> you probably won't have that option.

I'll move to a UNIX machine and explore some of theses limits. Your
link has opened a new world for me and I'm already understanding
my problem a lot better.
 
> > [code deleted]
> 
> If the problem is indeed a hard limit to the number of file descriptors
> one Windows process can have open, then you'll need to fork several
> 'slave' processes, each of which handles a number of connections 
> lower than this limit. However, you definitely don't want to fork a new
> process for each connection, as the 300+ simultaneous execution
> requests would almost certainly bring the system to its knees for a
> while, if not thrash the disk enough swapping to seriously risk your
> filesystem integrity.

I'm going to try this soon. My parallel programming skills are just
now starting to develop. :)
 
> In general, though, I highly recommend the articles linked at the bottom
> of the following page for general high-load server tuning ideas:
> 
> http://www.nightmare.com/medusa/index.html
> 
> The author of the above page also wrote the Medusa asynchronous 
> networking library for Python, but there are a number of excellent 
> articles about building servers capable of handling rediculous numbers 
> (i.e., thousands or tens of thousands) of concurrent connections. Some
> of the content is a bit dated, but the basic techniques are all very sound.
> (The short summary, for the impatient: the 'select()' method is your 
> friend and helper, esp. with continuations and/or lightweight threading.)

Thanks for the info.

Emiel
-- 
E F van de Laar
+31648183479
www.il.fontys.nl/~emiel