From: Rick Nooner Date: 2005-09-30T13:38:39+09:00 Subject: Re: Threading Performance? On Fri, Sep 30, 2005 at 12:20:07PM +0900, Jeff McNeil wrote: > Greetings. > > I apologize if this has been answered somewhere obvious. I did a fair > bit of Googling prior to piping up. If this has been addressed, > please feel free to simply point and grunt! > > In a nutshell, I have concerns surrounding the Ruby threads > implementation as it appears to be a home-grown system. How does > would performance stack up against FreeBSD 5.x KSD threads? I've run > into problems with the Python dummy_thread module before with respect > to performance, especially when dealing with intensive IO. I have a > bit of a fear that I'll have the same issue here. > > I've been tasked with writing a dynamic HTTPS/HTTP gateway of sorts, > and as such, it ought to get quite busy in the network IO > department. I'd like to take advantage of thread pooling and whatnot. > > I'd really love to use Ruby as I'm quickly falling in love with it - > what a nice language. > > Thoughts? > > Jeff I wrote a data collection system that gathers statistics from over 2000 servers every five minutes 24/7. This has both high network usage patterns as well as high disk usage patterns. I also had a 4 processor box (a Sun E4500). Ruby threads cannot take advantage of a multiprocessor server. In order to have each collection cycle finish within the 5 minute window alloted, all 4 processors must be fully utilized. The architecture that I settled on was similar to a threaded worker pool, except instead of threads I used processes with the main process acting as the scheduler and the child processes reading work tasks from a distributed queue (Ruby makes this easy with Rinda). This allows scaling both by adding more processors (and processes) to the server OR because of Rinda (and dRB) simply by adding more collection servers. So far, this solution has been running over a year and a half on the single 4 processor server with no unscheduled down time. My take is don't use the Ruby threading model for network intensive tasks. Rather think about using dRB and process level parallelism. You might be suprised at how well it works and how scalable this makes your system. Rick -- Rick Nooner rick@nooner.net http://www.nooner.net