From: nate Date: 2007-11-11T01:08:37+09:00 Subject: Re: Ruby/Fastcgi going into uninterruptible after random periods of time M. Edward (Ed) Borasky wrote: > Now *that's* bizarre -- I/O wait without disk utilization. I wonder if > you need to check your kernel memory split and high memory I/O > configuration. You may not be getting all the page cache space you can > have with an 8 GB box. It's using the default SMP kernel, no special parameters. > 5 GB occupied in an 8 GB machine? Do you mean you have 3 GB of *free* > memory? *That* would be a big problem. What kernel versions did the old > and new systems have? Well 5GB spread across all the ruby processes for that app(~20), not 1 process with 5GB. I don't believe memory usage to be an issue either, if the memory was constrained the system would swap. We have a java process that runs on the same systems that consume about 1GB of memory in a single process and it hasn't even blinked. One system I just restarted the apps on about 20 minutes ago, and within about 10 minutes one process was already in the "D" state (unusual, usually it's a few hours). Memory didn't get to 5GB after a day or so of normal traffic load, Saturdays are real low traffic. Old kernel: 2.6.17-1.2142_FC4smp (last release for fedora core 4) New kernel: 2.6.9-55.0.9.ELsmp (2nd most current release for CentOS 4.5 / Red Hat enterprise 4.0 update 5) I believe I traced the problem in mod_fcgid down, so I'm continuing the test in in the test environment with the expectation it will pass now, and try it out in production on Monday. The problem with mod_fcgid was that it seems to be ignoring a configuration parameter for timeouts, and defaults to aborting the ruby process after 40 seconds, one particular url can take longer than 40 seconds due to some bad SQL(SQL alone takes up to 30 seconds), I found in the source that the default IPC timeout is 40 seconds, which matches exactly with all of the failures in the apache logs(every one stops at 40 seconds), changed the source, recompiled and pushed out the updated version to the test servers. thanks! nate