From: "M. Edward (Ed) Borasky" Date: 2007-11-11T04:37:01+09:00 Subject: Re: Ruby/Fastcgi going into uninterruptible after random periods of time nate wrote: > M. Edward (Ed) Borasky wrote: > >> Now *that's* bizarre -- I/O wait without disk utilization. I wonder if >> you need to check your kernel memory split and high memory I/O >> configuration. You may not be getting all the page cache space you can >> have with an 8 GB box. > > It's using the default SMP kernel, no special parameters. > >> 5 GB occupied in an 8 GB machine? Do you mean you have 3 GB of *free* >> memory? *That* would be a big problem. What kernel versions did the old >> and new systems have? > > Well 5GB spread across all the ruby processes for that app(~20), not > 1 process with 5GB. I don't believe memory usage to be an issue > either, if the memory was constrained the system would swap. We have > a java process that runs on the same systems that consume about 1GB > of memory in a single process and it hasn't even blinked. Sorry ... I wasn't clear. *Free* memory in large quantities is a problem, not the fact that you've got 5 GB full out of 8 GB. Linux uses memory for lots of things, but the two biggest are application programs (code and data space) and page cache (where stuff read from and written to disk) goes. It tries as hard as it can to fill all available memory with those two things and balance the applications' demand between the two of them. The way you figure this out is to look at the first few lines of "top". You'll see something like this (from a 1 GB machine): Mem: 901256k total, 890708k used, 10548k free, 178360k buffers Swap: 996020k total, 0k used, 996020k free, 341112k cached Ignore the swap -- my machine isn't swapping and you say yours isn't either. "used" + "free" should equal "total". "used" - "buffers" - "cached" is what's taken up by the code and data. So on my machine: 10548 free 178360 buffers 341112 cached 371236 used - buffers - cached = code + data ------ 901256 Since you're not swapping, any free memory you have above the minimum set by the kernel *should* be going into "buffers" and "cached" to reduce the amount of I/O you need to do. > Old kernel: 2.6.17-1.2142_FC4smp (last release for fedora core 4) > New kernel: 2.6.9-55.0.9.ELsmp (2nd most current release for CentOS 4.5 / > Red Hat enterprise 4.0 update 5) Well, when you get all the other things humming, an upgrade to CentOS 5 will give you a newer kernel, Ruby, Apache, MySQL and PostgreSQL. But there have been lots of "enhancements" between 4.5 and 5.0 in the area of security -- I found the learning curve for all the firewall stuff rather steep. It's worth it, of course, but it wasn't a bullet I had the energy to bite at the time. > I believe I traced the problem in mod_fcgid down, so I'm continuing the > test in in the test environment with the expectation it will pass now, > and try it out in production on Monday. The problem with mod_fcgid was > that it seems to be ignoring a configuration parameter for timeouts, > and defaults to aborting the ruby process after 40 seconds, one particular > url can take longer than 40 seconds due to some bad SQL(SQL alone takes > up to 30 seconds), I found in the source that the default IPC timeout > is 40 seconds, which matches exactly with all of the failures in the > apache logs(every one stops at 40 seconds), changed the source, recompiled > and pushed out the updated version to the test servers. Interesting ... you actually fixed something in software rather than throw hardware at it. Most folks would have moved the database to something that could complete the query in 3 seconds and sent the bill to their clients. ;)