From: nate <ruby@...>
Date: 2007-11-11T08:46:24+09:00
Subject: Re: Ruby/Fastcgi going into uninterruptible after random periods                of      time

M. Edward (Ed) Borasky wrote:


> Since you're not swapping, any free memory you have above the minimum
> set by the kernel *should* be going into "buffers" and "cached" to
> reduce the amount of I/O you need to do.

Yep I understand all that, to illustrate, one of the server's memory
usage -

http://portal.aphroland.org/~aphro/memory-usage.png

> Well, when you get all the other things humming, an upgrade to CentOS 5
> will give you a newer kernel, Ruby, Apache, MySQL and PostgreSQL. But
> there have been lots of "enhancements" between 4.5 and 5.0 in the area
> of security -- I found the learning curve for all the firewall stuff
> rather steep. It's worth it, of course, but it wasn't a bullet I had the
> energy to bite at the time.

Fortunately the firewall isn't something I'll have to deal with, these
machines are part of a larger network behind shared network infrastructure,
firewalls, load balancers, etc. So no firewalls on the local systems are
needed.


> Interesting ... you actually fixed something in software rather than
> throw hardware at it. Most folks would have moved the database to
> something that could complete the query in 3 seconds and sent the bill
> to their clients. ;)

HAH, yeah my last job was like that to some degree, a phrase the
VP of engineering used to say on outage conference calls "is there
anything I can buy that would make this problem go away". It was a
good place to play with new stuff, always buying.. The servers would
average about 1.5 transactions a second before falling over, their
main OLTP database is probably in the 50TB range now(orders of
magnitude larger than any other OLTP database in the world). Pretty
poor software..led to lots of problems..

Fortunately the developers at my current place are much more,
how should I say..customer oriented(with me being the customer I'd
say), makes my job easier..

And that bad query, is supposed to be fixed soon by adding another
kind of index, there is already in index on the column but the query
forces the search to be all lower case, so the DB ends up doing a
full scan across about 4.5 million rows(oracle 10g R2)

nate