From: Igal Koshevoy <igal@...>
Date: 2008-07-03T01:52:05+09:00
Subject: Re: Ruby 1.9.0/1.8.7/1.8.6/1.8.5 new releases (Security Fix)

Doug Alcorn wrote:
> Igal Koshevoy wrote:
>
>   
>> Can you or one of your engineers try to figure out how to trigger this
>> memory ballooning problem? That'd be really valuable to the Ruby 
>> community
>
> Not only to the community, but also my client.  I've been working on it 
> on and off for several weeks.  I'm not sure I have the right skills atm 
> to do memory profiling.  I'd be interested in some good pointers on how 
> to memory profile ruby/rails apps
>   
Your findings may be very important. Most people are running versions of 
Ruby with patches backported to p111/p114, whereas you're running a 
patch against p230. Relatively few people are using that newer release, 
but its code is much closer to what the Ruby maintainers plan to ship in 
the near future. Because you're closer to the edge, you've likely hit 
something the rest of us haven't.

As for identifying the problem, unless someone has a better suggestion, 
I think you just have to use each part of your application until you 
identify an action that triggers this bug. You should clone your 
production environment as closely as possible, including the database 
and run the app in production mode, and use that for your testing. You 
should also keep a written list of all controllers, actions and logical 
paths within actions, and then check these off as you've validated them 
so you can be systematic about checking everything. After each action, 
glance at the application's memory consumption to see if you caught the 
problem. Once you know what sequence of user actions will cause it to 
balloon, you can start narrowing down the code causing it by commenting 
stuff out and such.

A good way to keep an eye on memory consumption is to start up a 
terminal and run a command like this in it:
    watch -n1 'ps aux | egrep "ruby|PID" | egrep -v egrep'

The above scans the output of "ps" each second and displays all the 
processes matching the "ruby". You may want to substitute 
"mongrel_rails" or whatever the name is for your server process, and 
then just watch the RSS and VSZ columns.

Another possible way to identify the problem may be NewRelic's RPM 
(http://newrelic.com/). Be sure to get the "Production Mode" with the 2 
month free trial and enable the feature which uploads reporting data 
back to them. Among other useful features, this Rails plugin will help 
you quickly identify the actions that are taking a long time to 
complete. If lucky, this will help you identify the actions causing 
leakage; and if less lucky, you'll know where to prioritize your tuning 
efforts later, which is still useful.

Anyone else have ideas on how to identify the problem?

-igal