From: "M. Edward (Ed) Borasky" Date: 2008-07-03T12:37:53+09:00 Subject: Re: Ruby 1.9.0/1.8.7/1.8.6/1.8.5 new releases (Security Fix) Igal Koshevoy wrote: > Doug Alcorn wrote: >> Igal Koshevoy wrote: >> >> >>> Can you or one of your engineers try to figure out how to trigger this >>> memory ballooning problem? That'd be really valuable to the Ruby >>> community >> >> Not only to the community, but also my client. I've been working on >> it on and off for several weeks. I'm not sure I have the right skills >> atm to do memory profiling. I'd be interested in some good pointers >> on how to memory profile ruby/rails apps >> > Your findings may be very important. Most people are running versions of > Ruby with patches backported to p111/p114, whereas you're running a > patch against p230. Relatively few people are using that newer release, > but its code is much closer to what the Ruby maintainers plan to ship in > the near future. Because you're closer to the edge, you've likely hit > something the rest of us haven't. > > As for identifying the problem, unless someone has a better suggestion, > I think you just have to use each part of your application until you > identify an action that triggers this bug. You should clone your > production environment as closely as possible, including the database > and run the app in production mode, and use that for your testing. You > should also keep a written list of all controllers, actions and logical > paths within actions, and then check these off as you've validated them > so you can be systematic about checking everything. After each action, > glance at the application's memory consumption to see if you caught the > problem. Once you know what sequence of user actions will cause it to > balloon, you can start narrowing down the code causing it by commenting > stuff out and such. > > A good way to keep an eye on memory consumption is to start up a > terminal and run a command like this in it: > watch -n1 'ps aux | egrep "ruby|PID" | egrep -v egrep' > > The above scans the output of "ps" each second and displays all the > processes matching the "ruby". You may want to substitute > "mongrel_rails" or whatever the name is for your server process, and > then just watch the RSS and VSZ columns. > > Another possible way to identify the problem may be NewRelic's RPM > (http://newrelic.com/). Be sure to get the "Production Mode" with the 2 > month free trial and enable the feature which uploads reporting data > back to them. Among other useful features, this Rails plugin will help > you quickly identify the actions that are taking a long time to > complete. If lucky, this will help you identify the actions causing > leakage; and if less lucky, you'll know where to prioritize your tuning > efforts later, which is still useful. > > Anyone else have ideas on how to identify the problem? > > -igal > > "watch" is for wimps ... just about every Linux system has "top". :) What's *really* nice about "top" is that you can do "top -b > logfile" and it will do a "top" at the default interval to the log file. Then you can parse that log file with Ruby and watch your process leak memory. I'm just about to set up some shell scripts (sorry, all you Rake fans -- my Rakefiles are too ugly so I'm dropping back to bash :) ) to build a profiling Ruby. I might just take the opportunity to learn how to use "valgrind" while I'm at it.