From: Lionel Bouton Date: 2007-08-12T22:55:10+09:00 Subject: Re: Interesting garbage collection article on LTU John Joyce wrote the following on 12.08.2007 07:47 : > The OP DID mention [long running Rails processes] ! > OP Quote: > "I immediately thought of Ruby because of my experience with long running > Rails processes. It might be the occasion to have my gut feelings > checked by people that really know the inner workings of Ruby 1.8 too." > To be more accurate on my experience: I have a Rails application with the usual CRUD behaviour which isn't especially memory intensive (processes happily sit around 30-40MB, which seems the minimum for a Rails application). But there are some actions that process incoming CSV files with a rather bad memory behaviour. The process size jumps as soon as these actions are called with a size roughly proportionnal to the CSV line count. In practice this means process sizes jumping to 100-150 MB with the current usage pattern. This particular problem can be corrected through a different algorithm but this illustrates quite well what happens when Ruby must allocate lots of memory: - once the action is done, all objects can be freed (no nasty reference can lie around, this is a rather naive and simple algorithm that just allocates too much objects). - though the excess memory should be freeable the Ruby process size remains stable at the peak value. - once the total size of Ruby process approach the available memory on the system, paging occur (even if the users don't hit the CSV processing methods and so don't allocate much). This is this paging which unintuitively last though no real application memory pressure exists that I believe could be linked to the problem described in the article. In fact previously I thought about handing these memory intensive actions to BackgroundRB which I believe now uses processes killed on task completion. This means that even if the process size becomes huge, it wouldn't be more than a temporary problem, not one that might push my servers in swapping Hell. Previously I thought that the only proper solution was to recompact the memory after garbage collection (although I don't know the specifics, it might be unworkable for various reasons), but the bookmarking concept may be easier to implement. Even if it doesn't seem really right (at least for me) to have this sort of communication between the OS and the application regarding memory management, garbage collection breaks some assumptions that the VM makes so maybe it's unavoidable... Lionel.