From: Eric Hodel <drbrain@...7.net>
Date: 2006-02-26T21:01:07+09:00
Subject: Re: Memory Question

On Feb 26, 2006, at 1:51 AM, zdennis wrote:

> I am doing some large queries with Mysql and the memory that gets  
> allocated never seems to go back to the system. In this test I am  
> querying 47,000 records.
>
> My setup for my test is ubuntu breezy, rails 1.0, mysql 5.0.18 and  
> a compiled mysql 2.7 ruby driver running in production mode. I have  
> wrote a helper to my program which monitors object count, memory  
> utilization and a threshold of 10% to determine how many objects  
> are sticking around and how many are being garbage collected.
>
> On first run...
>
>   Loaded suite test/unit/table1_test
>   Started
>   String count                             73685
>   Building query                           21Mb
>   String count                             73745
>
>   Received query results                   89Mb
>   String count                             901034
>   Threshold breaker String                 (827236) started w/  
> 73743 ended w/ 900979
>   Threshold breaker Table1                 47000 started w/ 0 ended  
> w/ 47000
>
>   Starting GC                              89Mb
>   String count                             25041
>   Done with GC                             82Mb
>   Threshold breaker String                 -48704 started w/ 73743  
> ended w/ 25039
>
> The first block of text shows that when I started my test ruby was  
> utilizing 21Mb of memory and over 70,000 strings were in existance.  
> After the query results were constructed there was over 900,000  
> strings in existance and my ruby process had grown to 89Mb. The  
> threshold is shown as a gain in 47,000 Table1 objects and 827,236  
> Strings since the the first object count was captured (which  
> occured right before the original String count).
>
> After garbage collecting the Stringcount is down 48,704 from when  
> it was first captured, and there is no threshold break for Table1  
> because when the program first started there were 0 in existance.  
> Thus meaning that 0 Table1 objects are in existance. However the  
> memory size never seems to leave 82Mb. It never seems to leave that  
> size.
>
> If I query again, memory goes up to 133Mb, after all Strings and  
> ActiveRecord models have been garbage collected, memory goes down  
> ever so slightly again. And this continues as long as I keep querying.

Ruby does not feature a compacting garbage collector so you may  
simply have a lot of nearly-empty pages of memory that cannot be  
returned to the operating system.

If you still suspect a leak I suggest you first see if you can  
achieve the same results by simply generating objects without any DB  
and also with just the mysql driver, no Rails.

-- 
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com