Re: GC Question

ts wrote:
>>>>>>"z" == zdennis  <[e-mail]> writes:
> 
> 
> z> Thank you Guy for posting this with your comments. So this is a bug I am taking it, is this 
> z> something that can be fixed rather easily or is this problem more difficult then that?
> 
>  Well this is not really a bug, this is how work a conservative GC which
>  can mark object, because it find a reference on the stack. even if these
>  objects are potentially dead (i.e. never referenced by other objects).

 From some tests, it appears that ruby will taper off memory usage and reuse unused slots in the 
heaps. My big worry was that I need to do processing on millions of records in a database. I didn't 
want to process the data in chunks of 50,000, if each time I did a new query for a new set of 50,000 
ruby wasn't going to GC well-enough to where I ran out of memory. It appears this will taper off at 
a certain point.

Here are my results from querying 1,000 new records in iteration:

   iteration: 0
   Calling GC.start
   String
     count = 63,455
   Array
     count = 516
   Mem usage: 10Mb
   -----------
   iteration: 1
   Calling GC.start
   String
     count = 63,295
   Array
     count = 516
   Mem usage: 11Mb
   -----------
   iteration: 2
   Calling GC.start
   String
     count = 63,332
   Array
     count = 516
   Mem usage: 11Mb
   -----------
   iteration: 3
   Calling GC.start
   String
     count = 63,121
   Array
     count = 516
   Mem usage: 11Mb

It tapers off at 11Mb. Now moving up to querying a new set of 50,000 records for each iteration:

   iteration: 0
   Calling GC.start
   String
     count = 2,708,846
   Array
     count = 531
   Mem usage: 177Mb
   -----------
   iteration: 1
   Calling GC.start
   String
     count = 2,657,524
   Array
     count = 531
   Mem usage: 189Mb
   -----------
   iteration: 2
   Calling GC.start
   String
     count = 2,631,215
   Array
     count = 531
   Mem usage: 193Mb
   -----------
   iteration: 3
   Calling GC.start
   String
     count = 2,656,035
   Array
     count = 531
   Mem usage: 189Mb
   -----------
   Calling GC.start
   String
     count = 9,794
   Array
     count = 514
   Mem usage: 189Mb

It tapers off at 189Mb, which I can live with. I was afraid it would jump 189Mb to 300Mb to 500Mb, 
and so on.


> 
>  Like I've said I think that the Boehm GC has some way to "clean" a stack
>  when it's unused, just for this reason (to don't retrieve old reference to
>  an object) but I don't think that :
> 
>   1) it can solve *all* these problems
> 
>   2) it's easy to do it
> 
> 
>  Some time you must just try to adapt you to the ruby GC, which generally do a
>  really good job.
> 

I am trying to adapt. I think I need to become more familiar with Garbage Collection. It would be 
very useful if ruby GC's was compacting.

Thank you for your reply,

Zach