Re: Garbage collection and define_finalizer

Hi Robert,

On 30/11/11 23:51, Robert Klemme wrote:
> On Wed, Nov 30, 2011 at 4:11 AM, Garthy D
> <[e-mail]>  wrote:
>
>> Thankyou for the detailed reply. :)
>
> You're welcome!
>
>> On 30/11/11 00:07, Robert Klemme wrote:
>>>
>>> On Tue, Nov 29, 2011 at 1:43 PM, Garthy D
>>> <[e-mail]>    wrote:
>
>>> What you observe might mean that you
>>> simply haven't created enough Ruby garbage for GC to think it needs to
>>> work.  I have no idea how you allocate texture memory but if it is in
>>> a C extension written by you I would check whether there is a way to
>>> go through MRI's allocation in order to correct MRI's idea of used
>>> memory.  Implementation of Ruby's String might be a good example for
>>> that.
>>
>> Your assumption re the C extension for texture memory is pretty-much spot
>> on. :)
>
> :-)
>
>> Re texture memory, it's a little more complicated than a standard
>> allocation. Some memory will be the standard allocated sort, but some will
>> be on the video card, and they're effectively coming from different "pools"
>> (or heaps), neither of which I'll have direct control over. Unless the Ruby
>> GC directly understands this concept (I don't know if it does, but I'm
>> guessing not), I'm not going to be able to use Ruby to manage that memory.
>> Unfortunately, the problem goes a bit beyond just textures, as I'm wrapping
>> a good chunk of a 3D engine in Ruby objects. That's my problem to worry
>> about though.
>
> Sounds tricky indeed.

It certainly keeps things interesting. ;)

>> And that's assuming I can redirect the allocation calls anyway- I'm not sure
>> if I can. It'd be nice to be able to inform the Ruby GC of allocated memory
>> (or an estimate, if it keeps changing) without actually leaving the GC to
>> allocate it. Please correct me if I'm wrong, but I'm assuming this can't be
>> done, and you must use the ALLOC/ALLOC_N-style functions?
>
> I'm not such a good source for this as the source (or Matz).  So you
> probably better look elsewhere for a definitive answer.

I had a dig around the source (gc.c in particular) to see if I could 
find any answers, but I'm afraid it's beyond me at this point. I'd 
hazard a guess that the functionality doesn't yet exist, but I could 
very well be completely wrong. It's just a guess at this stage.

> Maybe for
> your issue it is sufficient to just reduce the threshold from which on
> Ruby starts considering GC at all.  I can't really tell.

On thinking about it, I really need to solve the problem "properly", 
particularly for large memory-guzzling things such as textures and 3D 
models. As an experiment I set up a means to force an object to drop its 
data ahead of time (where "drop" is defined as deleting the user data, 
but keeping the Ruby object), effectively invalidating it. For textures 
(and similar entities), the plan is to give each entity that requires 
one its own copy of a wrapped smart pointer. Thus the textures will be 
freed once every entity holding a texture object has either manually 
dropped its copy, or the GC has collected any references I've missed 
(and the plan of course is to try not to miss any references!).  I've 
got it working on simple objects at this point. I'll have to recode a 
few things to get it going for textures and models, but this is 
*probably* the solution I'll go for. I'll have to see how it turns out 
in practice.

However, once this is done for the bigger entities, there will still be 
quite a large number of small entities with minimal additional memory 
footprint that Ruby is unaware of. Reducing the threshold may be enough 
to solve the problem for the majority of these small entities- thankyou 
for the suggestion. :)

>> Unfortunately it causes a lot of problems in my case, as I had assumed one
>> thing from general reading on the topic, and observed another. That's my
>> problem to deal with though, not anyone else's. Still, there seems to be lot
>> of information online that talks about Ruby performing mark and sweep,
>> either stating outright that unreferenced objects are freed on first GC, or
>> heavily implying it at least. From your description, and my observations,
>> this information appears to be incorrect. Basically, there seems to be a lot
>> of misinformation about what the GC is doing. Apart from the source, is
>> there some place where the correct behaviour is discussed, that could be
>> referred to instead, particularly if someone is suggesting that the MRI GC
>> implementation should immediately clean up these references on next GC
>> (which, as we've established, isn't accurate)?
>
> I am not aware of any.  I haven't searched extensively though.  But we
> might need to update locations you found to reflect the truth.
> Ideally someone from core team would do a writeup which explains how
> GC works - at least for MRI.  Chances are also that behavior has
> changed between 1.8 and 1.9 as these are quite different internally.

I'll keep an eye out for such a thing if it appears.

Garth