From: zdennis <zdennis@...>
Date: 2006-04-14T06:31:09+09:00
Subject: Re: Managing memory consumption

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Luke Kanies wrote:
> Hi all,
> 
> I've got a framework [1] that transfers files either locally or over
> XML::RPC.  In either case, memory consumption is significantly higher than I
> would think it should be, but in the case of XML::RPC it's so high that I
> can't actually test it on VMWare images because they don't have enough RAM
> (and I don't have enough RAM to allocate all of them enough RAM).
> 
> Is there anything I can do to force deallocation of memory when I know I'm
> done with it?
> 
> I'm confident it's not a memory leak on my part because memory usage levels
> off after the first full iteration (well, I know it levels off if I don't
> use XML::RPC, but I get failures with it so I can't test that), even if I
> recreate all of the objects from scratch on each loop (and I'm not using any
> class variables or globals or anything).
> 
> The application, in this case, supports two methods:   Send a file and
> receive a checksum in return, or send a checksum and receive a file in
> return.
> 
> Here's a basic plot of what my memory consumption looks like when copying my
> vim binary around (it's about 3.5 MB):
> 
>         Start the test method:              16124K
>         Make a copy of the file:            18604K
> 
>     In the 'send' method on the client:
>         Read the file in the 'send' method: 19388K
>         Bas64 encode the file:              27708K
> 
>     In the 'send' method on the server:
>         Base64 decode the file:             31020K
>         Read in the server-side file
>           for comparison:                   34284K
> 
> After this point, no matter how many times I run this loop, memory never
> grows or shrinks.  I've tried setting variables to "" to encourage memory
> shrinkage, but (unsurprisingly) this doesn't help.

Because ruby GC doesn't do compaction. I found out the hard way to, thanks to Guy Decoux and H. Yamamoto for pointing this out to me.

> 
> Any ideas?
> 

I had a similar question a while back on ruby-core in regards to GC and memory usage in ruby. In my case it was in regards to the
Mysql adapter, since it was never actually releasing some memory. If

Here here a comment from H. Yamamoto:

H> Because rb_gc_mark_locations marks all objects in the range of STACK_END to rb_gc_stack_start.
H> if GC.start runs inside block, block needs more stack, so more objects can be wrongly
H> marked as alive. (As you can see, last `pause` is outside of block, so less stack is used,
H> huge array goes out of stack range, it is freed)

H> And Eric Hodel is right. ([ruby-talk:181568])
H> The memory heap slot uses is greater than the memory C string itself uses.
H> Even after array is freed, heaps_slot itself still eats 26MB memory,
H> and we cannot reduce it because ruby's GC cannot do compaction.

Follow this thread for the full story: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/7442

To see some immediate reasoning view Guy Decoux's post, http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/7458
and read the comments as Guy explains what is happening.

My problem: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/7463
See Guy's reasoning: http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/7467

Hope this helps,

Zach


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFEPsRIMyx0fW1d8G0RAtQJAKCBFVwmwWiEVv8S3tfZWdYu/7hdyQCeK5a5
EKYw3YqAixr6pc6p+7/TNRg=
=M0JG
-----END PGP SIGNATURE-----