From: Calamitas <calamitates@...>
Date: 2010-10-30T02:11:10+09:00
Subject: Re: Ruby 1.9.1 linear memory increase in a simple for loop.

On Fri, Oct 29, 2010 at 3:17 PM, Ritwik Banerjee
<ritwik.banerjee@gmail.com> wrote:
> The following text code shows a linear rise in memory usage. Since this
> is a simple loop, I am guessing there's something fundamentally wrong
> (either in my understanding of ruby's garbage collection, or in ruby's
> garbage collection):
>
> memtest.rb
> ----------
> require 'matrix'
> a = [[1,0,1,0,0,1,1,1],[0, 0, 0, 0, 0, 1, 1, 0],[1, 1, 1, 1, 0, 1, 1,
> 1],[0, 0, 1, 1, 1, 1, 0, 1],[0, 0, 1, 1, 1, 1, 0, 1],[0, 0, 0, 1, 1, 1,
> 1, 1],[0, 1, 1, 1, 1, 1, 1, 1],[0, 1, 0, 1, 1, 0, 1, 1]]
>
> m = Matrix.rows(a)
> n = Matrix.identity(8)
>
> for i in 1..100 do
>    n = n * m;
>    physical_mem_usage = `ps -o rss= -p #{Process.pid}`.to_i;
>    virtual_mem_usage = `ps -o vsz= -p #{Process.pid}`.to_i;
>    puts "#{i}: VMU = #{virtual_mem}; PMU = #{physical_mem}."
>    puts ObjectSpace.count_objects()
> #   ObjectSpace.garbage_collect
>    STDOUT.flush
> end
>
> ---------
>
> This code shows a linear increase in memory usage. The
> ObjectSpace.count_objects call shows a steady linear increase in counts
> of T_ARRAY, T_HASH, T_STRUCT, T_STRING and T_OBJECT.
>
> Can anyone explain this? Typically memory explosions occur in complex
> codes with lots of references, and in those cases the memory rise is
> exponential. In the above loop, invoking garbage collection manually
> (it's commented out in the snippet above) slows the memory increase, but
> only by a constant factor.
> For example, if the loop used up 30 kB in 100 iterations originally, it
> uses up 30kB in 500 iterations after invoking garbage collection
> manually.
>
> Understanding this is critical for me since I am dealing with matrices
> of enormous sizes, so even a small memory leak can jeopardize my
> project.

You are calculating n=m**i (m to the i-th power) in subsequent
iterations. The entries in n grow exponentially with rising i. Storage
size of integers is logarithmic (in the Bignum range), meaning that
storage size for n increases linearly. That's why you see memory
increase linearly.

Peter