From: Mathieu Bouchard <matju@...>
Date: 2002-01-16T08:51:33+09:00
Subject: Re: memory usage question


On Wed, 16 Jan 2002, Eric Lee Green wrote:

> >   h = {}; s = 'aaaaaaaaaaaaaaaa'
> >   for i in 1..4_000_000
> >     h[i] = (s = s.succ)
> >   end
> > This ran relatively fast, but it consumed 265 megs of windows memory and
> Uhm. Nobody has answered this question. Let's look:
> 64mb of string data
> 12mb of string length data (since strings have a length)
> 12mb of hash data
> 12mb of hash pointer data

let me revise this:

64mb of strings
32mb of string malloc frame
40mb of string objects
64mb of hashnodes
32mb of hashnode malloc frame
32mb for the hash table itself
 1mb for ruby

= 265mb total

verified by reading .h's and doing tests on malloc().
however running the real program gives me a projected size of ~296mb
(don't have enough ram to run the real test), so I don't know what's the
missing part. (265 flat was too good to be true.) (malloc frame depends on
the version of libc.so)

> I am running the test here on my Linux system with 192mb of RAM.  My
> RSS usage gets to around 128mb, then the garbage collector kicks in
> and kicks me down to around 95mb. 

RSS does not include swapped-out memory. The SIZE field is more accurate
for that scale. (On a small scale, RSS is more accurate because swapping
does not occur and shared libraries make the program look bigger)

> Note that Ruby simply accumulates temporary strings and other such
> cruft until its heap allocator hits a "highwater" mark,

there are several highwater marks; the first one that is reached will
trigger a GC.

> if my experience here holds that it invokes the GC at around 2/3rds of
> physical RAM).

There is not such highwater mark. Ruby invokes the GC whenever there are
not enough unused-but-preallocated objects. after such a GC, if that count
is still too low, then a new object-table gets allocated. In Ruby 1.6, a
fixed number is allocated every time. In Ruby 1.7 (or 1.6 with my patch), 
a big process will grow by bigger chunks, for scalability.

For curing the process size: malloc frames may be reduced using a custom
malloc() function; string objects can be eliminated by writing a custom
hash class and merging the 4_000_000 strings into one big string, such
that you can index by strings that are not standalone objects. I bet you
can get the whole down (SIZE) to ~130mb total. If you can make your hash
implementation really tight (by not using hashnodes) you may get down to
100mb *while also* making the custom-malloc irrelevant (since there
wouldn't be any mallocs anymore).

________________________________________________________________
Mathieu Bouchard                   http://hostname.2y.net/~matju