Re: memory usage question

In article <[e-mail]>, Mark Hahn wrote:
> I need to write a script that will use a hash with 4 million strings of 16
> chars each.  As a test I wrote this quick script:
> 
>   h = {}; s = 'aaaaaaaaaaaaaaaa'
>   for i in 1..4_000_000
>     h[i] = (s = s.succ)
>   end
> 
> This ran relatively fast, but it consumed 265 megs of windows memory and
> failed with out of memory error at around 3 million elements.
> 
> Why did Ruby fail at 265 megs and is there another way for me to do this?

Uhm. Nobody has answered this question. Let's look:

64mb of string data
12mb of string length data (since strings have a length)
12mb of hash data
12mb of hash pointer data

We're still at only 100 megabytes of data here. Even if Ruby overhead
was 15%, that'd be only 150 megabytes of data. 

I am running the test here on my Linux system with 192mb of RAM.  My
RSS usage gets to around 128mb, then the garbage collector kicks in
and kicks me down to around 95mb. 

Note that Ruby simply accumulates temporary strings and other such
cruft until its heap allocator hits a "highwater" mark, at which point
it invokes the garbage collector to get rid of the cruft.  It occurs
to me that Windows is refusing to allocate more memory to your process
prior to when Ruby invokes the garbage collector (which would be at
around the 2.8gb mark, if my experience here holds that it invokes the
GC at around 2/3rds of physical RAM). I.e., that you may be running
into a Windows limitation setting your max per-process memory to 256
megabytes, not a Ruby problem.

-- 
Eric Lee Green       [e-mail]    http://www.badtux.org
    GnuPG public key at http://badtux.org/eric/eric.gpg
            BadTux: Linux Penguin Gone Bad