From: ara.t.howard@...
Date: 2006-08-25T15:58:49+09:00
Subject: Re: Can Anyone Explain This Memory Leak?

On Fri, 25 Aug 2006, Zed Shaw wrote:

> Nope, I can't agree with this because the ram goes up, the OS will kill it
> eventually, and if I remove the guard the ram doesn't do this.

that makes perfect sense.  with the gaurd only one thread at a time can be
initializing the huge array at once, it takes time, because of this the number
of threads grows quite large - thus the large amount of memory consumed as
they are added the the thread group.  without the mutex the threads simply
race right through their work - creating the array and quickly exiting.

so, when the threads can be created and die quickly the maximum memory used by
the process simly never gets that big.  with the mutex the maximum memory is
larger simply because the threads take longer to run.

run this on your system:

     harp:~ > cat a.rb
     pid = Process.pid
     a = []
     new_array = lambda{|size| Array.new(size){ 42 }.map}
     eat_memory = lambda{|n| n.times{ a << new_array[4242] }}
     free_memory = lambda{a.clear and GC.start }

     eat_memory[4242]
     puts
     puts "after malloc"
     system "ps v #{ pid }"

     free_memory[]
     puts
     puts "after free"
     system "ps v #{ pid }"


     harp:~ > ruby a.rb

     after malloc
       PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
     31264 pts/10   R+     0:24      0   595 27164 26364  15.6 ruby a.rb

     after free
       PID TTY      STAT   TIME  MAJFL   TRS   DRS  RSS %MEM COMMAND
     31264 pts/10   S+     0:24      0   595 27164 26368  15.6 ruby a.rb


it shows the sort of behaviour i'm talking about

> And where are you getting your information that free doesn't free
> memory?  I'd like to read that since all my years of C coding says that
> is dead wrong.

http://groups.google.com/group/comp.unix.programmer/browse_frm/thread/44a5705312cc5df9/829197acdf6651ac?lnk=gst&q=memory+not+really+freed&rnum=4#829197acdf6651ac
http://groups.google.com/group/comp.unix.programmer/browse_frm/thread/23e7be26dd21434a/2f84b3dc080c7519?lnk=gst&q=memory+not+really+freed&rnum=9#2f84b3dc080c7519

the thing is that this has nothing to do with c and everything to do with os

   http://www.linuxjournal.com/article/6390

"When a process needs memory, some room is created by moving the upper bound of
the heap forward, using the brk() or sbrk() system calls. Because a system call
is expensive in terms of CPU usage, a better strategy is to call brk() to grab
a large chunk of memory and then split it as needed to get smaller chunks. This
is exactly what malloc() does. It aggregates a lot of smaller malloc() requests
into fewer large brk() calls. Doing so yields a significant performance
improvement. The malloc() call itself is much less expensive than brk(),
because it is a library call, not a system call. Symmetric behavior is adopted
when memory is freed by the process. Memory blocks are not immediately returned
to the system, which would require a new brk() call with a negative argument.
Instead, the C library aggregates them until a sufficiently large, contiguous
chunk can be freed at once.

For very large requests, malloc() uses the mmap() system call to find
addressable memory space. This process helps reduce the negative effects of
memory fragmentation when large blocks of memory are freed but locked by
smaller, more recently allocated blocks lying between them and the end of the
allocated space. In this case, in fact, had the block been allocated with
brk(), it would have remained unusable by the system even if the process freed
it."

> Care to tell me how malloc/free would report 80M with Mutex but properly show
> the ram go down when there is no-Mutex?

it's a sort of race

> And why Linux would kill processes if the ram get too high?  Why whole
> VPS servers crash?  I mean if this ram was just "fake" reporting (which
> is very hard to believe) then why are all these things happening?

all memory is fake.  that's why you can do this

   harp:~ > free -b
                total       used       free     shared    buffers     cached
   Mem:    1051602944 1032327168   19275776          0  145215488  563167232
                                   ^^^^^^^^

   harp:~ > ruby -e'  way_too_big = "42" * 19275776;  p way_too_big.size  '
   38551552

;-)

regarding the crashes - there are still limits whic something is apparently
exceeding.

> So please, point me at where in the specifications for malloc/free on
> Linux it says that the memory reported will be high even though free and
> malloc is called on 80M of ram slowly cycled out, and that linux will
> still kill your process even though this ram is not really owned by the
> process.

http://groups.google.com/group/comp.unix.programmer/search?group=comp.unix.programmer&q=memory+not+really+freed&qt_g=1&searchnow=Search+this+group

in any case, try running under electric fence, which will use a cleaner
malloc/free and show the 'real' behaviour.  it really does seem fine.

regards.

-a
-- 
to foster inner awareness, introspection, and reasoning is more efficient than
meditation and prayer.
- h.h. the 14th dalai lama