[ruby-core:47096] [ruby-trunk - Bug #5437][Rejected] Using fibers leads to huge memory leak

Issue #5437 has been updated by drbrain (Eric Hodel).

Status changed from Assigned to Rejected

=begin
Your test does not illustrate any leak and is likely not capable of demonstrating a leak in a diagnosable way.

You seem to have some misconceptions about how malloc() and free() work and how ruby's GC works.

Finalizers are not invoked immediately after object collection.  They are invoked at some point in the future so more Fiber objects may be collected than indicated by your counter.

Calling GC.start is not guaranteed to collect all objects without visible references.  Ruby's GC is conservative and walks the stack looking for possible references to objects, even if the stack value only coincidentally points to an object.

Ruby calls malloc() to allocate memory. If malloc() does not have any free memory available to assign to ruby it will ask for more from the kernel via mmap() (and sometimes sbrk(), but less frequently on modern OSs).  When ruby is done with memory it wall call free().  This will tell malloc() the memory is no longer being used and is available for a future call to malloc().  Sometimes free() will result in returning memory to the OS, but not always since that reduces performance.  (malloc() may immediately need to ask the OS for more memory.)

You should not always expect to see a reduction of the resident size of a process when Ruby's GC runs, especially on a very small script that runs for a very short period.

To demonstrate a leak you need to demonstrate process growth in a long-running program.  Running for a few seconds and checking the size of ((%ps%)) is insufficient.

I made a better test for Fiber leaks:

  require 'fiber'
  
  loop do
    fibers = 10.times.map do
      n = Fiber.new do
        f = Fiber.current
        Fiber.yield
      end
      n
    end; nil
  
    fibers.each(&:resume); nil
    fibers = []
  end

This allocates a small number of fibers and runs continuously.  If Ruby is leaking fibers we should see a small but steady growth in process size over time.  We should also be able to run tools on the process like OS X's leaks(1) to detect the leak.

I ran my test for over 25 minutes.  The memory size reached 39.6MB and remained constant for the duration of the test.

leaks(1) reported:

  Process:         ruby20 [96158]
  Path:            /Users/drbrain/Work/svn/ruby/trunk/ruby20
  Load Address:    0x104616000
  Identifier:      ruby20
  Version:         ??? (???)
  Code Type:       X86-64 (Native)
  Parent Process:  make [96143]
  
  Date/Time:       2012-08-09 17:18:55.562 -0700
  OS Version:      Mac OS X 10.8 (12A269)
  Report Version:  7
  
  leaks Report Version:  2.0
  Process 96158: 16276 nodes malloced for 8215 KB
  Process 96158: 0 leaks for 0 total leaked bytes.

As you can see there is no leak detected.  Also, note that only about 20% of the resident memory capacity is part of an active allocation.

Due to these results I am reasonably certain that no leak has occurred.

We will reopen if you can demonstrate a leak in a long-running process.

=end
----------------------------------------
Bug #5437: Using fibers leads to huge memory leak
https://bugs.ruby-lang.org/issues/5437#change-28749

Author: rupert (Robert Pankowecki)
Status: Rejected
Priority: Normal
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 
ruby -v: ruby 1.9.3dev (2011-10-11 revision 33457) [x86_64-linux]


It appears to me that there is something wrong with reallocating (reusing?) memory used by fibers. Here is a little script:

<code>
require 'fiber'

fibers = 10_000.times.map do
  Fiber.new do
    f = Fiber.current
    Fiber.yield
  end
end; nil

fibers.each(&:resume); nil
fibers.each(&:resume); nil
fibers = []

GC.start
</code>

Running this code in IRB multiple times leads to consuming more and more memory every time. However every time I execute this code the amount of newly consumed memory is lower. I was able to repeat this bug with ruby 1.9.2.p290 and 1.9.3-head using 32 and 64 bit architecture.

Here are some memory stats:
a) 32bit
Iter | VSZ | RSS
1 186268 63508
2 360196 119580
3 360340 147600
4 360484 178552
5 360616 199968
6 360612 210716
7 360612 221920
8 360612 224544
9 360612 229976

b) 64bit
1 395152 107600
2 739924 176532
3 739924 187972
4 739924 203300
5 739924 217716
6 739924 218344
7 739924 218540
8 739924 234040
9 739924 234256

We can see on our production instance that the more Fibers are in use the more memory is never reclaimed back to OS and the bigger the leak is.


-- 
http://bugs.ruby-lang.org/