From: mame@...
Date: 2018-05-19T14:03:48+00:00
Subject: [ruby-core:87193] [Ruby trunk Feature#14759] [PATCH] set	M_ARENA_MAX for glibc malloc

Issue #14759 has been updated by mame (Yusuke Endoh).


I tried to change Mike's script to use I/O, and I've created a script that works best with glibc with no MALLOC_ARENA_MAX specified.

    # glibc (default)
    $ time ./miniruby frag2.rb
    VmRSS:	  852648 kB
    real	0m26.191s

    # glibc with MALLOC_ARENA_MAX=2
    $ time MALLOC_ARENA_MAX=2 ./miniruby frag2.rb
    VmRSS:	 1261032 kB
    real	0m29.072s

    # jemalloc 3.6.0 (shipped with Ubuntu)
    $ time LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./miniruby frag2.rb
    VmRSS:	  966624 kB
    real	1m6.730s

    $ ./miniruby -v
    ruby 2.6.0dev (2018-05-19) [x86_64-linux]

As you see, default glibc is the fastest and most memory-efficient.
Comparing to that, glibc with MALLOC_ARENA_MAX=2 uses 1.5x memory, and jemalloc 3.6.0 is twice slow.

I'm unsure what happens.  I guess it is difficult to discuss this issue without glibc/jemalloc hackers.

```
# frag2.rb

THREAD_COUNT = (ARGV[0] || "10").to_i

File.write("tmp.txt", "x" * 1024 * 64)

Threads = []

THREAD_COUNT.times do
  Threads << Thread.new do
    a = []
    100_000.times do
      a << open("tmp.txt") {|f| f.read }
      a.shift if a.size >= 1600
    end
  end
end

Threads.each {|th| th.join }

IO.foreach("/proc/#{$$}/status") do |line|
  print line if line =~ /VmRSS/
end if RUBY_PLATFORM =~ /linux/
```

----------------------------------------
Feature #14759: [PATCH] set M_ARENA_MAX for glibc malloc
https://bugs.ruby-lang.org/issues/14759#change-72182

* Author: normalperson (Eric Wong)
* Status: Open
* Priority: Normal
* Assignee: 
* Target version: 
----------------------------------------
Not everybody benefits from jemalloc and the extra download+install
time is not always worth it.  Lets make the user experience for
glibc malloc users better, too.

Personally, I prefer using M_ARENA_MAX=1 (via MALLOC_ARENA_MAX
env) myself, but there is currently a performance penalty for
that.


gc.c (Init_GC): set M_ARENA_MAX=2 for glibc malloc

glibc malloc creates too many arenas and leads to fragmentation.
Given the existence of the GVL, clamping to two arenas seems
to be a reasonable trade-off for performance and memory usage.

Some users (including myself for several years, now) prefer only
one arena, now, so continue to respect users' wishes when
MALLOC_ARENA_MAX is set.

Thanks to Mike Perham for the reminder [ruby-core:86843]


This doesn't seem to conflict with jemalloc, so it should be safe
for all glibc-using systems.


---Files--------------------------------
0001-gc.c-Init_GC-set-M_ARENA_MAX-2-for-glibc-malloc.patch (1.46 KB)


-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>