[ruby-core:95033] [Ruby master Misc#16124] Let the transient heap belong to objspace

Issue #16124 has been updated by methodmissing (Lourens Naud=E9).


nobu (Nobuyoshi Nakada) wrote:
> I'm positive about this, except for the performance.
> Do you have any numbers?

Using the rdoc gc bench tooling Koichi used in https://bugs.ruby-lang.org/i=
ssues/14858, it's difficult to get consistent results between this change a=
nd master.

Master:

```
lourens@CarbonX1:~/src/ruby/trunk$ make gcbench-rdoc
Script: ./benchmark/gc/rdoc.rb
{:count=3D>137,
 :heap_allocated_pages=3D>10124,
 :heap_sorted_length=3D>10124,
 :heap_allocatable_pages=3D>0,
 :heap_available_slots=3D>4126494,
 :heap_live_slots=3D>4117009,
 :heap_free_slots=3D>9485,
 :heap_final_slots=3D>0,
 :heap_marked_slots=3D>2396930,
 :heap_eden_pages=3D>10124,
 :heap_tomb_pages=3D>0,
 :total_allocated_pages=3D>10124,
 :total_freed_pages=3D>0,
 :total_allocated_objects=3D>24630985,
 :total_freed_objects=3D>20513976,
 :malloc_increase_bytes=3D>23046392,
 :malloc_increase_bytes_limit=3D>32225676,
 :minor_gc_count=3D>113,
 :object_id_collisions=3D>0,
 :major_gc_count=3D>24,
 :remembered_wb_unprotected_objects=3D>2482,
 :remembered_wb_unprotected_objects_limit=3D>4964,
 :old_objects=3D>2387069,
 :old_objects_limit=3D>4774138,
 :oldmalloc_increase_bytes=3D>48011656,
 :oldmalloc_increase_bytes_limit=3D>41096563}
ruby 2.7.0dev (2019-09-22T07:39:47Z master 2272efa463) [x86_64-linux] ["USE=
_RGENGC", "RGENGC_DEBUG", "RGENGC_ESTIMATE_OLDMALLOC", "GC_ENABLE_LAZY_SWEE=
P"]

./benchmark/gc/rdoc.rb
      user     system      total        real
 18.842867   0.687732  19.530599 ( 19.722540)
GC total time (sec): 0

VmHWM: 411940 kB

Summary of rdoc on 2.7.0dev	19.722539804002736	0	137
         (real time in sec, GC time in sec, GC count)
```

Objspace linked theap:

```
lourens@CarbonX1:~/src/ruby/ruby$ make gcbench-rdoc
Script: ./benchmark/gc/rdoc.rb
{:count=3D>137,
 :heap_allocated_pages=3D>10122,
 :heap_sorted_length=3D>10122,
 :heap_allocatable_pages=3D>0,
 :heap_available_slots=3D>4125809,
 :heap_live_slots=3D>4116170,
 :heap_free_slots=3D>9639,
 :heap_final_slots=3D>0,
 :heap_marked_slots=3D>2396677,
 :heap_eden_pages=3D>10122,
 :heap_tomb_pages=3D>0,
 :total_allocated_pages=3D>10122,
 :total_freed_pages=3D>0,
 :total_allocated_objects=3D>24630955,
 :total_freed_objects=3D>20514785,
 :malloc_increase_bytes=3D>23055352,
 :malloc_increase_bytes_limit=3D>32225676,
 :minor_gc_count=3D>113,
 :object_id_collisions=3D>0,
 :major_gc_count=3D>24,
 :remembered_wb_unprotected_objects=3D>2482,
 :remembered_wb_unprotected_objects_limit=3D>4964,
 :old_objects=3D>2387070,
 :old_objects_limit=3D>4774140,
 :oldmalloc_increase_bytes=3D>47981528,
 :oldmalloc_increase_bytes_limit=3D>41096563}
ruby 2.7.0dev (2019-09-22T11:43:20Z objspace-theap 11b7839f09) [x86_64-linu=
x] ["USE_RGENGC", "RGENGC_DEBUG", "RGENGC_ESTIMATE_OLDMALLOC", "GC_ENABLE_L=
AZY_SWEEP"]

./benchmark/gc/rdoc.rb
      user     system      total        real
 18.517571   0.764057  19.281628 ( 19.303723)
GC total time (sec): 0

VmHWM: 411780 kB

Summary of rdoc on 2.7.0dev	19.303723024990177	0	137
         (real time in sec, GC time in sec, GC count)

```

Array specific (adapted from the original issue to include `RUBY_DESCRIPTIO=
N`):


```
p RUBY_DESCRIPTION
require 'benchmark'
N =3D 10_000_000

def n_times str, args =3D ''
  eval <<-EOS
    proc{|max, #{args}|
      i =3D 0
      while i < max
        #{str}
        i+=3D1
      end
    }
  EOS
end

m =3D n_times 'ary =3D Array.new(size)', 'size'

Benchmark.bm(10){|x|
  0.step(to: 16){|i|
    size =3D i
    x.report(size){
      m.call(N, size)
    }
  }
}
```

`objspace-theap` branch:

```
lourens@CarbonX1:~/src/ruby/ruby$ ./miniruby -Ilib $HOME/src/theap_array.rb
"ruby 2.7.0dev (2019-09-22T11:43:20Z objspace-theap 11b7839f09) [x86_64-lin=
ux]"
                 user     system      total        real
0            1.060482   0.000000   1.060482 (  1.060477)
1            1.063371   0.000000   1.063371 (  1.063398)
2            1.049839   0.000000   1.049839 (  1.049846)
3            1.066307   0.000000   1.066307 (  1.066311)
4            1.187135   0.000000   1.187135 (  1.187147)
5            1.130953   0.000000   1.130953 (  1.130964)
6            1.160799   0.000000   1.160799 (  1.160805)
7            1.220265   0.000000   1.220265 (  1.220272)
8            1.160544   0.000000   1.160544 (  1.160554)
9            1.262271   0.000000   1.262271 (  1.262282)
10           1.182926   0.000000   1.182926 (  1.182933)
11           1.305176   0.000000   1.305176 (  1.305187)
12           1.170598   0.000000   1.170598 (  1.170605)
13           1.243933   0.000000   1.243933 (  1.243940)
14           1.197929   0.000000   1.197929 (  1.197955)
15           1.257225   0.000000   1.257225 (  1.257232)
16           1.203359   0.000000   1.203359 (  1.203365)
```

master:

```
lourens@CarbonX1:~/src/ruby/trunk$ ./miniruby -Ilib $HOME/src/theap_array.rb
"ruby 2.7.0dev (2019-09-22T07:39:47Z master 2272efa463) [x86_64-linux]"
                 user     system      total        real
0            1.034709   0.000000   1.034709 (  1.035330)
1            1.073345   0.000000   1.073345 (  1.073350)
2            1.046693   0.000000   1.046693 (  1.046704)
3            1.046481   0.000000   1.046481 (  1.046487)
4            1.135405   0.000000   1.135405 (  1.135410)
5            1.152307   0.000000   1.152307 (  1.152308)
6            1.163179   0.000000   1.163179 (  1.163184)
7            1.229176   0.000000   1.229176 (  1.229193)
8            1.171530   0.000000   1.171530 (  1.171541)
9            1.229018   0.000000   1.229018 (  1.229023)
10           1.218360   0.000000   1.218360 (  1.218369)
11           1.258088   0.000000   1.258088 (  1.258093)
12           1.214547   0.000000   1.214547 (  1.214557)
13           1.246675   0.000000   1.246675 (  1.246693)
14           1.217621   0.000000   1.217621 (  1.217620)
15           1.240967   0.000000   1.240967 (  1.240972)
16           1.246610   0.000000   1.246610 (  1.246620)

```

Optcarrot (noisy too):

```
lourens@CarbonX1:~/src/optcarrot$ benchmark-driver -e "objspace-theap::~/sr=
c/ruby/ruby/ruby -I~/src/ruby/ruby/lib -I~/src/ruby/ruby/. -I~/src/ruby/rub=
y/.ext/x86_64-linux" -e "trunk::~/src/ruby/trunk/ruby -I~/src/ruby/trunk/li=
b -I~/src/ruby/trunk/. -I~/src/ruby/trunk/.ext/x86_64-linux" --repeat-count=
 24 --output=3Dall -v benchmark.yml
objspace-theap: ruby 2.7.0dev (2019-09-22T11:43:20Z objspace-theap 11b7839f=
09) [x86_64-linux]
trunk: ruby 2.7.0dev (2019-09-22T07:39:47Z master 2272efa463) [x86_64-linux]
Calculating -------------------------------------
                           objspace-theap                 trunk =

           optcarrot    42.40538956057937     42.50095691198821 fps
                        42.98025714388297     43.19539965591333 =

                        43.17850049383824     43.69380960879121 =

                        44.34221997472645     43.75925724088885 =

                        45.23015773087887     43.95609333352798 =

                        45.60259428038163     44.30873259117270 =

                        45.62534786154934     44.50561054139629 =

                        45.69848233271295     44.74074528123102 =

                        45.72577876475985     44.75584411276473 =

                        45.82588019886050     44.90017657928921 =

                        45.84436354482752     44.93827167130445 =

                        46.72210774150135     45.32808901780013 =

                        46.78934236282241     45.40725425997227 =

                        46.83007726756269     45.50752517888226 =

                        46.84959507374096     45.51210655013940 =

                        46.93537135184541     45.51427328345052 =

                        47.20234803912213     45.61616028100521 =

                        47.36550159135397     45.90162191663695 =

                        47.95694822288893     46.18908487592959 =

                        48.00296030568151     46.19919623309980 =

                        49.21328383054638     46.47376631857036 =

                        49.23558095502081     46.85201420357048 =

                        49.30196173221466     47.39761548726396 =

                        50.07004688586536     47.70501205501753
```

----------------------------------------
Misc #16124: Let the transient heap belong to objspace
https://bugs.ruby-lang.org/issues/16124#change-81664

* Author: methodmissing (Lourens Naud=E9)
* Status: Assigned
* Priority: Normal
* Assignee: ko1 (Koichi Sasada)
----------------------------------------
As per comment from Nobu in https://github.com/ruby/ruby/pull/2303#issuecom=
ment-523248875 , I took an initial stab @ a tighter integration between obj=
space and the transient heap in https://github.com/ruby/ruby/pull/2400

### Benefits

* Multi-VM (MVM) friendly - ( vm -> objspace -> theap )
* The 32MB (current size) arena lazy allocated on ruby init is now properly=
 freed on shutdown as well
* It feels strange that the evacuation from the current global theap is to =
objspace, whereas the space evacuated from is a global arena.

### Not so great

* A fast reference to a global variable `global_transient_heap` becomes a f=
unction call to `rb_objspace_get_theap()` and related pointer chasing from =
vm -> objspace -> theap
* Some internal transient heap structs moved to the header file now leaks i=
nto all other reference sites where this source file (`transient_heap.c`) a=
s previously just used for API
* I'm not sure exactly of the boundary Koichi had in mind for the GC compil=
e module and how tightly it should (or shouldn't) be coupled to the transie=
nt heap. `struct rb_objspace*` declarations elsewhere for example reveals n=
othing about the structure members for example, whereas with this PR a lot =
of transient heap internals are exposed via the header file now
* Also possible to move `transient_heap.c` into `gc.c` - I feel theap is no=
t an experimental feature anymore and has been stable for quite some time w=
ith plausible performance benefits. The downside of that is `gc.c` is quite=
 dense already, but then all ruby heap management concerns belong to one co=
mpile unit.

In a similar vein the global method cache could perhaps belong to the VM in=
stance as well, effectively better alignment with MVM and also easier to ha=
ve a balanced VM setup and teardown sequence without anything left dangling=
 on ruby shutdown.

Thoughts?



-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>