From: "jeremyevans0 (Jeremy Evans) via ruby-core" <ruby-core@...>
Date: 2026-10-24T04:35:14+00:00
Subject: [ruby-core:127226] [Ruby Feature#22466] Allow disabling the shape redblack cache via RUBY_SHAPE_REDBLACK_CACHE=disable

Issue #22466 has been updated by jeremyevans0 (Jeremy Evans).


alanwu (Alan Wu) wrote in #note-1:
> > If it is preferable to specify the cache size via an environment
> variable, instead of just the ability to disable the cache, I can work
> on that approach instead.
> 
> 
> My 2 cents, since we have had `RUBY_{THREAD_VM,THREAD_MACHINE,FIBER_VM,FIBER_MACHINE}_STACK_SIZE` for a long time, specifying a size seems more natural. Not my call to make, though.

I implemented support for specifying the cache size: https://github.com/ruby/ruby/pull/19509

However, it results in assertion failure if the cache size is set to a low value. Maybe it's a bug in the patch, but based on the error, I'm not sure whether the shape redblack cache correctly handles cases where the cache becomes full. Here's the error:

```
  1) Failure:
  TestRubyVM#test_shape_redblack_cache_size [/home/runner/work/ruby/ruby/src/test/ruby/test_rubyvm.rb:36]:
  pid 89225 killed by SIGABRT (signal 6) (core dumped)
  | ../src/shape.c:275: Assertion Failed: redblack_insert_aux:right != LEAF
  | ruby 4.1.0dev (2026-10-24T03:21:24Z shape-redblack-cache-size 702701bef0) +PRISM [x86_64-linux]
```

With the huge cache used currently, I'm guessing this is not an issue, since no reasonable application will exceed the cache size. If this can be fixed, being able to specify the cache size seems more flexible, but if not, I would at least like the ability to disable the cache.

----------------------------------------
Feature #22466: Allow disabling the shape redblack cache via RUBY_SHAPE_REDBLACK_CACHE=disable
https://bugs.ruby-lang.org/issues/22466#change-119642

* Author: jeremyevans0 (Jeremy Evans)
* Status: Open
----------------------------------------
The shape redblack cache mmaps 384MB of memory at startup. There is
currently no way to configure the amount of memory used. The code
already handles an mmap failure cleanly. I would like a way to skip
the mmap attempt, and the simplest way I can think of is an
environment variable.

The reason behind the current behavior (I think) is that ruby
currently assumes that the mmap is basically free, since the memory
is not actually allocated in most cases. However, when operating
system strict memory overcommit is used, this 384MB mmap allocation
can result in the serious problems later with the operating system
killing the process if it is low on memory. This provides a way to
accept a performance hit and avoid the memory allocation.

We are currently working around this issue in production by decreasing
the amount of memory available to the process at startup to 256MB
(causing the mmap allocation to fail), and then increasing it after
startup using `Process.setrlimit`. However, that results in a lot of
additional complexity that I would prefer to avoid.

If it is preferable to specify the cache size via an environment
variable, instead of just the ability to disable the cache, I can work
on that approach instead.

I submitted a pull request to implement this: https://github.com/ruby/ruby/pull/19499



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/