[ruby-core:107997] [Ruby master Bug#18627] segmentation fault when doing a lot of redundant Module#include

Issue #18627 has been updated by nagachika (Tomoyuki Chikanaga).

Backport changed from 2.6: DONTNEED, 2.7: DONTNEED, 3.0: REQUIRED, 3.1: REQUIRED to 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONE, 3.1: REQUIRED

ruby_3_0 e0146e6cc8f3578b02ad5f228f86bf1aef566d16 merged revision(s) 97426e15d721119738a548ecfa7232b1d027cd34.

----------------------------------------
Bug #18627: segmentation fault when doing a lot of redundant Module#include
https://bugs.ruby-lang.org/issues/18627#change-96950

* Author: Ethan (Ethan -)
* Status: Closed
* Priority: Normal
* ruby -v: ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x86_64-linux]
* Backport: 2.6: DONTNEED, 2.7: DONTNEED, 3.0: DONE, 3.1: REQUIRED
----------------------------------------
I'm adding support for ruby 3 and consistently encountering segfaults.

my library does a fair bit of extending objects with modules in an #initialize. I instantiate objects corresponding to nodes in a JSON document. each one extends itself with several modules, depending on its role in the document. some of these dynamically create a module, include some other modules into that module, and then extend themself with that module.

at some point (when seems nondeterministic, but the code path is consistent), ruby segfaults while including from a module (which was dynamically created in #initialize) with another module. this happens on 3.0 and 3.1. it doesn't seem to on 3.2.0-dev, my tests pass fine there.

I'm not sure how much to try to explain the code - it is a fairly complex, and massively inefficient. in investigating this issue I identified a way to go from O(n^2) calls to the segfaulting code path down to O(1), and it didn't segfault anymore. I also realized that, in the unoptimized version, almost all of the calls to the segfaulting code path (all but the O(1)) are redundant - I'm calling Module#include when the receiver module already includes the argument module. adding a check to skip the redundant Module#include stopped segfaulting in the inefficient version, too.

so, it looks like Module#include is somehow segfaulting when it should be doing a noop, skipping inclusion of an argument module because its reciever already includes that module.

I've made some attempt at minimally reproducing this, but haven't made it as far as separating it from the rest of the library. so my steps to reproduce at the moment involve cloning my library on the branch where I've added ruby 3 support (branch splat+msim - commit 2a719a23) and invoking that:

```
git clone -b splat+msim https://github.com/notEthan/jsi.git
ruby -Ijsi/lib -rjsi -e 'JSI::JSONSchemaOrgDraft06.new_schema({items: {items: {items: {items: {items: {items: {}}}}}}})'
```

it may a little bit intermittent - that consistently segfaults for me; shallower depths of the object passed to new_schema are less consistent. in case it does not segfault on another computer, adding further depth should trigger it.

attached are outputs of that with segfault backtrace on 3.0 and 3.1.

finally, here is a high level description of what is occurring when segfault occurs, to hopefully give some idea of the context. I can explain any part in more detail if it is helpful.

- JSI::MetaschemaNode#initialize - called an excessive number of times (hundreds to low thousands)
   - extends this JSI::MetaschemaNode with 1-3 modules (namely: JSI::PathedHashNode, JSI::PathedArrayNode, JSI::Metaschema)
   - for certain nodes:
      - dynamically creates a new module (named jsi_schema_module)
      - calls #include on this jsi_schema_module with 1+ other modules (named metaschema_instance_modules)
         - this is the part that segfaults
         - almost all of these include calls are redundant and should be noop
         - may be relevant that this module metaschema_instance_module includes, directly and indirectly, some 39 other modules
   - instantiates zero or more other JSI::MetaschemaNode instances
   - extends this MetaschemaNode with one or more other modules


---Files--------------------------------
git_2a719a_ruby_3.0.2p107-b.log (88.7 KB)
git_2a719a_ruby_3.1.1p18-a.log (67.7 KB)


-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>