[ruby-core:100724] [Ruby master Feature#17307] A way to mark C extensions as thread-safe, Ractor-safe, or unsafe

Issue #17307 has been updated by Eregon (Benoit Daloze).


From discussing with @ko1,
another way to mark would be to call a function in `Init_foo`, like:
```c
static VALUE foo(VALUE self) {
  ...
}

void Init_foo(void) {
  rb_mark_extension_as_thread_safe();
  rb_mark_extension_as_ractor_safe();

  VALUE cMyClass = rb_define_class("MyClass", rb_cObject);
  rb_define_method(cMyClass, "foo", foo, 0); // foo() can executed in parallel safely
}
```

Concretely, `rb_mark_extension_as*` would set some thread-local state so the following `rb_define*` would know they are thread/ractor-safe.
That state would be reset after executing `Init_foo`.
We need to use the save/restore pattern in case `Init_foo` loads another C extension.
That assumes loading a C extension happens on a single-thread, but that seems very much the case.

It's an imperative vs declarative approach.
Declarative seems slightly better to me because the semantics are a bit simpler (there can be no code before `rb_mark_extension_as*`, it is possible to know before starting to execute `Init_foo`).

----------------------------------------
Feature #17307: A way to mark C extensions as thread-safe, Ractor-safe, or unsafe
https://bugs.ruby-lang.org/issues/17307#change-88368

* Author: Eregon (Benoit Daloze)
* Status: Open
* Priority: Normal
----------------------------------------
I would like to design a way to mark C extensions as thread-safe, Ractor-safe, or unsafe (= needs process-global lock).
By default, if not marked, C extensions would be treated as unsafe for compatibility.

Specifically, TruffleRuby supports C extensions, but for scalability it is important to run at least some of them in parallel (e.g., HTTP parsing in Puma).
This was notably mentioned in my [RubyKaigi talk](https://speakerdeck.com/eregon/running-rack-and-rails-faster-with-truffleruby?slide=17).
TruffleRuby defaults to acquire a global lock when executing C extension code for maximum compatibility (Ruby code OTOH can always run in parallel).
There is a command-line option for that lock and it can be disabled, but then it is disabled for all C extensions.
The important property for TruffleRuby is that the C extension does not need a global lock, i.e., that it synchronizes any mutable state in C that could be accessed by multiple threads, such as global C variables.
I believe many C extensions are already thread-safe, or can easily become thread-safe, because they do not rely on global state and do not share the RData objects between threads.

Ractor also needs a way to mark C extensions, to know if it's OK to use the C extension in multiple Ractors in parallel, and that the C extension will not leak non-shareable objects from one Ractor to another, which would lead to bugs & segfaults.
Otherwise, C extensions could only be used on the main/initial Ractor (or need to acquire a process-global lock whenever executing C extension code and ensure no non-shareable objects leak between Ractors), which would be a very big limitation (almost every non-trivial application depends on a C extension transitively).

In both cases, global state in the C extension needs synchronization.
In the thread-safe case, mutable state in C that could be accessed by multiple Ruby threads needs to be synchronized too (there might be no such state, e.g., if C extension objects are created per Thread).
In the Ractor case, the C extension must never pass an object from a Ractor to another, unless it is a shareable object.

What do you think would be a good way to "mark" C extensions?
Maybe defining a symbol in the C extension, similar to the `Init_foo` we have, like say `foo_is_thread_safe`/`foo_is_ractor_safe`?
A symbol including the C extension name seems best, to avoid any possible confusion when looking it up.

Maybe there are other ways to mark C extensions than defining symbols, that could still be read by the Ruby implementation reliably?

I used the term `C extensions` but of course it would apply to native extensions too (including C++/Rust/...).

cc @ko1



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>