[ruby-core:118521] [Ruby master Bug#20587] dir.c calls blocking filesystem APIs/system calls while holding the GVL

Issue #20587 has been updated by ivoanjo (Ivo Anjo).


jeremyevans0 (Jeremy Evans) wrote in #note-2:
> @ivoanjo Can you be specific about which places this affects?  I examined all `opendir` calls in `dir.c`:
> 
> * Called in `nogvl_opendir`
> * Called in `opendir_without_gvl` when VM is not initialized
> * Called in `nogvl_opendir_at`
> * Called in `nogvl_dir_empty_p`
> 
> Tracing those function calls, if the VM is initialized, it appears that all calls use the `IO_WITHOUT_GVL` macro to release the GVL around the call.

Hey Jeremy, thanks for taking a look at this!

You are right -- I think I was overeager in including `readdir` on the list. I'll update my original description to correct for this.

Here's my attempt at a more specific report of blocking calls while holding the GVL in `dir.c` (I've omitted cases where the GVL is released):

* `fdopendir` -> called by `dir_s_for_fd` 
* `readdir` -> called by `dir_read`, `dir_each_entry`, `glob_helper`, `glob_getent`
* `chdir` -> called by `dir_chdir0` (Although this one is somewhat weird already so perhaps it being blocking is not the weirdest part?)
* `fchdir` -> called by `dir_fchdir` (same as `chrdir`)
* `chroot` -> called by `dir_s_chroot`
* `lstat` -> called by `do_lstat`
* `stat` -> called by `do_stat`
* `fstatat` -> called by `do_stat`/`do_lstat`
* `fgetattrlist` -> called by `need_normalization`, `is_case_sensitive`
* `getattrlist` -> called by `is_case_sensitive`, `replace_real_basename`, `rb_dir_s_empty_p`, `need_normalization`, `dir_initialize`
* `getpwnam` -> called by `dir_s_home`

----------------------------------------
Bug #20587: dir.c calls blocking filesystem APIs/system calls while holding the GVL
https://bugs.ruby-lang.org/issues/20587#change-109041

* Author: ivoanjo (Ivo Anjo)
* Status: Open
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
Hey! I work for Datadog on the Ruby profiler part of the [`datadog` (previously `ddtrace`)](https://github.com/datadog/dd-trace-rb) gem.

While I was investigating https://bugs.ruby-lang.org/issues/20586, I spotted that there's a number of cases where, in `dir.c`, blocking system calls are being made (e.g. `readdir()`, `opendir()`, etc) without releasing the GVL.

This means that if they block for a long time (as happens in the gcsfuse example in https://bugs.ruby-lang.org/issues/20586 ), the Ruby VM will just be blocked and not make any progress.

The combination of not releasing the GVL + slow system calls actually makes the issue in https://bugs.ruby-lang.org/issues/20586 more likely to happen with the Datadog profiler, although even if the code releases the GVL the underlying issue could still happen, and this is why I decided to file this bug separately.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/