[ruby-core:42716] [ruby-trunk - Feature #5654] Introduce global lock to avoid concurrent require

Issue #5654 has been updated by Charles Nutter.


Finally looping back to this to add some thoughts. Sorry for the delay.

The biggest problem with concurrent code in Ruby (at least from the JRuby perspective) has not been thread safe collections, deadlocks, or anything in the core runtime...it has been the fact that lazy requires can cause a class to be visible before it is ready to be used.

Users are forced to choose between loading all files up front (and taking whatever startup hit comes along with that) or lazily loading files and hoping nobody tries to access the same class during that load. If requires were not allowed to run concurrently, there would be far less chance for a class to be partially initialized before it is used.

There are very few justifications for allowing concurrent requires.

One is the case Yehuda mentions, where you have a blocking call or loop in the body of a file you are requiring. I would argue this case is better served by loading, or by requiring the library you want to require and then calling a method. Using the example of the JVM...it is almost universally forbidden to do long, blocking operations in classes' static code executed on load, because it not only blocks loading but causes that thread to pause in the middle of initializing a given class.

The only other case would be to paralellize loading of many libraries. But this is flawed as well, since your loading process will potentially also see partially-initialized classes, and it could easily deadlock (with require locks as in 1.9) if libraries depend on each other.

So the only case that seems like it would be penalized is the case of running a server or loop or blocking call in the body of a script. If that case is deemed valid, perhaps "load" is the right choice instead. I do not believe "load" should have the same restriction as "require", since to me it represents the "raw" way to load code, and it can be an escape hatch for people who know what they're doing (like I hope those server/loop/blocking-call authors do).

A side note: the locking that has been added to "help" this problem has actually introduced additional problems. Specifically, both "thread safe autoload" and per-file require locks can lead to deadlocking at *load time*. This would be impossible if there were a single global lock for requires, and the goal of those fixes would still be fulfilled.

To further experimentation with this feature, we have added a flag to JRuby that turns on a global lock for requires: -Xglobal.require.lock=true. This is available in "master" builds of JRuby, downloadable from http://ci.jruby.org/snapshots/master.

I strongly encourage the Ruby elders to make requires lock on a single global lock.


----------------------------------------
Feature #5654: Introduce global lock to avoid concurrent require
https://bugs.ruby-lang.org/issues/5654

Author: Hiroshi Nakamura
Status: Open
Priority: Normal
Assignee: 
Category: core
Target version: 2.0.0


=begin
Current implementation of "require" has locks for each file (expanded name from required feature) and serializes file loading from Threads.  The first Thread acquires the lock for the file and starts loading. The second Thread waits for acquiring the lock, and when the first Thread release the lock, it acquires the lock, then returns immediately.
This can cause deadlock by cross-require from Threads.

And code that does not properly use "require" could meet several problems;
 * constants can be defined before they're ready for use
 * classes can be modified while they're being used
 * global state can be initialized at the same time it's in use

Proposal: introduce global (per VM) lock to avoid concurrent file loading by "require" so that only one Thread can call "require" at the same time.

I don't have pros/cons list at this moment.  Let's discuss it, too.

Derived from a discussion at #5621 (thread-safe autoload)
=end



-- 
http://bugs.ruby-lang.org/