[ruby-core:41173] Re: [Backport93 - Backport #5621] Please backport thread-safe autoloading patch

(2011/11/19 9:08), Charles Oliver Nutter wrote:
>>> Each classloader has a single lock, and if it needs to call a
>>> parent classloader the locks are still always acquired in the same
>>> order (meaning deadlock is impossible).
>>
>> 1) No parallel loading in single classloader
>>   (single lock, disallowing concurrent loading)
>=20
> Correct.
>
>> 2) Classloaders know class loading order at compile time to avoid
>>   deadlock
>=20
> No, but classloading always progresses in the same direction, from
> dependent classes to dependencies and from child classloaders to
> parent classloaders, so having multiple locks involved can't deadlock.

I see.  Maybe I misunderstood 'a single lock' concept of 1).  1) is not
needed, and 2) should be enough for safe classloading normally.
Initialization of classes (and interfaces) are defined in JLS[1]
carefully, and no deadlock occurred in normal classloading scheme
(parent to child.)

Customized classloader can cause a deadlock, and Java SE 7 introduced a
manual way to avoid this for classloader developer.

[1]
http://java.sun.com/docs/books/jls/third_edition/html/execution.html#12.4=
=2E2
[2] http://docs.oracle.com/javase/7/docs/technotes/guides/lang/cl-mt.html=


> Also, classloading does not in itself trigger any code execution; not
> until the class has successfully loaded do static initializers run.
> Ruby's problem with parallel loading is that files are executed
> *while* they're loaded, which makes running them in parallel
> inherently unsafe.

Sure. We cannot control code execution order of initializers (that's
just a chunk of code in Ruby) and deadlock is inherently unavoidable if
developer writes a 'wrong code' such as;

# file foo.rb
class Foo
  require 'bar'
  Bar.something
end

# file bar.rb
class Bar
  require 'foo'
  Foo.something
end

As the existing warning 'circular require considered harmful', current
Ruby treats it as a developer's job to avoid this 'wrong code'.  I was
trying to solve threading issues of require and autoload, and I think we
solved it at #921 unless developers don't write 'wrong code'.

I understand that Yehuda and you are saying that it's not a 'wrong
code.'  It's hard to define 'wrong code' I agree.  I still wanna see
concrete examples which are not 'wrong code' but I think it's OK to fix
the issue inherently by introducing global (per VM) file loading lock if
it doesn't cause serious degradation in 2.0.

Isn't there an application which uses Threads + parallel execution by
"load" method?  I don't have concrete example :)

>>> You're right, though...it's important to point out Yehuda's problem
>>> is not an autoload problem. Autoload thread-safety itself is
>>> achievable. Require thread-safety doesn't exist even today. That's
>>> the real problem.
>>
>> Sure.  Moreover, I think that require thread-safety could not be
>> solved easily in Ruby, so I want to find a different way to solve
>> Yehuda's real problem.
>=20
> I think we could claim autoload is thread-safe in itself if it always
> just used a single lock. In other words, only one thread can be
> processing autoloads at a time. This doesn't fix the parallel require
> issue, but it would fix parallel autoloading.
>=20
> I think that's an ok fix, no?

Agreed.  At least it's worth trying.

I'll create a new ticket for the change (single lock require.).  This
ticket is for backporting thread-safe autoload (unless you don't write
'wrong code' :-) by Mike.  I'll wait for the report if the existing
thread-safe autoload on trunk works for him or not.