Re: Ruby lacks atfork : The evil that lives in fork...

On Tue, 7 Oct 2008, Yukihiro Matsumoto wrote:

> When you fork off the process, the entire resources are (virtually)
> copied, so that there's no way to ensure (copied) mutex to share
> locking status across processes.  The basic rule is: don't mix threads
> (and thread related resources like mutex) with processes.

The problem with "don't mix threads with processes" is unless you
inspect the source code of each version of each library in turn...  it
is very hard to prove that nothing in your system is using a thread
and a process together.

Ah, but that's the point of pthread_atfork... it gives you several
possible strategies to allow you to mix threads with processes...

1) Give the resource to the child.

    Use the atfork handler to lock the mutex in the prepare handler
    (blocking if need be until you can obtain it) perform the fork,
    release the lock in the child handler. Thereafter the resource will be
    unobtainable in the parent until the child exits, and the child may
    continue to use the resource as need be.

2) Give the resource to the parent.

    Use atfork to lock the resource in the prepare handler, ensure it is
    locked in the child handler, unlock in the parent
    handler. Thereafter the child process will find it is always
    locked, and the parent process will be able to access it.

3) Lock both out for the duration.

4) Mutate the mutex into a valid interprocess lock like flock or
    fcntl.


Sun solaris has a "fork_all" variant that creates copies of all
actives threads as well... but I'm not sure that 'fork_all" really
solves the problem instead of multiplying it.

What the open group has to say on the subject is informative...

http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html

     There are at least two serious problems with the semantics of
     fork() in a multi-threaded program. One problem has to do with
     state (for example, memory) covered by mutexes. Consider the case
     where one thread has a mutex locked and the state covered by that
     mutex is inconsistent while another thread calls fork(). In the
     child, the mutex is in the locked state (locked by a nonexistent
     thread and thus can never be unlocked). Having the child simply
     reinitialize the mutex is unsatisfactory since this approach does
     not resolve the question about how to correct or otherwise deal
     with the inconsistent state in the child.

     It is suggested that programs that use fork() call an exec
     function very soon afterwards in the child process, thus resetting
     all states. In the meantime, only a short list of
     async-signal-safe library routines are promised to be available.

     Unfortunately, this solution does not address the needs of
     multi-threaded libraries. Application programs may not be aware
     that a multi-threaded library is in use, and they feel free to
     call any number of library routines between the fork() and exec
     calls, just as they always have. Indeed, they may be extant
     single-threaded programs and cannot, therefore, be expected to
     obey new restrictions imposed by the threads library.

     On the other hand, the multi-threaded library needs a way to
     protect its internal state during fork() in case it is re-entered
     later in the child process. The problem arises especially in
     multi-threaded I/O libraries, which are almost sure to be invoked
     between the fork() and exec calls to effect I/O redirection. The
     solution may require locking mutex variables during fork(), or it
     may entail simply resetting the state in the child after the
     fork() processing completes.

     The pthread_atfork() function provides multi-threaded libraries
     with a means to protect themselves from innocent application
     programs that call fork(), and it provides multi-threaded
     application programs with a standard mechanism for protecting
     themselves from fork() calls in a library routine or the
     application itself.

     The expected usage is that the prepare handler acquires all mutex
     locks and the other two fork handlers release them.

     For example, an application can supply a prepare routine that
     acquires the necessary mutexes the library maintains and supply
     child and parent routines that release those mutexes, thus
     ensuring that the child gets a consistent snapshot of the state of
     the library (and that no mutexes are left
     stranded). Alternatively, some libraries might be able to supply
     just a child routine that reinitializes the mutexes in the library
     and all associated states to some known value (for example, what
     it was when the image was originally executed).

     When fork() is called, only the calling thread is duplicated in
     the child process. Synchronization variables remain in the same
     state in the child as they were in the parent at the time fork()
     was called. Thus, for example, mutex locks may be held by threads
     that no longer exist in the child process, and any associated
     states may be inconsistent. The parent process may avoid this by
     explicit code that acquires and releases locks critical to the
     child via pthread_atfork(). In addition, any critical threads need
     to be recreated and reinitialized to the proper state in the child
     (also via pthread_atfork()).

     A higher-level package may acquire locks on its own data
     structures before invoking lower-level packages. Under this
     scenario, the order specified for fork handler calls allows a
     simple rule of initialization for avoiding package deadlock: a
     package initializes all packages on which it depends before it
     calls the pthread_atfork() function for itself.

Yes, I'm aware the author of that document was describe POSIX pthreads
not ruby threads. But clearly the same problems exist in both.


John Carter                             Phone : [phone]
Tait Electronics                        Fax   : [phone]
PO Box 1645 Christchurch                Email : [e-mail]
New Zealand