From: John Carter Date: 2008-10-07T09:41:55+09:00 Subject: Re: Ruby lacks atfork : The evil that lives in fork... On Tue, 7 Oct 2008, Yukihiro Matsumoto wrote: > When you fork off the process, the entire resources are (virtually) > copied, so that there's no way to ensure (copied) mutex to share > locking status across processes. The basic rule is: don't mix threads > (and thread related resources like mutex) with processes. The problem with "don't mix threads with processes" is unless you inspect the source code of each version of each library in turn... it is very hard to prove that nothing in your system is using a thread and a process together. Ah, but that's the point of pthread_atfork... it gives you several possible strategies to allow you to mix threads with processes... 1) Give the resource to the child. Use the atfork handler to lock the mutex in the prepare handler (blocking if need be until you can obtain it) perform the fork, release the lock in the child handler. Thereafter the resource will be unobtainable in the parent until the child exits, and the child may continue to use the resource as need be. 2) Give the resource to the parent. Use atfork to lock the resource in the prepare handler, ensure it is locked in the child handler, unlock in the parent handler. Thereafter the child process will find it is always locked, and the parent process will be able to access it. 3) Lock both out for the duration. 4) Mutate the mutex into a valid interprocess lock like flock or fcntl. Sun solaris has a "fork_all" variant that creates copies of all actives threads as well... but I'm not sure that 'fork_all" really solves the problem instead of multiplying it. What the open group has to say on the subject is informative... http://www.opengroup.org/onlinepubs/009695399/functions/pthread_atfork.html There are at least two serious problems with the semantics of fork() in a multi-threaded program. One problem has to do with state (for example, memory) covered by mutexes. Consider the case where one thread has a mutex locked and the state covered by that mutex is inconsistent while another thread calls fork(). In the child, the mutex is in the locked state (locked by a nonexistent thread and thus can never be unlocked). Having the child simply reinitialize the mutex is unsatisfactory since this approach does not resolve the question about how to correct or otherwise deal with the inconsistent state in the child. It is suggested that programs that use fork() call an exec function very soon afterwards in the child process, thus resetting all states. In the meantime, only a short list of async-signal-safe library routines are promised to be available. Unfortunately, this solution does not address the needs of multi-threaded libraries. Application programs may not be aware that a multi-threaded library is in use, and they feel free to call any number of library routines between the fork() and exec calls, just as they always have. Indeed, they may be extant single-threaded programs and cannot, therefore, be expected to obey new restrictions imposed by the threads library. On the other hand, the multi-threaded library needs a way to protect its internal state during fork() in case it is re-entered later in the child process. The problem arises especially in multi-threaded I/O libraries, which are almost sure to be invoked between the fork() and exec calls to effect I/O redirection. The solution may require locking mutex variables during fork(), or it may entail simply resetting the state in the child after the fork() processing completes. The pthread_atfork() function provides multi-threaded libraries with a means to protect themselves from innocent application programs that call fork(), and it provides multi-threaded application programs with a standard mechanism for protecting themselves from fork() calls in a library routine or the application itself. The expected usage is that the prepare handler acquires all mutex locks and the other two fork handlers release them. For example, an application can supply a prepare routine that acquires the necessary mutexes the library maintains and supply child and parent routines that release those mutexes, thus ensuring that the child gets a consistent snapshot of the state of the library (and that no mutexes are left stranded). Alternatively, some libraries might be able to supply just a child routine that reinitializes the mutexes in the library and all associated states to some known value (for example, what it was when the image was originally executed). When fork() is called, only the calling thread is duplicated in the child process. Synchronization variables remain in the same state in the child as they were in the parent at the time fork() was called. Thus, for example, mutex locks may be held by threads that no longer exist in the child process, and any associated states may be inconsistent. The parent process may avoid this by explicit code that acquires and releases locks critical to the child via pthread_atfork(). In addition, any critical threads need to be recreated and reinitialized to the proper state in the child (also via pthread_atfork()). A higher-level package may acquire locks on its own data structures before invoking lower-level packages. Under this scenario, the order specified for fork handler calls allows a simple rule of initialization for avoiding package deadlock: a package initializes all packages on which it depends before it calls the pthread_atfork() function for itself. Yes, I'm aware the author of that document was describe POSIX pthreads not ruby threads. But clearly the same problems exist in both. John Carter Phone : (64)(3) 358 6639 Tait Electronics Fax : (64)(3) 359 4632 PO Box 1645 Christchurch Email : john.carter@tait.co.nz New Zealand