[ruby-talk:13224] Re: Native/pthreads in Ruby [Long]

Yukihiro Matsumoto wrote:

> Hi,
>
> In message "[ruby-talk:12981] Re: Native/pthreads in Ruby"
>     on 01/03/21, "Marc Butler" <[e-mail]> writes:
>
> |>From cursory inspection of the source code Ruby appears to use a
> |conservative Mark-Sweep garbage collector.  This would require all threads
> |that my potentially 'mutate' the heap be suspended before the garbage
> |collector runs, and remain suspened until it has finished.  POSIX threads
> |does *not* provides mechanism by which threads may be suspended without
> |cooperation. [LinuxThreads, 2001]
>
> Giant interpreter lock like Python's may solve this.

> |Garbage collection and Threading are of interest to me and I would be happy
> |to discuss in more detail with like minded list participants.
>
> Inform me since I'm not a thread expert.
>
> Currently, I'm thinking of the scheme following.
>
>   * add configure option to enable native-thread (which is off by
>     default).
>
>   * add interpreter lock for native-thread enabled interpreter.
>
>   * wrap pthread_create() to preserve stack address for each new
>     thread like Bohem GC.
>
> But there might be better way.
>
>                                                         matz.

Hi matz,

I apologize in the long delay in replying to you.  Family commitments have kept
me away from the computer for some time.  I have developed and modified threaded
C++ software on Solaris and Linux for close to 4 years now, but I doubt I am
entitled to consider myself an expert.  I apologize if this appeared to be my
pretense.

I do have some questions and observations for you regarding Ruby's garbage
collector and the possible integration of native threads.

Certainly a global interpreter lock would achieve the synchronization necessary
to run the mark-sweep collector.  It seems natural to place this lock in the
memory allocation routine, so that only threads that are attempting to allocate
memory are locked during garbage collection.

I am aware of the Bohem GC but entirely unfamiliar with its implementation.
Given it is a well established and widely used implementation of a conservative
collector; it seems like a great candidate for a template for Ruby.

I know there are certain annoyances that you would have to accommodate for like
whether the stack grows up or down from the address retrieved from
pthread_getstackaddr and so forth; however such things are trivial.  It is not
clear to me however whether the library reserves the right to move the stack
address around or not.  If this may be the case you might want to set the stack
address and size explicitly when you create the thread.

I am curious as to why ruby uses a conservative collector.  I know that Ruby is
a type-less language but I would have thought the run time structures used by
the interpreter would provide enough information to avoid the need of the
collector.  I am only just learning about garbage collection, and have not
developed any intuition about such systems.

I had imagined that it might be possible to implement a thread oriented
collector using the concepts that drive a generational collector.  If we ignore
thread recycling for convenience, threads often fall into two categories:
short-lived work threads that perform specific tasks; and long-lived threads
that handle blocking system calls (like I/O).  By using the thread id like a
special reference you can dereference all the data allocated in that threads
lifetime, if other threads now have references to that data it will still be
safe.

I realize that the previously described scheme is at best "half-baked" as I am
unable to cook up a nice scheme to deal with long lived threads. However I was
hoping it might be a seed for someone.  Perhaps one of the distributed garbage
collection schemes could be modified to work in the context of Ruby?

marc