From: Eric Hodel <drbrain@...7.net>
Date: 2008-06-30T16:06:49+09:00
Subject: Re: multi-threaded access to shared memory space

On Jun 29, 2008, at 23:58 PM, Greg Willits wrote:
> I have a pure Ruby project (no Rails) where I would like multiple
> "tasks" (ruby processes more or less) to run in parallel (collectively
> taking advantage of multiple CPU cores) while accessing a shared  
> memory
> space of data structures.
>
> OK, that's a mouthful.
>
> - single machine, multiple cores (4 or 8)
>
> - step one: pre-load a number of arrays and hashes (could be a  
> couple GB
> worth in total) into memory
>
> - step two: launch several independent Ruby scripts to search and read
> from the data pool in order to aggregate data in new sets to be  
> written
> to text files.
>
> Ruby 1.8's threading would seem poorly suited to this. Can 1.9 run
> multiple threads each accesing the same RAM-space while using all  
> cores
> of the machine?

At present, 1.9 has a global VM lock, so only one C thread can be  
running ruby code at a time.

> I've looked at memcache, but it seems like it could store and retrieve
> one of my pool's arrays, but it cannot look inside that array and
> retrieve just a single row of it? It would want to return the whole
> array, yes? (not good if that array is 100MB).

memcache is just a cache and not designed to be used as a persistent  
store.  It may loose your data if you are not careful.

You're probably looking for something mmap and several forked  
cooperative processes.