From: "Ara.T.Howard" <Ara.T.Howard@...>
Date: 2005-09-08T10:24:43+09:00
Subject: Re: sharing memory in ruby

On Thu, 8 Sep 2005, Yukihiro Matsumoto wrote:

> Hi,
>
> In message "Re: sharing memory in ruby"
>    on Thu, 8 Sep 2005 09:48:48 +0900, "Ara.T.Howard" <Ara.T.Howard@noaa.gov> writes:
>
> |any thoughts on a good approach for people that want to do some memory sharing
> |between ruby objects?  say strings and mmaps and narrays and images all
> |sharing some memory regions.  any do's and dont's or other thoughts?
>
> I don't thing you need to share memory within same processes.  Just
> refer to a same object.  mmap seems reasonable solution to share
> memory between processes.  The another idea is using some kind of RPC,
> such as dRuby instead of sharing memory.

attm.  i'm writing some code to process images for katrina, check out

   http://dmsp.ngdc.noaa.gov/interest/katrina.html

i sometimes must shared memory between an mmap, ruby string, and an narray to
i can work on it in memory.  our machines have 8gb, but a couple channels and
some duplicate data and i run out...  bascially i have an extension to narray
that allows it to share it's storage with an mmap in the same process but,
apparently, this is frowned upon?  bascially i allow any string type object to
be passed to an narray and it uses that string's memory without copying it -
in this way i can

   na = NArray::str mmap.to_s, samples, lines, NArray::SINT

any comments on this approach?

my narray.c patch is below...

the really critical bit is this:

   +  ary->ptr = RSTRING(str)->ptr;
   +  ary->ref = Qfalse;

the 'ary->ref' means it will not be free'd...

so now the NArray and String, which is itself an Mmap are all sharing memory.
the neat thing is that i can do this:

   na = na + 1

and a the file backing is affected with no io on my part.  also i can load
image larger than memory and, therfore, many large images at once.  is this a
terrible idea?  if not what alternative might i use?  perhaps a generic
Pointer (shudder) object that behaves like a String might be useful...

cheers.

-a
-- 
===============================================================================
| email :: ara [dot] t [dot] howard [at] noaa [dot] gov
| phone :: 303.497.6469
| Your life dwells amoung the causes of death
| Like a lamp standing in a strong breeze.  --Nagarjuna
===============================================================================

--- narray.c.org	2004-10-15 20:57:34.000000000 -0600
+++ narray.c	2004-10-15 21:40:21.000000000 -0600
@@ -757,6 +757,54 @@


  /* singleton method: 
+   NArray.str( string, type, size1,size2,...,sizeN )
+*/
+static VALUE
+ na_s_str(int argc, VALUE *argv, VALUE klass)
+{
+  struct NARRAY *ary;
+  VALUE v;
+  VALUE str;
+  int i, type, len=1, str_len, *shape, rank=argc-2;
+
+  if (argc < 1)
+    rb_raise(rb_eArgError, "String Argument required");
+
+  if (argc < 2)
+    rb_raise(rb_eArgError, "Type and Size Arguments required");
+
+  type = na_get_typecode(argv[1]);
+
+  str = rb_funcall(argv[0], rb_intern("to_str"), 0);
+
+  str_len = RSTRING(str)->len;
+
+  if (argc == 2) {
+    rank  = 1;
+    shape = ALLOCA_N(int,rank);
+    if ( str_len % na_sizeof[type] != 0 )
+      rb_raise(rb_eArgError, "string size mismatch");
+    shape[0] = str_len / na_sizeof[type];
+  }
+  else {
+    shape = ALLOCA_N(int,rank);
+    for (i=0; i<rank; i++)
+      len *= shape[i] = NUM2INT(argv[i+2]);
+    len *= na_sizeof[type];
+    if ( len != str_len )
+      rb_raise(rb_eArgError, "size mismatch");
+  }
+ 
+  v = na_make_object( type, rank, shape, cNArray );
+  GetNArray(v,ary);
+  ary->ptr = RSTRING(str)->ptr;
+  ary->ref = Qfalse;
+
+  return v;
+}
+
+
+/* singleton method:
     NArray[object]
  */
  static VALUE
@@ -1189,6 +1237,7 @@
      rb_define_singleton_method(cNArray,"to_na",na_s_to_na,-1);
      rb_define_singleton_method(cNArray,"to_narray",na_s_to_na,-1);
      rb_define_singleton_method(cNArray,"[]",na_s_bracket,-1);
+    rb_define_singleton_method(cNArray,"str",na_s_str,-1);

      /* methods */
      rb_define_method(cNArray, "[]", na_aref,-1);