From: woodsc@...00f.net (Conan)
Date: 2004-06-09T07:08:18+09:00
Subject: Re: ruby and mustard

Can you just use a faster computer?  If your working for a company its a
good excuse to bug your boss for better hardware, and if your working at
a university you might be able to convince them to let you use their
high-performance computer.(every university has at least 1 supercomputer
right? :) )

The nice thing about code is that your algorithms complexity will be the
same no matter what language it's written in, so scaling hardware is
often a nice easy solution. (Unless your already running it on top of
the line system).



On Tue, Jun 08, 2004 at 11:53:42PM +0900, Ara.T.Howard wrote:
> 
> what do you do when ruby just won't quite cut the mustard?
> 
> here's a scenario from last week:
> 
> i'm doing some trival satelite image processing: find some pixels in one 
> huge
> image which are essentially pointers (indexes) into another image.  for each
> pixel found to be a certain value set those indexes in a bunch of other
> images.  here's the catch: the images are all HUGE - 1-3gb - and i've got to
> be processing sets of 6-8 of them at a time.  most amazingly, a combination 
> of
> guy's mmap and strscan does a good job with this task:  the mmap ensures 
> good
> memory management w/o blowing the top off system limits and strscan is nice
> and fast.  using this combination i'm able to process an image set in 2-4
> minutes.  if you think about it this is a really amazing to be doing with a
> scripting language.  however, we are going to be using this code on 10s of
> thousands of image sets - so every second counts.  i spent a day writing
> equivalent code in c which runs in < 1 minute - nice.  the problems is this:
> it took a DAY!  the ruby code took about 35 minutes to write.  we move at an
> insane pace around here and i hardly ever have a DAY to do anything.  i 
> spent
> monday scanning the web to check out the latest developments in languages.
> ocaml grabbed my eye.  then i spent 4 hours writing my first program in it
> that used Bigarray and it's mmap facility to behave as my c program did.  it
> takes about 1.5 minutes to run and, i must admit, i wasn't really enjoying 
> the
> functinal paradigm - suppose that might change though...
> 
> so here's my delima:
> 
> when you want to write something faster than ruby, but want basic (IMHO)
> tools at your disposal like hashes, good string handling, exceptions, etc.
> what is the way to go?  as i see it there are a few options:
> 
>   * do it in another lang.  i really don't like this because of learning
>     curve and adding extra dependancies (ocamlc for eg.).
> 
>   * c++ is simply out.  ;-)
> 
>   * do it in pure c.  have you used getoptlong lately - sheesh.
> 
>   * do it using a nice library for c.  glib is good - lot's of bells.
>     extra dependancy though...
> 
> these are the options i've been mulling over.  lately, however, i'm starting
> to favour this option:
> 
>   * just code it in c using ruby's builtin libs.  gives you hashes, eval'ing
>     code, lists, GC, etc.  i'm not adding additional dependancies and
>     guarunteed (if my c stays pretty posix) that my code will run where ruby
>     will.  don't need autotools, no stl., etc. etc.
> 
> are there any other options i'm missing?
> 
> what do you do when it needs to be __really__ fast BUT you also have to
> develop it __really__ fast and would __prefer__ not to add dependancies.
> 
> regards.
> 
> -a
> --
> ===============================================================================
> | EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
> | PHONE   :: 303.497.6469
> | A flower falls, even though we love it; and a weed grows, even though we 
> do
> | not love it. --Dogen
> ===============================================================================