From: ara.t.howard@...
Date: 2007-03-03T00:55:32+09:00
Subject: Re: [ANN] rq-3.0.0 : ruby queue gets gem'd

On Thu, 1 Mar 2007, Jeremy Hinegardner wrote:

YO - i'm cc'ing ruby-talk on this.  FYI.

> Ara, what the heck has rq turned into :-).  A chicken and the egg situation?
>
>    % gem specification rq | grep packages
>     - all/packages
>     - all/packages/arrayfields-3.6.0.tgz
>     - all/packages/sqlite-ruby-1.3.1.tar.gz
>     - all/packages/lockfile-1.4.1.tgz
>     - all/packages/INSTALL
>     - all/packages/posixlock-0.0.1.tgz
>     - all/packages/ruby-1.8.6.tgz              # !!!!!
>     - all/packages/sqlite-2.8.15.tar.gz
>     - all/packages/rubygems-0.9.2.tgz          # !!!!!
>    %
>
> Now I completely understand shipping lockfile, posixlock, arrayfields,
> sqlite-ruby and sqlite-2.8.15 with rq.  But shipping ruby 1.8.6 and rubygems
> 0.9.2 with rq?   I have to know why!

because the typical installation scenario is

   ls /clusterwide/nfs/  # ./bin ./lib ./man for our cluster

   tar xvfz rq-3.0.0.tgz

   cd rq-3.0.0/all

   ./install.sh  /clusterwide/nfs

   export PATH /clusterwide/nfs/bin:$PATH

   # done

say you have 30 machines, like we do.  if you want to use 'gem install rq' to
actually populate an nfs installed location, then you first need ruby and
sqlite installed on that location, then you need LD_RUN_PATH and
LD_LIBRARY_PATH setup too, then you need to install the gem.  so, intead, it's
all in one.  the tar ball with everything is also the gem so, now matter what
you grab, you're good.

the only reason i made a gem is so people could do

   gem install rq --no-wrappers

and play around on the local machine but, in reality, rubygems is rather
useless for managing a cluster - unless you're advanced, like you are.

now, having said that, i'm all ears as to how best manage rq's installation.
here are the issues

first some rubygems ones (arrgghh)

   - users cannot use rubygmes if ANY softlinks exist along the installation
     path - this is __very__ common when a shared nfs location is involved

       http://rubyforge.org/tracker/index.php?func=detail&aid=8811&group_id=126&atid=575

   - wrappers don't work

       http://rubyforge.org/tracker/index.php?func=detail&aid=8819&group_id=126&atid=575
       http://rubyforge.org/tracker/index.php?func=detail&aid=8985&group_id=126&atid=575
       http://rubyforge.org/tracker/index.php?func=detail&aid=8987&group_id=126&atid=575

   - finally, gem cannot be used easily to install a bunch of stuff into
     another location, even though, in theory, it's possible a.k.a
     LD_RUN_PATH/-R encoding.  in otherwords, GEM_HOME isn't that useful

       http://rubyforge.org/tracker/index.php?func=detail&aid=8818&group_id=126&atid=578

   - depdencies don't work correctly when mixing local/remote installs (that's
     why i bundled them)

second, some generic ones

   - people who actually use clusters (scientists) tend not to be too good at
     compiling things, especially into non-standard locations

   - people who use clusters tend to be on 'stable' (crappy) operating systems
     with auto-updates of crappy rubys.  we're still on 1.6.8 with
     enterprise!!!!!!!!  only our newest machines switched.

   - sqlite is a biatch to install. sqlite1,2,3, wtf? the ruby installer loses
     some information during install (i'm forgetting the issue attm).  plus the
     ruby bindings have not remained backward comptible over time.  same goes
     for the sql understood by the api.  thus the choice to bundle it.

so, my thinking was that an rq gem install is really just marketing: download
it, play with it, then you have all the stuff locally to do a __real__ install
onto your cluster without doing another thing.


> And does rq need to do its own cleanup?  There's a 'build' directory in
> the gem which has the dross from building sqlite-2.8.15 and
> sqlite-ruby-1.3.1.

you mean it should blow it away after building?  the only issue with that is
that an error along the way means all the compilation is for nothing (you have
to start over) but that's proably fine.  i'll tweak it

> I still want to create ActiveQueue based upon rq at some point.

yeah, sure.  i'm all ears.  maybe we could get it out before mtnwest
rubyconf!?

thanks a bunch for the feedback.

-a
-- 
be kind whenever possible... it is always possible.
- the dalai lama