Re: RubyGems in Ruby HEAD

On Thu, 2005-09-22 at 02:46 +0900, Hugh Sasse wrote:
> On Thu, 22 Sep 2005, Aredridel wrote:
> 
> >> It seems to me that rubygems is going some of the way towards making
> >> this easier:- gem help unpack gives
> >>
> >> Usage: gem unpack GEMNAME [options]
> >
> > [snip]
> >
> >> What could be added that would facilitate creating package/rpm/???
> >> for those who need to?  Given that we are discussing a TODO list at
> >> the moment...
> >
> > Yes, that is useful -- though thanks to Mauricio Fern叩ndez, the gem
> > format is a tarball and not that hard to unpack anyway.
> >
> > The real trouble comes from gems altering the way ruby works. There's
> > code out there that requires gems, there's code out there that assumes
> > ruby works the way ruby works without gems, so loading gems /at all/
> > will break other things.
> 
> I think we need concrete examples of this.  Not because we don't
> believe you :-), but because we need test cases

I'm trying to come up with one -- the example that came to mind, though
unrelated, is when I tried to override 

> [OT] What is the correct way to markup insertion of a newline when
> quoting someone?

Just do it and try not to break what they meant.

> > And now with some libraries being distributed
> > both only as gems and with hard dependencies on it, that's where life
> > gets tough.
> >
> >>> Indeed.  Note that this is partly due to a different point of view.  For
> >>> Debian we package the libs and apps ourselves.  I hate to say it but
> >>> there is almost no need for RubyGems in our case, whereas it is very
> >>
> >> Except that rubyists expect things to work cross platform.  So how
> >> can we add things to make it easier?
> >
> > Make sure that the semantics of require are the same in both cases.
> > Don't mess with the load_path silently.
> 
> But $: is a VARIABLE, for goodness sake! :-)
> 
> Maybe we should reset it when our work is complete.

Definitely, if it's reset in the course of work being done. Particularly
problematic is unshifing things, because I depend on local override dirs
being before system dirs.

> >>
> >> Is there a limited set of things that you need to patch to do this?
> >> Presumably it would then be possible to integrate this into the gem
> >> command....  Rather like $ patch < boilerplate;
> >
> > Depends on the code. Some gems are trivial.
> >
> > Rails is a couple hours of work to patch. Rubygems dependency it pretty
> > deep there.
> 
> This leaves a broad area to cover.  I don't know how we can meet
> that, even half way if we don't know where the end is.

A good portion of the solution, I think, is to provide and encourage the
use of tools that keep things portable. Give developers the tools to fix
it right.

> > The salient point is 'let "require 'foo'" stay "require 'foo'"' -- that
> > would help most of my repackaging woes too.
> 
> Ruby is open in nature, and redefining methods to do what we want is
> particularly rubyesque.   Maybe we need to restore the old usage
> after we are done....

That helps. Just make code that doesn't grok rubygems work as if it's
not there. Hide it completely when possible.

> > The other assumption -- and a tough problem to address in general -- is
> > where to locate _non-ruby_ data files. In gems, it's
> > File.dirname(__FILE__) or so -- data lives in the ruby datadir.
> 
> Can't this go in the gemspec, then?   And if you want it on a
> shareable filesystem when it isn't then symlink.  What have I missed
> with that though?

That's the problem -- code written with gems in mind will, say, to open
a data file, do File.open(File.join(File.dirname(__FILE__), 'icon.png')
in a gem. And without, might do File.open('/usr/share/pixmaps/icon.png')
-- there's no way to seek out and find the right path at runtime,
without something like a config file or token replacement -- which then
runs into the problem of how to find the config file... a nasty
chicken-and-egg problem that Ruby avoids by compile-time encoding
library paths into the binary. The same sort of solution can be done
with datadirs -- it just isn't (yet?).

> >
> > The FHS that many linux distributions follows says it should
> 
> Which seems to be this:
> http://www.pathname.com/fhs/pub/fhs-2.3.html

Yes, though there's some debate over 2.3 vs 2.1. 2.3 has some silly
rationales that are still under debate.

> > be /usr/share/#{packagename}, or something more generic, if it's a
> 
> Although many GNU packages still put things in /usr/local/bin rather
> than /usr/local/share/bin

Yeah. Version 2.1 of the FHS specified /usr/local/bin for executables
and /usr/local/share for data.

> >>> * Create some generizable installer, maybe assist with Package[2] or
> >>>  come up with something better, definitely useful to have a
> >>>  distutils[1]-alike system in Ruby Core IMO.
> >>
> >> I think this is a big project:  I know of ZIP, tar.gz files, Sun
> >> package files, RPMs, and it seems no-one has found the be all and
> >> end all of installers.  This problem needs to be bounded more, I
> >> think.  If we are aiming for 1.8.4 anyway.
> >
> > What would help more than anything is having an inspectable place to put
> > data files, that works akin to rbconfig. At the moment, the only way to
> 
> Would an entry in gemspec be sufficient?

Only if there's a way to locate the gem spec at runtime -- therein lies
the problem -- how do you find the file that tells you where to find
files? It's a solved problem for a gem, but more problematic in general.
Can we move that into the core (rbconfig.rb?) enough that a program or
library can be robust in respect to its own needs and the admin's
demands?

> > locate data files is by configuration, in every library that uses them.
> > Ruby has no central config file like PHP for this, so every library
> 
> You could scour rbconfig.rb but then
> 
> neelix hgs 36 %> cat CONFIG.rb
> #!/usr/local/bin/ruby -w
> 
> require 'rbconfig'
> 
> puts Config::CONFIG.inspect
> 
> neelix hgs 37 %>
> 
> > hard-codes paths as robustly as the author thinks neccesary, which is
> > never a complete solution.
> >
> > If ruby had a datadir mechanism, and perhaps even a path-searching
> > File.open-alike method that would try each until it found one, the
> 
> require or load ...?


File.open_with_pathlist(list, filename) perhaps

require and load already internally search $: -- they have the mechanism
-- it's data that goes with those libraries that is difficult to locate.

The only reasons this relates to gems is the one gem == one dir
paradigim. Code written assuming gems now has a different assumption
than code written pre-gems does. It actually can assume that the data is
relative to the library in a meaningful sense, and have it work in a
non-trivial number of cases. Just not enough of them, sadly.

> > problem could be neatly solved that would make both gems and non-gems
> > libraries robust, and provide an elegant mechanism that no other
> > language provides natively.
> >
> > It is undoing the assumptions that one gem = one dir that makes
> > repacking them hard. With PLD in particular, we're trying to solve
> 
> PLD isn't Programmable Logic Device, or a Polish distro of GNU/Linux, 
> so what is it?  (They're the first results from Google.)

http://pld-linux.org/ -- essentially, an up-to-date RPM-based distro. 

> 
> > problems with a dual-architecture system like Linux/AMD64 and
> > Linux/Sparc64, where the CPU can run two different kinds of code. Having
> > things split up in FHS fashion lets one install packages from both
> > architectures, allowing both 32-bit and 64-bit versions of things to
> 
> Or you could do it by sharing different parts of the hierarchy?

There's /usr/lib and /usr/lib64 on those architectures -- the
architecture-specific DSOs go in those, respectively. That directly goes
against either DRY or One Gem = One Dir.

> > run. This is entirely out of rubygems scope, so what I'd hope to see is
> > simply the tools to let this be possible. A good datadir mechanism, and
> > just trying hard not to make assumptions about the relative locations of
> > files would help immensely. If this gets added to the ruby stdlib or
> 
> That would seem to require us to change require and/or $: to be as
> flexible as that?  Does any of the above help, though.

Not require and $:, but to make an analogue for data files (and as a
third set, config files)

> > core, and gem authors were encouraged to use it, then my work would be
> > much easier -- debian's too -- and we'd make more progress on making the
> > ruby packages truly friendly, easy to use and complete.
> >
> > Aredridel
> >
> Much of what you mean is clearer to me now. Thank you.

Glad to help shed light. This is actually fun to tackle as an
engineering problem, I think. 

Aredridel