Re: RubyGems TODO

On 9/22/05, ES <[e-mail]> wrote:
> Jim Weirich wrote:
> >>>On 21-Sep-05, at 7:17 PM, why the lucky stiff wrote:
> >
> >
> >>>>So, what needs to happen with RubyGems to neutralize everybody?
> >
> > [... items 1..4 elided ...]
> >
> > Chad Fowler said:
> >
> >
> >>>Thanks, Why.  I'd like to add (at least, for now):
> >
> > [... items 5..7 elided ...]
> >
> > This is how I would sort the Why/Chad todo list:
> >
> > * Things that are Urgent and need done immediately:
> >
> > (5) Incremental Source Index downloads.
> >
> >     I understand that RubyForge is currently having issues with the
> >     large monolithic index file, which makes this a high priority
> >     item.  I meant to deal with this earlier, but got sidetracked with
> >     life.
>
> I had, at some point or another, envisioned a transparent distributed
> system where a given server not only delegates out downloads but the
> very indexing itself to any other servers it knows about. I can not
> recall if I wrote any code for this but it is quite doubtable as I
> usually would do no such thing :)
>
> > * Things that should be done for a 1.8.4 merge:
> >
> > (1) Data dir support.
> >
> >     This is really more of an issue with the Ruby environment rather
> >     than with RubyGems itself.  The reason it tends to come up in
> >     RubyGems context is that it is gems that people want to package.
> >
> >     I have some vague suggestions in this area, but the discussion
> >     should probably move to a different thread.  I'll post more when I
> >     organize some thoughts.
> >
> > (3) Semantics of require and needing to require rubygems.
> >
> >     In the absence of explicit require_gem commands, RubyGems only
> >     interfers with the require process when a file cannot be found by
> >     the normal require process.  In order to do this, it currently
> >     aliases Kernel.require and traps load errors.
> >
> >     I would like to see two things happen as RubyGems becomes part of
> >     the core.  (1) a general hook for require LoadError handlers be
> >     implemented, and (2) RubyGems become a default handler in that
> >     hook.  This keeps the impact of RubyGems at an absolute minimum
> >     until the moment it is needed, but users can get to gems without
> >     doing explicit require 'rubygems', -rubygems or RUBYOPT hacks.
> >
> > (7) Deprecate require_gem and start using use_gem (or an appropriate
> >     alias).
> >
> > * Good things that are necessarily needed for 1.8.4 merging.
> >
> > (6) Local / Remote installer unification.
> >
> >     This is a major area of functionality that really needs to be
> >     addressed.  Fixing this will fix a number of inconsistancies with
> >     the way local gems are handled.
> >
> >     Chad would like to see this before 1.8.4, and I agree; but this
> >     change is big enough that it will probably need some extensive
> >     burn in time, so I am hesitant to insist that it be prior to
> >     1.8.4.
> >
> > (2) gem setup / gem lint
> >
> >     Excellent ideas.
> >
> > * Vague items that need more definition
> >
> > (4) Distro compatibility.
> >
> >     I think the DATADIR issue will go a long way to resolving some of
> >     these issues.  But I would like to hear specifics in this area.
> >     Although RubyGems is primarily a *Ruby* packaging system (and so
> >     our priorities are set accordingly), we certainly don't want to
> >     make it any harder for repackagers and specific suggestions are
> >     always welcome.
> >
>


This might be a good time to expand on my thoughts. I think it is a
large change but fortunately I think it still leverages the current
technology:

I mentioned that I would like to see three main things in my last
post: smart aliases, user scripts, and most importantly a modular gem
database/setup (I can't think of a good name for this yet).

#1

Smart aliases would allow the one to have special names applied to a
specific gem. In an example you might see some gem xyz with available
versions 1, 2, and 3 (possibly from specific sources too). You could
alias xyz-1 to the gem that corresponds to that specific version. You
might also want to link to a specific minor version. This is not that
powerful by itself but it does allow a few nice things. One is the
fact that someone who is picky about naming conventions can now
specifically fix this problem. Another would be the ability to fix bad
code that was written around a case insensitive file system. And last
but not least there would be a way to swap out libraries that
implement a standard API (like loggers) by using the generic name in
code rather than having config files to edit (it doesn't change the
fact that is still has to be configured but, IMHO, i think it is more
elegant -- especially in combination with the other proposals).

Someone mentioned namespace collisions above. This would be a problem
if we were both reckless and did nothing to support handling of
collisions. This is still a problem even without aliasing. Meta-gems
would have the same possible clashes as could normal gems even
(multiple sources). Solving this issue would benefit gems regardless
of whether or not any of my ideas are incorporated. I think this could
be fixed one way by giving priority to an alias over a gem and having
each database be restricted to one source (more on that) along with
the ability to have a combination of databases available which could
be ordered for search priority also. Just a thought.

#2

User scripts are a more exotic tool. They allow a range of
customization that I think would make quite a few people happy. User
scripts would be ruby scripts attached to a gem database that would
hook into specific events (pre-, post-, and wrap- styles). These would
allow packagers a little more control over the gem. If designed
properly it could control things like when, where, and how a gem is
compiled, linked, installed, tested, and documented. They would also
allow for testing the environment. The database concept, which I will
hit on soon, could use these when deployed in specific environments.
For example, if a certain database needs a specific gem installed on
mswin32 platforms only then it could check the environment and include
or exclude that specific gem upon analysis. Other users would not be
stuck with an extra gem or even having to put up with a failure when
installing platform specific gems on unsupported platforms. User
scripts would offer enough open-ended-ness that I am sure there are
plenty of more useful applications of this feature. This is not
particularly dependent on the other two feature but it does alleviate
some of the short comings.

#3

Gem databases. I don't know what to name this really because it
wouldn't actually hold the final gems (I guess there could be an
option to bundle gems with it internally). What it would hold is
information on what gems are installed, what versions, what aliases,
what user scripts, what source to look for non local gems (ones that
aren't bundled)... any meta-info needed to reconstruct the specific
gem environment as accurately as possible. When running rubygems could
check the simple configuration available. This configuration would
tell it which databases to check and in which order. It could
optionally give placement to a database relative to the standard ruby
lib, platform, and site directories. It might be nice if the gem tool
could help generate or change these files from the CLI. These gem
databases would be really cool if they could be packaged back into a
gem. :) gem install rpa

Now the applications of this are really cool... here are a few example
scenarios:

I have an application that works under setup option A and B. I want to
ensure that my application works consistently across both. Good
testing is the quick answer to this but how do I quickly switch
between the setups? Normally this would be non-trivial without
hardcoding versions all over the place in the tests. However if I
could just tell rubygems, via some environment var, what databases to
use then I could construct these once and be done.

Even better, the database files could be version controlled along with
the rest of the application! Now you might be able to really roll back
to an old version immediately rather than rolling code then spending
time trying to replicate your old environment.

Now add to that. One of your users is having trouble and you suspect
that it is related to their environment. You could just ask for his
"database_description_file". bam. You've now got all the information
you need. Are his user scripts messing it up? Is it another gem
interfering? A specific combination of library versions? Well you at
least have a place to start answering those types of questions from.

Another capability of a database system would be to allow things like
RPA to provide a very specific set of libraries with specific
versions. User scripts could also weed out dangerous combinations to
help spot stability issues.

I think that wraps up my thought right now. I am sure I forgot
something but this should give you a taste of what angle I am coming
from. A few notes to finish up that I thought of but didn't want to
clutter my original idea with..

Database description files could contain more database description
files. The final atomic file could be what we currently call a gem.
The rubygems configuration file could also be a database description
itself. the ruby standard library and other directories provided as
_special_ types of databases that may be included. This might be nice
for systems that don't want to stick with the standard ruby layout
(again user scripts would make this option much more potent).

Also, before you reply to me, I use the terms database and database
description loosely right now. I have not really come up with a name.
Perhaps I should just call them gems still if we include my last
thoughts.

Brian.