Re: RubyGems TODO

On 9/22/05, Brian Mitchell <[e-mail]> wrote:
> On 9/22/05, ES <[e-mail]> wrote:
> > Jim Weirich wrote:
> > >>>On 21-Sep-05, at 7:17 PM, why the lucky stiff wrote:
> > >
> > >
> > >>>>So, what needs to happen with RubyGems to neutralize everybody?
> > >
> > > [... items 1..4 elided ...]
> > >
> > > Chad Fowler said:
> > >
> > >
> > >>>Thanks, Why.  I'd like to add (at least, for now):
> > >
> > > [... items 5..7 elided ...]
> > >
> > > This is how I would sort the Why/Chad todo list:
> > >
> > > * Things that are Urgent and need done immediately:
> > >
> > > (5) Incremental Source Index downloads.
> > >
> > >     I understand that RubyForge is currently having issues with the
> > >     large monolithic index file, which makes this a high priority
> > >     item.  I meant to deal with this earlier, but got sidetracked with
> > >     life.
> >
> > I had, at some point or another, envisioned a transparent distributed
> > system where a given server not only delegates out downloads but the
> > very indexing itself to any other servers it knows about. I can not
> > recall if I wrote any code for this but it is quite doubtable as I
> > usually would do no such thing :)
> >
> > > * Things that should be done for a 1.8.4 merge:
> > >
> > > (1) Data dir support.
> > >
> > >     This is really more of an issue with the Ruby environment rather
> > >     than with RubyGems itself.  The reason it tends to come up in
> > >     RubyGems context is that it is gems that people want to package.
> > >
> > >     I have some vague suggestions in this area, but the discussion
> > >     should probably move to a different thread.  I'll post more when I
> > >     organize some thoughts.
> > >
> > > (3) Semantics of require and needing to require rubygems.
> > >
> > >     In the absence of explicit require_gem commands, RubyGems only
> > >     interfers with the require process when a file cannot be found by
> > >     the normal require process.  In order to do this, it currently
> > >     aliases Kernel.require and traps load errors.
> > >
> > >     I would like to see two things happen as RubyGems becomes part of
> > >     the core.  (1) a general hook for require LoadError handlers be
> > >     implemented, and (2) RubyGems become a default handler in that
> > >     hook.  This keeps the impact of RubyGems at an absolute minimum
> > >     until the moment it is needed, but users can get to gems without
> > >     doing explicit require 'rubygems', -rubygems or RUBYOPT hacks.
> > >
> > > (7) Deprecate require_gem and start using use_gem (or an appropriate
> > >     alias).
> > >
> > > * Good things that are necessarily needed for 1.8.4 merging.
> > >
> > > (6) Local / Remote installer unification.
> > >
> > >     This is a major area of functionality that really needs to be
> > >     addressed.  Fixing this will fix a number of inconsistancies with
> > >     the way local gems are handled.
> > >
> > >     Chad would like to see this before 1.8.4, and I agree; but this
> > >     change is big enough that it will probably need some extensive
> > >     burn in time, so I am hesitant to insist that it be prior to
> > >     1.8.4.
> > >
> > > (2) gem setup / gem lint
> > >
> > >     Excellent ideas.
> > >
> > > * Vague items that need more definition
> > >
> > > (4) Distro compatibility.
> > >
> > >     I think the DATADIR issue will go a long way to resolving some of
> > >     these issues.  But I would like to hear specifics in this area.
> > >     Although RubyGems is primarily a *Ruby* packaging system (and so
> > >     our priorities are set accordingly), we certainly don't want to
> > >     make it any harder for repackagers and specific suggestions are
> > >     always welcome.
> > >
> >
>
>
> This might be a good time to expand on my thoughts. I think it is a
> large change but fortunately I think it still leverages the current
> technology:
>
> I mentioned that I would like to see three main things in my last
> post: smart aliases, user scripts, and most importantly a modular gem
> database/setup (I can't think of a good name for this yet).
>
> #1
>
> Smart aliases would allow the one to have special names applied to a
> specific gem. In an example you might see some gem xyz with available
> versions 1, 2, and 3 (possibly from specific sources too). You could
> alias xyz-1 to the gem that corresponds to that specific version. You
> might also want to link to a specific minor version. This is not that
> powerful by itself but it does allow a few nice things. One is the
> fact that someone who is picky about naming conventions can now
> specifically fix this problem. Another would be the ability to fix bad
> code that was written around a case insensitive file system. And last
> but not least there would be a way to swap out libraries that
> implement a standard API (like loggers) by using the generic name in
> code rather than having config files to edit (it doesn't change the
> fact that is still has to be configured but, IMHO, i think it is more
> elegant -- especially in combination with the other proposals).
>
> Someone mentioned namespace collisions above. This would be a problem
> if we were both reckless and did nothing to support handling of
> collisions. This is still a problem even without aliasing. Meta-gems
> would have the same possible clashes as could normal gems even
> (multiple sources). Solving this issue would benefit gems regardless
> of whether or not any of my ideas are incorporated. I think this could
> be fixed one way by giving priority to an alias over a gem and having
> each database be restricted to one source (more on that) along with
> the ability to have a combination of databases available which could
> be ordered for search priority also. Just a thought.
>
> #2
>
> User scripts are a more exotic tool. They allow a range of
> customization that I think would make quite a few people happy. User
> scripts would be ruby scripts attached to a gem database that would
> hook into specific events (pre-, post-, and wrap- styles). These would
> allow packagers a little more control over the gem. If designed
> properly it could control things like when, where, and how a gem is
> compiled, linked, installed, tested, and documented. They would also
> allow for testing the environment. The database concept, which I will
> hit on soon, could use these when deployed in specific environments.
> For example, if a certain database needs a specific gem installed on
> mswin32 platforms only then it could check the environment and include
> or exclude that specific gem upon analysis. Other users would not be
> stuck with an extra gem or even having to put up with a failure when
> installing platform specific gems on unsupported platforms. User
> scripts would offer enough open-ended-ness that I am sure there are
> plenty of more useful applications of this feature. This is not
> particularly dependent on the other two feature but it does alleviate
> some of the short comings.
>
> #3
>
> Gem databases. I don't know what to name this really because it
> wouldn't actually hold the final gems (I guess there could be an
> option to bundle gems with it internally). What it would hold is
> information on what gems are installed, what versions, what aliases,
> what user scripts, what source to look for non local gems (ones that
> aren't bundled)... any meta-info needed to reconstruct the specific
> gem environment as accurately as possible. When running rubygems could
> check the simple configuration available. This configuration would
> tell it which databases to check and in which order. It could
> optionally give placement to a database relative to the standard ruby
> lib, platform, and site directories. It might be nice if the gem tool
> could help generate or change these files from the CLI. These gem
> databases would be really cool if they could be packaged back into a
> gem. :) gem install rpa
>
> Now the applications of this are really cool... here are a few example
> scenarios:
>
> I have an application that works under setup option A and B. I want to
> ensure that my application works consistently across both. Good
> testing is the quick answer to this but how do I quickly switch
> between the setups? Normally this would be non-trivial without
> hardcoding versions all over the place in the tests. However if I
> could just tell rubygems, via some environment var, what databases to
> use then I could construct these once and be done.
>
> Even better, the database files could be version controlled along with
> the rest of the application! Now you might be able to really roll back
> to an old version immediately rather than rolling code then spending
> time trying to replicate your old environment.
>
> Now add to that. One of your users is having trouble and you suspect
> that it is related to their environment. You could just ask for his
> "database_description_file". bam. You've now got all the information
> you need. Are his user scripts messing it up? Is it another gem
> interfering? A specific combination of library versions? Well you at
> least have a place to start answering those types of questions from.
>
> Another capability of a database system would be to allow things like
> RPA to provide a very specific set of libraries with specific
> versions. User scripts could also weed out dangerous combinations to
> help spot stability issues.
>
> I think that wraps up my thought right now. I am sure I forgot
> something but this should give you a taste of what angle I am coming
> from. A few notes to finish up that I thought of but didn't want to
> clutter my original idea with..
>
> Database description files could contain more database description
> files. The final atomic file could be what we currently call a gem.
> The rubygems configuration file could also be a database description
> itself. the ruby standard library and other directories provided as
> _special_ types of databases that may be included. This might be nice
> for systems that don't want to stick with the standard ruby layout
> (again user scripts would make this option much more potent).
>
> Also, before you reply to me, I use the terms database and database
> description loosely right now. I have not really come up with a name.
> Perhaps I should just call them gems still if we include my last
> thoughts.
>

One more thing. I forgot to mention how busy I am right now ;)
actually, I wouldn't mind volunteering some time around rubyconf and
after to help get this done. So if time is an issue, hopefully we can
fix this by getting enough people helping out.

Brian.