Re: Core support for Gems, and namespace

On Wednesday, July 28, 2004, 9:51:42 PM, Austin wrote:

> On Wed, 28 Jul 2004 11:29:53 +0900, Gavin Sinclair
> <[e-mail]> wrote:
>> On Wednesday, July 28, 2004, 3:23:46 AM, Austin wrote:
>> [long and interesting post snipped because there's not one
>> particular part I wish to reply to]
>> 
>> Austin,
>> 
>> I don't want to ram library versioning down anyone's throat, but
>> on the one hand, you say you don't like the fact that RubyGems
>> bothers with versioning (it's overengineering, and you don't see a
>> use for it), and on the other hand, you point out two very good
>> reasons to have it: allowing apps/libraries to cope with changes
>> to APIs in other libraries; and avoiding "CPAN upgrade hell".
>> 
>> Since there are two good reasons to tackle versioning, why not
>> embrace it?

> Well, there's a few problems.

> 1. The binary and site_lib stubs breaks the versioning concept, and

That's mostly false.  The binary stub loads the most recently
installed version of the app.  This is not an ideal situation,
perhaps, but it's not terrible either.  (Chad has thought about
adding to the stub so you can do, for instance,
"rake --version 0.3.2 ...".)

The site_ruby stubs, however, do not break anything.  They provide a
mechanism by which you can access a gem library through the normal
'require' mechanism.  'require' knows nothing about versions, so
nothing is broken.  This is meant to be a convenience.  If you choose
to use 'require', you're getting exactly what you want.

>    there are other problems with the use of them from the developer/
>    deployment perspective. (I had to change ldiff in diff-lcs to not
>    use the "if __FILE__ == $0" common feature. It is expected that
>    the loaded file will return an error code -- and not all "bin"
>    files are written that way.)

I doubt there's a good reason to use "if __FILE__ == $0" in an
application.  An app is, after all, explicitly designed to be run.
That idiom makes sense for libraries, not (AFAICS) apps.

> 2. The disconnect between "require_gem 'diff-lcs'" and the nominal
>    "require 'diff/lcs'" is more than annoying, it's problematic.

It might be a good idea to name your package 'diff-lcs', then.  I know
I'm using a stupid argument here, as RubyGems has certainly robbed you
of some flexibility in this respect.  If it were my library, I'd
probably choose that name anyway.  (Hey, this thread is partially about
namespaces, right?)

>    It means that I can really only provide one loading mechanism for a
>    given library, whereas I might theoretically have more than one.
>    This is, in fact, a possibility with Diff::LCS; if you are
>    working with Arrays extensively, you can simply require
>    "diff/lcs/array" and it will require everything else. Now, I have
>    to do "require 'diff/lcs'" and "require 'diff/lcs/array'"
>    separately. In this way, the gem versioning has explicitly broken
>    a desired interface.

Well, what you have to do is this:

  require_gem 'diff-lcs'
  require 'diff/lcs/array'

You are obviously relying on the site_ruby stub, in which case you
have stumbled across a problem I've found as well: only one (in the
normal case) library stub gets installed.  I plan to add to the
gemspec the ability to list the stubs you want installed.  That would
solve your problem, I think.  At the moment, a 'diff/lcs.rb' stub is
installed; in the future, possibly, a 'diff/lcs/array.rb' stub will be
installed as well.

Another thing you'd be interested in is this: with my extensions
project, for which I'm preparing a gem, there is no autorequire, since
that would be inappropriate.  But I don't want to have to do this:

  require_gem 'extensions'
  require 'extensions/string'

So now (with RubyGems CVS) you can do this:

  require_gem 'extensions/string'

which is exactly shorthand for the above two lines.  However, this
only "works" if your gem is named the same as your project directory.

What remains "unsolved" here is your desire to name a gem with a slash
in it.  That's not allowed.  Perhaps it could be.  Hopefully Chad and
Rich will think about it.

> 3. Certain types of changes should not be permitted to exist
>    simultaneously (e.g., diff-lcs-1.0.1 and diff-lcs-1.0.2), whereas
>    others should be acceptable (e.g., the upcoming diff-lcs-1.1.0 vs
>    diff-lcs-1.0.2). This can't be solved by RubyGems alone, but
>    there is no way for me, as a developer, to mark older gems as
>    obsolete and incompatible with earlier versions.

Why shouldn't they be allowed to co-exist?  If someone writes a
really cool app that depends on diff-lcs-1.0.1, and they've thoroughly
tested it, then what's wrong with that?  It may not be the way I would
package it, but if they're happy with 1.0.1, why should they be forced
into an upgrade cycle controlled by you?

I don't think it's to anybody's detriment that older gems be lying
around.  Most people would try to get their code running with the
latest stuff anyway.

This is not to say I think there are no problems with the RubyGems
approach in this area.

> 4. The general case will be one version of a given library. Changing
>    between the standard mechanism and the versioning gem mechanism
>    is *painful* for library and application writers.

That's your experience, and, believe it or not, mine.  The release of
extensions.gem is seriously delayed because I want to fix all possible
RubyGems problems that stand in its way, however slightly :)  (That,
and a million other things to do.)

However, there are lots of people who are creating gems without any
reports of pain.  Using Rake makes it easy to produce tarballs and
gems at the same time.

The thing to remember is that for the most part, RubyGems adds, not
takes away.  Without RubyGems, your dependencies are done with
'require', i.e. a blind drunk stab in the dark hoping that xyz is
installed.  And if they are installed, you hope it's a compatible
version.  So I don't see what can be so painful about using RubyGems
capability, except that RubyGems itself becomes a dependency.  Well,
it never killed anyone to have RubyGems installed.

And if you program in a way that supports gem installations or
traditional installations, you fall back to 'require'.

I have a minor project on the boil that I plan to release as gem-only,
unless there are specific requests otherwise.  We all have a
conservative instinct that "we have to support the standard
installation mechanism", and it's not unjustified.  RubyGems is still
alpha.  But it's an instint that should be challenged sooner or later.

>    The only way to
>    take advantage of it is to package your application as a gem, and
>    that's not always appropriate. In fact, Alan and I are looking at
>    packaging Ruwiki as a Gem, and I have concerns about the
>    appropriateness of that.

In particular?

>    IMO, an rpa-base package is more likely
>    to be usable, but even then, Ruwiki may be a special case. Ruwiki
>    is now going to be depending on Diff::LCS, but we are going to
>    package the portions of the required version inside of Ruwiki
>    packages as we have been doing with Algorithm::Diff. This is
>    easier for us as a standalone application than relying on any
>    external packaging mechanism.

It sounds like a gem-only version of Ruwiki would run like a dream.
Not that I'm suggesting that :)

> 5. I haven't used require_gem. Ever. At this point, because the
>    require target is different and I don't have any libraries that
>    actually require specific versions of packages *and* I don't want
>    to require users actually have to have RubyGems installed, I
>    don't actually see using it. I only did some changes to ldiff in
>    Diff::LCS to better match RubyGems capabilities. I don't
>    personally ever see using the require/rescue form. I actually
>    spend time making sure that my libraries are laid out
>    intelligently, and RubyGems doesn't let me handle that.

I don't doubt it, but what do you mean by laying them out
intelligently?

>    I agree
>    with Luke that a 'require "foo/dependencies"' is not the right
>    way to do this, as it raises the dependencies from the level that
>    they probably belong to (e.g., Ruwiki's dependency on Diff::LCS
>    is, in fact, only in the flatfile backend -- not in the program
>    itselff, yet).

There might be some cases where it's not the best way to go, but in
many cases, I think it would be just the tonic.  And there's nothing
stopping you from having a ruwiki/dependencies.rb *and* a
ruwiki/backend/flatfile/dependencies.rb, if the scale of the project
demands it.

> The concepts behind RubyGems are wonderful. There are a number of
> great things that the RubyGems team has done. I'm just not sure that
> RubyGems versioning is ideal. I don't have an alternative solution,
> just a vague dissatisfaction with what's there now.

Fair enough.  I greatly appreciate the comments.  It's certainly true
that there's a philosophy behind RubyGems that must be embraced if one
is to enjoy using it.  RPA has a different philosophy, but I don't see
them as mutually exclusive.  It oversimplifies them both to call them
mere "package managers".  RubyGems is about the Ruby runtime as much
as package management.  And RPA, IIUC, is about the human processes in
building a complete platform - an admirable goal.

Cheers,
Gavin