Re: Core support for Gems, and namespace

On Wed, 28 Jul 2004 23:00:34 +0900, Gavin Sinclair
<[e-mail]> wrote:
> On Wednesday, July 28, 2004, 9:51:42 PM, Austin wrote:
>> 1. The binary and site_lib stubs breaks the versioning concept,
>> and
> That's mostly false. The binary stub loads the most recently
> installed version of the app. This is not an ideal situation,
> perhaps, but it's not terrible either. (Chad has thought about
> adding to the stub so you can do, for instance, "rake --version
> 0.3.2 ...".)
> 
> The site_ruby stubs, however, do not break anything. They provide
> a mechanism by which you can access a gem library through the
> normal 'require' mechanism. 'require' knows nothing about
> versions, so nothing is broken. This is meant to be a convenience.
> If you choose to use 'require', you're getting exactly what you
> want.

Um, you just made my point for me on both counts. The convenience
stubs in site_ruby *do* break the versioning features of RubyGems.
Don't get me wrong, they will help increase the usability of
RubyGems, but they also remove the version control capabilities
through their use.

The binary stub does load the most recently installed version. It
loads the most current version, but that's wrong, as the ldiff that
gets installed with diff-lcs-1.0.1 is broken. Yes, I did a test as a
deliberate borking of things, but if one installs 1.0.2 and then
installs 1.0.1, require_gem by itself will load the 1.0.2 version,
but the ldiff installed will use the most recently installed
version.

However, adding anything to the stub may actually interfere with my
program's options. This should be avoided. One way to do this is to
make it so that ldiff-1.0.1 and ldiff-1.0.2 are installed as stubs
and then handling things *that* way. (On filesystems that can handle
linking, ln -s ldiff-1.0.2 ldiff; otherwise ldiff.bat would point to
the most recent version).

>> there are other problems with the use of them from the developer/
>> deployment perspective. (I had to change ldiff in diff-lcs to not
>> use the "if __FILE__ == $0" common feature. It is expected that
>> the loaded file will return an error code -- and not all "bin"
>> files are written that way.)
> I doubt there's a good reason to use "if __FILE__ == $0" in an
> application. An app is, after all, explicitly designed to be run.
> That idiom makes sense for libraries, not (AFAICS) apps.

*shrug* I try to write my apps in such a way that they could be
wrapped by another app. Thus, I had used the execution block as:

  if __FILE__ == $0
    exit Diff::diffprog(ARGV, $stdout, $stderr)
  end

or something like that. Thus, one could have theoretically called
Diff::diffprog with different output items and then done some
massaging. Because of the way that RubyGems works -- by installing a
stub rather than letting me specify a file that can be used in place
of a stub -- I don't have a lot of choice. Yes, I can rework how I
have created my binary file so that it requires a library containing
the ldiff program and then the binary just calls "exit
Diff::diffprog(ARGV, $stdout, $stderr)", but my point here is that
the way that I had written the program was not supported -- and to
make it work, I have to go to a significant amount of extra work.

>> 2. The disconnect between "require_gem 'diff-lcs'" and the nominal
>> "require 'diff/lcs'" is more than annoying, it's problematic.
> It might be a good idea to name your package 'diff-lcs', then. I know
> I'm using a stupid argument here, as RubyGems has certainly robbed you
> of some flexibility in this respect. If it were my library, I'd
> probably choose that name anyway. (Hey, this thread is partially about
> namespaces, right?)

Okay; I think you're not understanding what I'm saying here. To
clarify: the package is diff-lcs. The gem names are diff-lcs-1.0.1
and diff-lcs-1.0.2 and will be diff-lcs-1.1.0. To use a specific
version, rather than doing

  require "diff/lcs"

which is the preferred mechanism, users must remember the package
name:

  require_gem "diff-lcs"

Ideally, there should be a way to map the require specification to
the package name, and it's the gem's responsibility to do this, IMO.
What I *want* is:

  require_gem "diff/lcs", "= 1.0.2"

to be interpreted as if it were:

  require_gem "diff-lcs", "= 1.0.2"

If that were to happen, then there'd be absolutely no reason for me
to not use require_gem in my top-level requires.

>> It means that I can really only provide one loading mechanism for a
>> given library, whereas I might theoretically have more than one.
>> [...]
> Well, what you have to do is this:
> 
> require_gem 'diff-lcs'
> require 'diff/lcs/array'
> 
> You are obviously relying on the site_ruby stub, [...]

Not quite. I have designed my library. I should not have to redesign
it in any way simply because of the packaging mechanism. As RubyGems
stands now, I have to do a bit *too much* work. (In fact,
diff-lcs-1.0.1 existed entirely because of problems with RubyGems.)
The ability to have multiple stubs will help significantly. The
ability to specify gem-maps (e.g., require_gem "diff/lcs/array")
would be useful, too. That still leaves me with the problem, though,
that I may have potentially two different versions I have to
distribute if I work that way.

Many of the issues I have will be eliminated if RubyGems is
integrated into Ruby directly, as I won't have to use a separate
command to require a gem.

> So now (with RubyGems CVS) you can do this:
> 
> require_gem 'extensions/string'
> 
> which is exactly shorthand for the above two lines. However, this
> only "works" if your gem is named the same as your project directory.

> What remains "unsolved" here is your desire to name a gem with a slash
> in it. That's not allowed. Perhaps it could be. Hopefully Chad and
> Rich will think about it.

No, you're not understanding. I don't want a gem with a slash. My
gem is diff-lcs. My *package* is diff-lcs. But the *namespace* is
Diff::LCS, therefore the require is of diff/lcs. I *do* keep the
directory namespace and the Ruby namespace in sync inasmuch as is
possible. I want what you're talking about above, but I don't want
it tied to the gem's name. That is, I don't want:

  require_gem "diff-lcs/lcs"

>> 3. Certain types of changes should not be permitted to exist
>> simultaneously (e.g., diff-lcs-1.0.1 and diff-lcs-1.0.2), whereas
>> others should be acceptable (e.g., the upcoming diff-lcs-1.1.0 vs
>> diff-lcs-1.0.2). This can't be solved by RubyGems alone, but
>> there is no way for me, as a developer, to mark older gems as
>> obsolete and incompatible with earlier versions.
> Why shouldn't they be allowed to co-exist? If someone writes a
> really cool app that depends on diff-lcs-1.0.1, and they've thoroughly
> tested it, then what's wrong with that? It may not be the way I would
> package it, but if they're happy with 1.0.1, why should they be forced
> into an upgrade cycle controlled by you?

For the same reason that security fixes should obsolete broken
libraries. You don't want broken libraries in use by anyone. (And
the bug in diff-lcs-1.0.1 essentially makes it unusable; there is no
way to actually build an app that depends on certain features
because the breaks are internal and prohibit the emission of the
information required.) Remember, I don't have a problem with 1.1.0
and 1.0.2 coexisting; however, 1.0.1 is something that should (1) be
removed from the gem repository and (2) be removed when upgrading to
1.0.2. Even though I used the example of:

  require "diff/lcs", "= 1.0.2"

I don't actually want that; I want:

  require "diff/lcs", "= 1.0"

which would allow any version of 1.0.*. This would mean that I would
have to absolutely abide by the implicit contract that 3rd-level
version numbers mean (e.g., no API changes, only bugfixes), but I
personally don't have a problem with that. IMO, users should not be
able to specify all three levels of versions (1.0.2). See, I would
argue against:

  require "diff/lcs", ">= 1.0"

...because there may be some minor incompatible changes from 1.0 to
1.1.

>> 4. The general case will be one version of a given library. Changing
>> between the standard mechanism and the versioning gem mechanism
>> is *painful* for library and application writers.
> That's your experience, and, believe it or not, mine. The release of
> extensions.gem is seriously delayed because I want to fix all possible
> RubyGems problems that stand in its way, however slightly :)  (That,
> and a million other things to do.)

> However, there are lots of people who are creating gems without any
> reports of pain. Using Rake makes it easy to produce tarballs and
> gems at the same time.

I haven't yet found something where I will use rake, though. I may
automate what I'm doing, but it's not a done deal, yet. If Alan and
I *do* produce a gem of Ruwiki, it will be automated because it will
be pulling specific files from multiple directories.

However, I don't forsee any reason for me to *not* use the standard
require mechanism as opposed to require_gem.

[...]

>> IMO, an rpa-base package is more likely to be usable, but even
>> then, Ruwiki may be a special case. Ruwiki is now going to be
>> depending on Diff::LCS, but we are going to package the portions
>> of the required version inside of Ruwiki packages as we have been
>> doing with Algorithm::Diff. This is easier for us as a standalone
>> application than relying on any external packaging mechanism.
> It sounds like a gem-only version of Ruwiki would run like a
> dream. Not that I'm suggesting that :)

Mmmm. I'm not convinced. One concern is that ruwiki_servlet
currently gets its data path as "./data/". The stub that would be
installed would assume that to be "basename `which ruby`" or where
the program was run from. The data, however, won't be there, but in
the gem directory. You can specify your own directory, but there's
no easy way to get that data "local" (maybe we should provide one,
but that again leads toward programming specifically to support the
packaging format!). There are a couple of other issues, including
data migration.

It's more complex than it might seem, and a gem would make sense
(IMO) only in a standalone computer situation. It would not make
sense otherwise.

>> 5. I haven't used require_gem. [...] I actually spend time making
>> sure that my libraries are laid out intelligently, and RubyGems
>> doesn't let me handle that.
> I don't doubt it, but what do you mean by laying them out
> intelligently?

This includes things like diff/lcs/array and diff/lcs/string.

>> I agree with Luke that a 'require "foo/dependencies"' is not the
>> right way to do this, as it raises the dependencies from the
>> level that they probably belong to (e.g., Ruwiki's dependency on
>> Diff::LCS is, in fact, only in the flatfile backend -- not in the
>> program itself, yet).
> There might be some cases where it's not the best way to go, but
> in many cases, I think it would be just the tonic. And there's
> nothing stopping you from having a ruwiki/dependencies.rb *and* a
> ruwiki/backend/flatfile/dependencies.rb, if the scale of the
> project demands it.

It feels like a horrid hacky kludge, and I don't care for it at all.

>> The concepts behind RubyGems are wonderful. There are a number of
>> great things that the RubyGems team has done. I'm just not sure
>> that RubyGems versioning is ideal. I don't have an alternative
>> solution, just a vague dissatisfaction with what's there now.
> Fair enough. I greatly appreciate the comments. It's certainly
> true that there's a philosophy behind RubyGems that must be
> embraced if one is to enjoy using it. RPA has a different
> philosophy, but I don't see them as mutually exclusive. It
> oversimplifies them both to call them mere "package managers".
> RubyGems is about the Ruby runtime as much as package management.
> And RPA, IIUC, is about the human processes in building a complete
> platform - an admirable goal.

I have a vague unease about RPA, too, as I don't necessarily think
that "rpa-set" versioning is the way to handle some of the concerns,
either. I do like the idea of setting the required versions *first*,
but I'm not sure that sets *per se* make sense. Like I said, as a
consumer of RubyGems, I don't *like* what it's forced me to do with
my libraries so far. I don't have a better solution, and I'm too
busy with other stuff to work on one. I think part of the problem is
that they both have parts which feel "inhumane."

-austin
-- 
Austin Ziegler * [e-mail]
               * Alternate: [e-mail]