Re: RubyGems in Ruby HEAD

On 9/21/05, Paul van Tilburg <[e-mail]> wrote:
> On Wed, Sep 21, 2005 at 03:28:01AM +0900, Austin Ziegler wrote:
>> On 9/20/05, Mauricio Fern疣dez <[e-mail]> wrote:
>>> Right now, RubyGems represents a step backwards relative to Minero
>>> Aoki's setup.rb in many regards as far as repackagers are concerned.
>> To be honest, this is one of my *least* worries. I know, Mauricio,
>> that it matters greatly to you, but I find that RubyGems has solved
>> things in a manner similar to stow.
> But it is our most important worry! As Debian developer this would
> increase my amount of work for packaging _and_ maintaining each Ruby
> library/app. Although this trend is more related to RubyGems
> popularity gain than the actual merge.

You know what, though? I don't care. Sorry, but I don't. If you insist
upon repackaging -- in a heavyhanded way -- gems, then I don't care what
sort of work you have to do. Are there things we can do to help this?
Sure, but the reality is that I do not believe that RubyGems should
pander to the various repackaging schemes if it harms the RubyGems core
functionality. I again repeat the point that I made about the FreeBSD
ports tree managing the process much easier than I'm hearing from
everyone else. Is ports *that much better*? I'm inclined to believe so,
at this point.

>>> To state it simply, it is often more difficult to package something
>>> released in .gem format than in an equivalent tarball + setup.rb. I
>>> know this from my personal experience when repackaging a large
>>> number of libraries and applications [2] and most importantly from
>>> what several developers working on established repackaging efforts
>>> (Debian, FreeBSD, PLD, Suse) have told me.
> Indeed.  Note that this is partly due to a different point of view.
> For Debian we package the libs and apps ourselves. I hate to say it
> but there is almost no need for RubyGems in our case, whereas it is
> very handy and good for platforms lacking such packaging and Q&A
> (win32, OSX, ...).  I certainly see the point and usefulness of the
> system.

I've had more problems because of Debian's repackaging than any other
platform, bar none. I've seen *no* benefit to the Debian philosophy.
I've had a lot more luck with FreeBSD, Gentoo, and Slackware. Frankly, I
don't believe Debian repackagers when they say "we don't need RubyGems."
I've still needed CPAN on Debian; I'll still need RubyGems. If I ever
have to use Debian again, which ghu willing I won't.

> I am still more interested in a system to generalize _all_ Ruby
> lib/app sources, like Package[2]. If RubyGems, repackagers, etc. all
> could use that, that would be great. And it makes more sense to have
> that in the core IMO.

Except that Package is entirely vapour at this point. It makes a lot
more sense to have something that works and exists in the core than
vapour. I respect Chris a lot, but RubyGems is *here*. Without real
code, you can't put something in the core. RubyGems is also proven and
works across platforms. The integration of RubyGems, I think, is a
given.

>>> This is due to RubyGems breaking source compatibility with
>>> non-RubyGems system in several areas, including, but not limited to
>>> the following:
>>> * lack of support for DATADIR
>> [...]
>>> * obviously, require_gem
>> Which should be going away, and has always been said that it would be
>> going away. The biggest problem will be the association between the
>> gem name (what you install) and the libraries (what you require) and
>> whether that should be maintained. I do not believe that it should.
> With this gone, it would stop creating source incompatibilities. This
> would result in far less patches of apps and libs for us. Now, an app
> can not work because of the absense of RubyGems in Debian (which will
> change soon though, but packages shouldn't depend on it, IMHO).

Ideally, the repackagers shouldn't be doing any patching without (1)
notifying the upstream developer and (2) not getting a positive response
to said patches for incorporation. Why? Because you don't know our
applications or libraries. It's that simple.

> The mapping between the gem and the libs (and an API to access it)
> might still be useful, see also the end of the mail.

Yes. This should be ok.

>>> * in general, problems due to the new directory layout
>> Please expand on this, because I see nothing different between this
>> and what stow does, except that RubyGems does it transparently.
>> Indeed, with the latest versions of RubyGems, aside from the
>> necessary "require 'rubygems'" line, there is no practical difference
>> between what RubyGems does and what RPA did (excepting, of course,
>> that RPA installed directly into site_ruby).
> And exactly that practical difference makes it violate the FHS and
> thereby our Debian policy. Say we were planning to make a transition
> to seperating arch-independant and arch-dependant libs into /usr/share
> and /usr/lib dirs, with this 1-gem-1-dir approach it would be
> virtually impossible.

Again, I don't care. The 1-gem-1-dir approach offers many more benefits
than separating arch-dependent and arch-independent libs, IMO. At a
minimum, if you want to offer real code that allows for the installation
and removal of arch-dependent and arch-independent code into two
different locations, it should still respect the basic RubyGems
architecture (say /usr/share/ruby/gems/1.8/gems/foo-gem/lib/foo and
/usr/lib/ruby/gems/1.8/gems/foo-gem/lib/foo/i386-crap). To go with this,
it might be ideal to have a "gem rebuild" command so that you can
rebuild an extension gem easily.

>>> Upstream developers aware of this problem have to write their code
>>> carefully to make sure it works in both RubyGems and all other
>>> systems, or to keep two lines of development and make duplicate
>>> releases.  To sum up, this is a lot of work that we could live
>>> without if the issues affecting RubyGems were fixed before it is in
>>> more widespread use.
>> I personally have issues with the idea that repackagers will be
>> patching my libraries. I appreciate the work you did with RPA and the
>> reviews you performed on my code, but the reality is that I'm not
>> sure that repackagers *should* patch incoming code unless it is clear
>> that the project has been abandoned and it is a security patch.
> I have issues with upstream adding code that has to do with packaging.
> As Debian/Ruby maintainers, we have agreed that packaging is and
> should always be _orthogonal_ to upstream software work.

And again, I don't care. Packaging is NOT orthogonal to upstream
software work in the real world, despite what you folks might think. I
have to worry about it all the time in my real job and in my Ruby
development. I've *always* had to worry about it. If everyone switched
to the Debian philosophy, I might not have to, but that would mean
*everyone*. Since everyone isn't switching, I think that it's time for
the Debian repackagers to start recognising that some of us don't really
care about Debian and need to solve real cross-platform problems.

>>> I believe that "the Ruby standard for publishing and managing third
>>> party libraries" should not make things any harder to package, for
>>> there are legitimate reasons to prefer existent tools (rpm, dpkg,
>>> etc.) to RubyGems, [...]
>> I disagree, but I'm obviously going to be in a minority here. I think
>> that the situation on Debian demonstrates that, sometimes, the
>> repackagers do far more harm than good.
> Thanks!

That wasn't a compliment. I have little positive to say about Debian
repackagers in general, having experienced little good. I'm not
suggesting that other repackagers (RedHat, SuSE, etc.) are much better,
but I have the most negative experience with Debian.

>> I'm glad that it's better, but it never should have been the mess
>> that it was.
> If this is about the stdlib being split up. That was a choice, and I
> can understand the maintainer's choice, it was a more logical thing
> for Ruby1.6 than 1.8, stdlib being much smaller. Some people are just
> dependency-nit-pickers.

It was a bad choice. It *harmed* Ruby, and it didn't handle dependencies
properly. What's to say that you won't make similarly bad choices for
modules that I create?

>>> To make things clear, I'll repeat once more that I'm not opposed to
>>> a Ruby standard being adopted for upstream releases, or to RubyGems
>>> becoming such a standard after all the above issues have been dealt
>>> with. But I believe the current RubyGems implementation shouldn't be
>>> considered the sanctioned standard before that happens.
>> I think that you need to clarify the issues. Everything you've stated
>> here has been, IMO, quite vague. It may be useful to highlight the
>> issues with specific cases *and provide suggested solutions*.
> The issues may sound vague because they are quite general (source
> incompatibility, being able to run stuff without gems, FHS compliance)
> and most count for all gems or the entire system.

Source incompatibility is not my issue. As I said, I've *got* no source
incompatibility in PDF::Writer, and it uses RubyGems. I just did two
installs of PDF::Writer and its dependencies last night on a Slackware
machine. The first was into site_ruby; the second into RubyGems. Both
times, the demos did exactly what they were supposed to do. The only
difference in running was the addition of -rubygems to the command-line
when running the demos from RubyGems. My source, however, has not
changed between versions. (And *that* I can state with absolute
authority, because I know what my packaging rake file does.)

Running stuff without gems is an irrelevancy; I package things
automagically to do this. When RubyGems is part of the Ruby core, then
gems will always be present.

FHS compliance is also something that I don't care about -- and probably
never will. Frankly, I've *never* cared about it when managing Unix
boxes, either.

>> Those solutions may involve changes that can only happen when
>> RubyGems is incorporated in Ruby, but let's be realistic here. If the
>> RubyGems developers aren't involved in repackaging efforts, those
>> issues are going to end up being low on their priority efforts unless
>> someone comes to them with concrete problems *and suggestions for
>> solutions*.
> Ok.  Some concrete stuff then.
> * Upstream should only have to create a spec file, not change stuff in
>   the code, let 'require "foo"' stay 'require "foo"'.

That's more or less all that has to happen now. I recommend changing
require, however, to accepting a version identifier.

> * Create some generizable installer, maybe assist with Package[2] or
>   come up with something better, definitely useful to have a
>   distutils[1]-alike system in Ruby Core IMO.

I don't see any benefit to this, and as I said, Package is vapour at the
moment.

> * Create a mapping from gems to libs. This way, we _can_ include
>   RubyGems into Debian without problems. The user can get libs that
>   haven't been packaged yet or newer versions but is warned when gems
>   are installed overriding a lib that already has been installed via
>   dpkg and vice versa.

Um. You can simply parse the gemspecs to do this.

>   This probably sound much more easy than it is, and it's also
>   more of a workaround.

No, I think it is pretty easy.

> Debian is about to unite much of Ruby lib/app packaging under a team.
> We are improving our system, now using setup.rb, to be able to package
> and maintain a lot very efficiently.  The thing we have in common with
> gems is the metadata and install part and we would very much like to
> have a suiting system for that in core.  However, currently, RubyGems
> increases our amount of work and we are about to go slower rather than
> faster.

*shrug* If Debian is the only system that's truly negatively impacted by
this, I don't see a problem here. RubyGems is here, it's real, and IMO
it's much more useful than distutils.

-austin
--
Austin Ziegler * [e-mail]
               * Alternate: [e-mail]