[ruby-talk:23925] Re: RubyGems Discussion

On Wed, 31 Oct 2001, Ryan Leavengood wrote:

> Wow, there has been a lot of discussion related to RubyGems over the
> past few days.  I'd like to address all the ideas and suggestions
> brought up.
> 
You're probably the one with the most knowledge and thinking about these
issues so I'd think your mostly correct, but let me play devils advocat
for a while... ;-) (It can only strengthen the final solution; either you
consider some of our ideas or your arguments for your choices grow
stronger. So be flattered and not threatened!)

> 1) RubyGems files as valid Ruby programs.
> 
>    This is an interesting idea and one that I never really considered.
> But is has a lot of drawbacks and really only one main benefit.  I'll
> start with the benefit: gems that are directly executable by the ruby
> interpreter (without a need to modify the interpreter.)  But I've
>
I would state this benefit as "in addition to being used as binary Gem
files, they are also valid Ruby programs".

Don't you agree that it would be simple to simply do

ruby a_nice_gem.rbg

and get a list of commands it accepts, see that it accepts --info and then
do

ruby --info a_nice_gem.rbg

to get the metadata and

ruby --list a_nice_gem.rbg

to get the contents of the gem? 

Agreed there are many ways to achieve this.

> mentioned several alternatives to achieve this functionality without
> having to make the entire gem file a valid Ruby program.  Other ideas
>
I don't propose making the whole gem file a valid Ruby program. I propose
adding a minimum Ruby "header" so that binary Gem files can also be
directly run. This should be pretty clear from the code in
[ruby-talk:22530]. If nothing else this would make another command line
program/script unnecessary (ie. we do not need a special gem command for
listing contents of gems etc).

> that were mentioned relating to this include having custom install
> behavior.  I think that in general this should be avoided for pure Ruby
> libraries and I would consider it to be bad design if this was needed.
>
Why? Could you elaborate?

IMHO, we need to think about non-pure Ruby libraries since its pretty
common in RAA. I'd like the same gem system to support both pure and
non-pure gems. But maybe its just complicating things; I'm not sure.

Its also possible that we are discussing different things. In Raa.succ
these things must be addressed. Could you clarify how large a
part of Raa.succ RubyGems aims to be/support?

> The drawbacks to this include:
>    - a format that is hard to read by the RubyGems system.  Most of the
> time gems will be libraries that will be accessible from 'require' like
> any other library.  But a Ruby format would require some strange 'eval'
> tricks to read them in (and wrapper classes like Rich Kilmer suggested)
> or a finished RubyInRuby parser which is complete overkill for this
> purpose.  Also imagine trying to find a single file in a gem on a system
> with 100s of gems.  You would have to load and eval each one until you
> found the one with that file.  This could be insanely slow.
> 
I think this is avoided by having a small Ruby header and then the binary
data after __END__. Doesn't it?

> On that note this format isn't ideal for my current design for RubyGems.
> In the current design gem metadata and the file list is cached but the
> actual file contents are only read when they are needed (to save
> memory.)  There is no need for an entire Ruby file to sit around in
> memory in a String when it is normally only evaluated once.  So
> generally loading files from RubyGems is as fast as Ruby loading files
> from disk (one seek and read) and also doesn't clog up memory.
> 
This only applies to the encoding-binary-in-ruby stuff, right?

>    - a format that is hard to create automatically.  Having to write a
> code generator and use base64 encoding and all that just to create a
> file that could just as well be a simple binary or text format is not
> worth it.  Or it could be worse if people having to code the gems
> themselves.  I'm creating RubyGems to make the Ruby developer's life
> easier, not harder.
> 
Yes, IMHO the encoding stuff is not a good idea.

>    - an easily modifiable format.  This probably isn't the best argument
> since it is sort of based on the "security through obscurity" concept,
> but it seems like if the gems are in Ruby then people might be tempted
> to change their metadata and what not.  This is something that should be
> avoided.  And the system will probably have checks anyways to avoid this
> but still having a binary format might be better.
> 
In general I find that a flexible approach is often superior to hard-coded
binary ones. I'm not sure to what extent your Gems format will be a
hard-coded/binary but I think we need to make sure the gems system is
highly adaptable. I don't think we can anticipate all changes and req's
it'll have to fulfill yet.

One example would be if a commercial entity wants to supply gems that use
a special disitrbuted licensing scheme that dynamically goes to their site
and registers that the gem is being used.

> 2) Different dependency types
> 
>    This is something I'm certainly open to, especially since I haven't
> coded dependency management yet.  Also Debian's package system was part
> of the influence in RubyGems, so I'm open to seeing what they have done.
> 
Great! I think they are a great addition and would pave the way for a
smarter Raa.succ.

> 4) Multiple format types
> 
>    I actually don't think this is a bad thing and have already coded two
> different file formats.  I did this so that new formats could be added
> without breaking all the already created gems.  But the main thing with
> these different file formats is that they need to support the same
> semantics: 
> 
>    - quickly reading the gem metadata without looking at the rest of the
> file.
>
I would think that the metadata is mostly used at dowload and install time
so the speed here might not be critical. Am I forgetting something?

>    - quickly reading the file list without looking at the rest of the
> file.
>    - quickly reading one of the embedded files without looking at the
> rest of the file.
> 
Ok, agreed. One bad thing with the last req is that we might loose some
opportunities for compression (compressing the whole gem will probably
give higher compression than compressing on a fil-by-file basis). I would
think that the zip format has a solution to this? (I thought zlib have
support for stream-based compression which allows positional access on
the compressed stream?)

> 5) Gems as classes following some contract
> 
>    I actually thought about this in the very beginning when planning
> RubyGems, but I decided against it since it just adds more work for the
> developer.  I want the transition from tarball to gem incredibly easy.
>
Why more work for developer? The default creation of gems simply uses the
standard classes/format. 

I fail to see how you can turn a tarball (assuming its a gzipped tar
archive) in a gem where each file is individually accessible (since gzip
works on the full tar archive)?

> It currently is (at RubyConf I demonstrated turning NQXML into a gem in
> under 30 seconds.)  The more difficult we make it to use RubyGems the
> slower it will be adopted.
> 
Agreed, however I think the contract-idea and simplicity can be
combined. Actually, I think of the contract-idea as just about the
simplest specification of a gems system there is. If we add more details
than what is really needed and things change things might break. I'd like
to avoid that. I'd like old gems to work even if the gems system is
updated. I'd like new gems to work with old gems systems. And I think this
can all be achieved without comprimising simplicity.

A compromise would be for your system to assume your binary format first
and then revert to the most flexible scheme if there is a failure.

If the flexible scheme is considered to give a performance penalty one
option would be that the gems system "converts" a gem in the flexible
scheme to its own format. If all gems only have to fulfill a minimum
contract this would be possible regardless of future updates to
RubyGems/Raa.succ. If not future systems would have to support reading
older binary formats.

Regards,

Robert