[ruby-core:112492] [Ruby master Feature#19024] Proposal: Import Modules

Issue #19024 has been updated by shioyama (Chris Salzberg).


I wanted to update this because I've changed my thinking since the original proposal.

TL;DR
1. I agree that we should not change `require`, `require_relative`, `load` or `autoload` (at least, not in ways that would break existing usage). Thanks @jeremyevans0 and others for convincing me of this.
2. Any new way to import code should be _opt-in_. Again, many voiced this opinion here and it makes sense to me now.

I decided to take these two as constraints and see what else was possible in Ruby 3.2, and came up with (a new version of) [Im](https://github.com/shioyama/im). Im is an "isolated module autoloader", a fork of Zeitwerk which autoloads constants under an anonymous namespace.

The interface for Im is nearly identical to Zeitwerk except that rather than loading to top-level, constants are loaded under loaders themselves, where `Im::Loader` subclasses `Module` and can therefore define its own namespace.

So for gem code, it looks like this:

```ruby
# lib/my_gem.rb (main file)

require "im"
loader = Im::Loader.for_gem
loader.setup # ready!

module loader::MyGem
  # ...
end

loader.eager_load # optionally
```

Notice here that `loader` encapsulates the entire loaded namespace. Further details in the [readme](https://github.com/shioyama/im/blob/main/README.md). I've also built a [sample Rails app](https://bugs.ruby-lang.org/issues/17881) which uses Im to load all its code under a single application namespace. Internally Im uses `load` with the second module argument (discussed here) and also `Module#const_added` ([ref](https://bugs.ruby-lang.org/issues/17881)), also added in Ruby 3.2.

The advantages to this approach:
- autoloading does not require "returning" anything, unlike `require`, where any change would face the problem of where to "receive" the thing you've loaded if not at toplevel. For the most part you can just write your code exactly as you would any other Zeitwerk-autoloaded code, without any `import` calls in each file, and for a gem you just `import` the gem once and get the whole tree of autoloaded code.
- the Zeitwerk convention for file naming/loading (inherited from Rails) is now widely adopted, and so the changes to make a gem "Im-compatible" should generally be quite small. (The exception here is Rails, which depends heavily on `Module#name` to map association names, etc.)
- Although the approach does not guarantee isolation (e.g. you can always "break out" by referencing toplevel with `::Foo`), it _can_ guarantee a kind of "opt-in" isolation, whereby _within your autoloaded code you own the toplevel (because your "toplevel" is the top of an anonymous-rooted module namespace). Similarly, a gem can entirely remove itself from the global namespace, instead allowing the gem consumer to determine the top constant name under which to load code. So two gems that follow the convention are isolated from each other provided they don't create/modify anything at the "absolute toplevel".

@rubyFeedback

> This can lead to problems sometimes. I agree it is not a huge
problem per se

I disagree here, I think this is actually a huge issue, and I think it's only because we as Rubyists are so used to it that we treat it as a "minor" inconvenience. It's fundamentally a scaling issue both in the code ecosystem space (rubygems) and in the application space.

As you noted, it's not just a problem that "my constant collides with your constant with the same name". It's that _every_ pair of collaborators in an application (every pair of gems in the `Gemfile`, plus every contributor to the application itself) have to follow a contract that says nobody will modify the same namespace in "unexpected" ways. When you start scaling things up, to an application with thousands of contributors with hundreds of gems, this becomes problematic at best.

I think Im is a potential solution to this problem. Moreover, allaying I think some of the concerns expressed here, a gem can opt to offer two "endpoints", one for Zeitwerk and one for Im, such that the gem consumer can decide how to "consume" the gem code (either at toplevel or under an anonymous-rooted namespace). So if you like your universe always pointing to the same toplevel, it would be possible to keep that, whereas others who want to "relativize" the toplevel would also be able to do that.

In any case, I don't really feel the need for further changes to Ruby other than any supporting the existing functionality in 3.2. I'm happy if this is closed, unless others want to keep it open.

----------------------------------------
Feature #19024: Proposal: Import Modules
https://bugs.ruby-lang.org/issues/19024#change-101935

* Author: shioyama (Chris Salzberg)
* Status: Open
* Priority: Normal
----------------------------------------
There is no general way in Ruby to load code outside of the globally-shared namespace. This makes it hard to isolate components of an application from each other and from the application itself, leading to complicated relationships that can become intractable as applications grow in size.

The growing popularity of a gem like [Packwerk](https://github.com/shopify/packwerk), which provides a new concept of "package" to enforce boundaries statically in CI, is evidence that this is a real problem. But introducing a new packaging concept and CI step is at best only a partial solution, with downsides: it adds complexity and cognitive overhead that wouldn't be necessary if Ruby provided better packaging itself (as Matz has suggested [it should](https://youtu.be/Dp12a3KGNFw?t=2956)).

There is _one_ limited way in Ruby currently to load code without polluting the global namespace: `load` with the `wrap` parameter, which as of https://bugs.ruby-lang.org/issues/6210 can now be a module. However, this option does not apply transitively to `require` calls within the loaded file, so its usefulness is limited.

My proposal here is to enable module imports by doing the following:

1. apply the `wrap` module namespace transitively to `require`s inside the loaded code, including native extensions (or provide a new flag or method that would do this),
2. make the `wrap` module the toplevel context for code loaded under it, so `::Foo` resolves to `<top_wrapper>::Foo` in loaded code (or, again, provide a new flag or method that would do this). _Also make this apply when code under the wrapper module is called outside of the load process (when `top_wrapper` is no longer set) &mdash; this may be quite hard to do_.
3. resolve `name` on anonymous modules under the wrapped module to their names without the top wrapper module, so `<top_wrapper>::Foo.name` evaluates to `"Foo"`. There may be other ways to handle this problem, but a gem like Rails uses `name` to resolve filenames and fails when anonymous modules return something like `#<Module: ...>::ActiveRecord` instead of just `ActiveRecord`.

I have roughly implemented these three things in [this patch](https://github.com/ruby/ruby/compare/master...shioyama:ruby:import_modules). This implementation is incomplete (it does not cover the last highlighted part of 2) but provides enough of a basis to implement an `import` method, which I have done in a gem called [Im](https://github.com/shioyama/im).

Im provides an `import` method which can be used to import gem code under a namespace:

```ruby
require "im"
extend Im

active_model = import "active_model"
#=> <#Im::Import root: active_model>

ActiveModel
#=> NameError

active_model::ActiveModel
#=> ActiveModel

active_record = import "active_record"
#=> <#Im::Import root: active_record>

# Constants defined in the same file under different imports point to the same objects
active_record::ActiveModel == active_model::ActiveModel
#=> true
```

With the constants all loaded under an anonymous namespace, any code importing the gem can name constants however it likes:

```ruby
class Post < active_record::ActiveRecord::Base
end

AR = active_record::ActiveRecord

Post.superclass
#=> AR::Base
```

Note that this enables the importer to completely determine the naming for every constant it imports. So gems can opt to hide their dependencies by "anchoring" them inside their own namespace, like this:

```ruby
# in lib/my_gem.rb
module MyGem
  dep = import "my_gem_dependency"

  # my_gem_dependency is "anchored" under the MyGem namespace, so not exposed to users
  # of the gem unless they also require it.
  MyGemDependency = dep

  #...
end
```

There are a couple important implementation decisions in the gem:

1. _Only load code once._ When the same file is imported again (either directly or transitively), "copy" constants from previously imported namespace to the new namespace using a registry which maps which namespace (import) was used to load which file (as shown above with activerecord/activemodel). This is necessary to ensure that different imports can "see" shared files. A similar registry is used to track autoloads so that they work correctly when used from imported code.
2. Toplevel core types (`NilClass`, `TrueClass`, `FalseClass`, `String`, etc) are "aliased" to constants under each import module to make them available. Thus there can be side-effects of importing code, but this allows a gem like Rails to monkeypatch core classes which it needs to do for it to work.
3. `Object.const_missing` is patched to check the caller location and resolve to the constant defined under an import, if there is an import defined for that file.

To be clear: **I think 1) should be implemented in Ruby, but not 2) and 3).** The last one (`Object.const_missing`) is a hack to support the case where a toplevel constant is referenced from a method called in imported code (at which point the `top_wrapper` is not active.)

I know this is a big proposal, and there are strong opinions held. I would really appreciate constructive feedback on this general idea.

Notes from September's Developers Meeting: https://github.com/ruby/dev-meeting-log/blob/master/DevMeeting-2022-09-22.md#feature-10320-require-into-module-shioyama

See also similar discussion in: https://bugs.ruby-lang.org/issues/10320



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- [e-mail]
 To unsubscribe send an email to [e-mail]
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/