[ruby-core:114047] [Ruby master Feature#19744] Namespace on read
From:
"tagomoris (Satoshi Tagomori) via ruby-core" <ruby-core@...>
Date:
2023-06-29 00:54:10 UTC
List:
ruby-core #114047
Issue #19744 has been updated by tagomoris (Satoshi Tagomori).
janosch-x (Janosch M=FCller) wrote in #note-1:
> are the described problems, apart of dependency conflicts, very common in=
your experience? (i've worked on several large codebases with ~1000 gems a=
nd never saw these gems trying to create the same module or edit each other=
's configs.)
Yes. I saw many problems around applications that I worked for in the past.=
Those things were "fixed" in any way (including making a closed fork of un=
maintained libraries), eventually, but I believe that it could be much easi=
er if we had namespaces.
> **concerning dependency conflicts:**
>=20
> are they perhaps the lesser evil?
>=20
> maybe "evolutionary pressure" on unmaintained dependencies is actually he=
althy for a language ecosystem?
I agree. From the viewpoint of the whole ecosystem, all libraries should be=
maintained well, and "pressure" can work well in most cases. But on the ot=
her hand, libraries that are not updated still exist even under very strong=
pressure. Loading different versions of gems can solve this problem.
I agree that the healthy pressure is very important for the ecosystem. So, =
we have to be careful when we design the UX/API of loading multiple version=
gems. If it doesn't make any pressure to update dependencies, it should ma=
ke another kind of library ecosystem nightmare: many old dependencies are n=
ever updated. We should avoid it.
> if there were multiple versions of the same dependency running in one pro=
cess, each with different bugs, wouldn't that be harder to understand?
In my current understanding, loading multiple versions can happen from depe=
ndencies of libraries (my app uses library A and B, A uses C ver 1.x, B use=
s C ver 2.x), not the direct dependency of the app itself. If the library A=
also has a bug coming from C ver 1.x, it's also a bug of A. We should upda=
te A. Isn't it?
> **concerning the scenario of running whole apps in namespaces:**
>=20
> couldn't the issues in the shared namespace, such as dependency conflicts=
, also happen within each individual namespace? would i then use namespaces=
within namespaces to work around it?
Let me check my understanding of your question. Is it?: Problems around nam=
espaces can happen in individual namespaces. Should we use namespaces to av=
oid those things?
(If my understanding is correct,) yes. And, I missed mentioning it in the o=
riginal description, namespaces can include other namespaces (just like mod=
ules can have submodules).
In other words, when we use namespaces to require/load other libraries, we =
don't take care of the namespaces used in those libraries. Namespaces in re=
quired libraries may happen, but users (the caller app/lib of NameSpace#req=
uire) don't have to consider it. Users just call the public API of those li=
braries.
> if i want to patch a library, e.g. by prepending a module with a bugfix, =
would i need to remember to apply that patch to all namespaces where i use =
that library?
Definitely, yes. And remember, monkey-patching different libraries is power=
ful but evil. With great power, great responsibility.
But I guess, we'll do NameSpace#require the code including both `require('a=
_buggy_lib')` and a monkey patch on ABuggyLib. If we do require a code with=
`require('a_buggy_lib')` without such patches, that code should not be aff=
ected from the bug.
----------------------------------------
Feature #19744: Namespace on read
https://bugs.ruby-lang.org/issues/19744#change-103711
* Author: tagomoris (Satoshi Tagomori)
* Status: Open
* Priority: Normal
----------------------------------------
# What is the "Namespace on read"
This proposes a new feature to define virtual top-level namespaces in Ruby.=
Those namespaces can require/load libraries (either .rb or native extensio=
n) separately from the global namespace. Dependencies of required/loaded li=
braries are also required/loaded in the namespace.
### Motivation
The "namespace on read" can solve the 2 problems below, and can make a path=
to solve another problem:
The details of those motivations are described in the below section ("Motiv=
ation details").
#### Avoiding name conflicts between libraries
Applications can require two different libraries safely which use the same =
module name.
#### Avoiding unexpected globally shared modules/objects
Applications can make an independent/unshared module instance.
#### (In the future) Multiple versions of gems can be required
Application developers will have fewer version conflicts between gem depend=
encies if rubygems/bundler will support the namespace on read.
### Example code with this feature
```ruby
# your_module
module YourModule
end
# my_module.rb
require 'your_module'
module MyModule
end
# example.rb
namespace1 =3D NameSpace.new
namespace1.require('my_module') #=3D> true
namespace1::MyModule #=3D> #<Module:0x00000001027ea650>::MyModule (or #<Nam=
eSpace:0x00...>::MyModule ?)
namespace1::YourModule # similar to the above
MyModule # NameError
YourModule # NameError
namespace2 =3D NameSpace.new # Any number of namespaces can be defined
namespace2.require('my_module') # Different library "instance" from namespa=
ce1
require 'my_module' # require in the global namespace
MyModule.object_id !=3D namespace1::MyModule.object_id #=3D> true
namespace1::MyModule.object_id !=3D namespace2::MyModule.object_id
```
The required/loaded libraries will define different "instances" of modules/=
classes in those namespaces (just like the "wrapper" 2nd argument of `Kerne=
l.load`). This doesn't introduce compatibility problems if all libraries us=
e relative name resolution (without forced top-level reference like `::Name=
`).
# "On read": optional, user-driven feature
"On read" is a key thing of this feature. That means:
* No changes are required in existing/new libraries (except for limited cas=
es, described below)
* No changes are required in applications if it doesn't need namespaces
* Users can enable/use namespaces just for limited code in the whole librar=
y/application
Users can start using this feature step by step (if they want it) without a=
ny big jumps.
## Motivation details
This feature can solve multiple problems I have in writing/executing Ruby c=
ode. Those are from the 3 problems I mentioned above: name conflicts, globa=
lly shared modules, and library version conflicts between dependencies. I'l=
l describe 4 scenarios about those problems.
### Running multiple applications on a Ruby process
Modern computers have many CPU cores and large memory spaces. We sometimes =
want to have many separate applications (either micro-service architecture =
or modular monolith). Currently, running those applications require differe=
nt processes. It requires additional computation costs (especially in devel=
oping those applications).
If we have isolated namespaces and can load applications in those namespace=
s, we'll be able to run apps on a process, with less overhead.
(I want to run many AWS Lambda applications on a process in isolated namesp=
aces.)
### Running tests in isolated namespaces
Tests that require external libraries need many hacks to:
* require a library multiple times
* require many different 3rd party libraries into isolated spaces (those ma=
y conflict with each other)
Software with plugin systems (for example, Fluentd) will get benefit from n=
amespaces.
In addition to it, application tests can avoid unexpected side effects if t=
ests are executed in isolated namespaces.
### Safely isolated library instances
Libraries may have globally shared states. For example, [Oj](https://github=
.com/ohler55/oj) has a global `Obj.default_options` object to change the li=
brary behavior. Those options may be changed by any dependency libraries or=
applications, and it changes the behavior of `Oj` globally, unexpectedly.
For such libraries, we'll be able to instantiate a safe library instance in=
an isolated namespace.
### Avoiding dependency hells
Modern applications use many libraries, and those libraries require much mo=
re dependencies. Those dependencies will cause version conflicts very often=
. In such cases, application developers should resolve those by updating ea=
ch libraries, or should just wait for the new release of libraries to confl=
ict those libraries. Sometimes, library maintainers don't release updated v=
ersions, and application developers can do nothing.
If namespaces can require/load a library multiple times, it also enables to=
require/load different versions of a library in a process. It requires the=
support of rubygems, but namespaces should be a good fundamental of it.
## Expected problems
### Use of top-level references
In my expectation, `::Name` should refer the top-level `Name` in the global=
namespace. I expect that `::ENV` should contain the environment variables.=
But it may cause compatibility problems if library code uses `::MyLibrary`=
to refer themselves in their deeply nested library code.
### Additional memory consumption
An extension library (dynamically linked library) may be loaded multiple ti=
mes (by `dlopen` for temporarily copied dll files) to load isolated library=
"instances" if different namespaces require the same extension library. Th=
at consumes additional memory.
In my opinion, additional memory consumption is a minimum cost to realize l=
oading extension libraries multiple times without compatibility issues.
This occurs only when programmers use namespaces. And it's only about libra=
ries that are used in 2 or more namespaces.
### The change of `dlopen` flag about extension libraries
To load an extension library multiple times without conflicting symbols, al=
l extensions should stop sharing symbols globally. Libraries referring symb=
ols from other extension libraries will have to change code & dependencies.
(About the things about extension libraries, [Naruse also wrote an entry](h=
ttps://naruse.hateblo.jp/entry/2023/05/22/193411).)
# Misc
The proof-of-concept branch is here: https://github.com/tagomoris/ruby/pull=
/1
It's still work-in-progress branch, especially for extension libraries.
--=20
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-c=
ore.ml.ruby-lang.org/