[ruby-core:97849] [Ruby master Misc#16778] Should we stop vendoring default gems code?

Issue #16778 has been updated by deivid (David Rodr=EDguez).


Hei @mame!

> When Ruby CI fails, we need to stop the error as soon as possible. It is =
not acceptable to send a pull request and wait for acceptance. Otherwise, t=
he notification flood makes it hard to detect a new type of failure. This m=
akes the maintainability of ruby/ruby low.

> Note that fixing the issue in ruby/ruby is sometimes incredibly hard. Som=
e types of errors only occur randomly on a limited subset of CIs, and we ca=
nnot reproduce the issue on our local machine. In this case, we commit a de=
bugging code to the source tree, wait for a new failure and debugging log, =
and repeat this process until the issue is identified.

From what I've seen, and please correct me if I'm wrong, the vast majority =
of the times these issues happen when testing default libraries inside ruby=
-core. So, if you stop testing default libraries inside ruby-core, this wil=
l become an non-issue, won't it?

> Consider a recent vulnerability issue of JSON gem (CVE-2020-10663). It lo=
oks that JSON gem maintains its only latest version (2.3). Actually, they r=
eleased only JSON gem 2.3 against the vulnerability.

> However, Ruby 2.5 bundles JSON 2.1. As Ruby's branch maintenance policy, =
a new feature is not backported to the released branches. So, even if a vul=
nerability is found in JSON gem, Ruby 2.5 cannot bundle JSON 2.3 as-is. Rub=
y 2.5 has a copy of source code of JSON, so we could fix the issue directly=
. But if there had been no copy, we couldn't have addressed the issue in Ru=
by 2.5.

The right solution for this would be that the json gem released a patch-lev=
el json 2.1 release. I believe it should be avoided that ruby ships with a =
copy of a default gem that claims to be "x.y.z" but has different code from=
 he "x.y.z" version of the gem at rubygems.org. In any case, I believe this=
 issue could be workarounded in different ways so that the final ruby insta=
llation includes the security fix. For example, include a ".patch" file in =
source control and apply it after downloading the default copy of the gem a=
nd before installing it. Or download a fork of the gem, or a specific branc=
h, including the security fix instead of the canonical one, and install tha=
t. Or any other similar alternatives.

----------------------------------------
Misc #16778: Should we stop vendoring default gems code?
https://bugs.ruby-lang.org/issues/16778#change-85073

* Author: deivid (David Rodr=EDguez)
* Status: Assigned
* Priority: Normal
* Assignee: hsbt (Hiroshi SHIBATA)
----------------------------------------
Currently ruby-core vendors all the code in default gems, and runs the test=
s for each of them.

Also, ruby-core continuously updates the vendored code of default gems to s=
ync with the upstream repos. That's overhead work, not only from syncronizi=
ng the code itself, but it also requires perfect syncronization of releases=
 to avoid including versions of default gems that are different from releas=
ed versions.

Also, this causes confusion for contributors because the code lives "duplic=
ated" in two different places. Some times contributors will open a PR in th=
e ruby-core repo, only to find out that they need to go to the upstream rep=
o and contribute it in there. And this rule is not even always followed and=
 sometimes ruby-core contributors apply patches to the vendored code direct=
ly (many times to fix test-only issues inherent to the different structure =
of the core repository). These patches then need to be contributed back to =
the upstream repo.

I believe that all of that kind of defeats the point of "gemification" of t=
he standard library.

Once some ruby code its gemified, it should be the new upstream's responsab=
ility to make sure the code works and it's properly tested, and ruby-core s=
hould be free'd from that responsability.

Maybe ruby-core could do something along the following lines:

* Remove all the vendored code from default gems.
* When this code is needed for internal tests, manage it as a development d=
ependency, clone it as necessary on non source controlled locations, and us=
e it from there.
* Maybe a file similar to `gems/bundled_gems` can be added for default gems=
 indicating their versions and upstream repos, to ease things.
* Upon `make install`, clone the proper version of each default library and=
 get it installed in the default $LOAD_PATH.
* Maybe add some bare high level CI checks to ensure that all default libra=
ries can be properly required after `make install`, and that their executab=
les (if they include any) can also be run.

This should bring several benefits to the development process:

* No more duplicated code.
* No more syncronization from upstream to ruby-core.
* No more syncronization from ruby-core to upstream.
* No more confusion around the canonical place to contribute.
* No more complexities derived from the different organization of the code =
depending on whether it lives in ruby-core or outside.  =


I believe jruby already does something like this so it'd be interesting to =
get some input from them.

If this is a direction the ruby-core team would like to take, I'm happy to =
help @hsbt with small steps towards slowly approaching to this high level g=
oal.



-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>