[ruby-core:98668] [Ruby master Feature#16352] Modify Marshal to dump objects larger than 2 GiB

Issue #16352 has been updated by duerst (Martin D=FCrst).


marcandre (Marc-Andre Lafortune) wrote in #note-6:
> Couldn't we dedicate a special "size" value to indicate "extended marshal=
 size" (say `SIZEOF_LONG - 1`), such that compatibility with all current an=
d future marshal dumps is maintained, with the exception of a marshal objec=
t that would actually happen to have exactly a size of `SIZEOF_LONG - 1`?

This sounds like a good idea because the chance of old files with a value o=
f exactly `SIZEOF_LONG - 1` is low. The problem is that, as far as I unders=
tand, it significantly increases the difficulties of diagnosing/debugging p=
roblems in the cases where `SIZEOF_LONG - 1` was actually used.

----------------------------------------
Feature #16352: Modify Marshal to dump objects larger than 2 GiB
https://bugs.ruby-lang.org/issues/16352#change-86014

* Author: seoanezonjic (Pedro Seoane)
* Status: Open
* Priority: Normal
----------------------------------------
Using a gem called Numo-array to handle matrix operations, I found the foll=
owing error while saving a large matrix:

```
in `dump': long too big to dump (TypeError)
```

Github thread is https://github.com/ruby-numo/numo-narray/issues/144. Diggi=
ng with the authors, I found the following code that reproduces the error:

```
ruby -e 'Marshal.dump(" "*2**31)'
```

Executed in:
ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]

The marshal library has a limit based on constant `SIZEOF_LONG`. This check=
 is performed in [here](https://github.com/ruby/ruby/blob/e7ea6e078fecb70fb=
c91b04878b69f696749afac/marshal.c#L301L321). I don't understand the motivat=
ion of this limit. It has a great impact on libraries that need to serializ=
e large objects such as numeric matrix. In this case, the limit >=3D 2 GiB =
is reached easily, and it blocks ruby development. I found another related =
bug report: #1560, but the Marshal problem was not addressed in it.



-- =

https://bugs.ruby-lang.org/

Unsubscribe: <mailto:[e-mail]?subject=3Dunsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>