On Tue, Nov 19, 2019 at 9:05 PM <shyouhei@ruby-lang.org> wrote:

Issue #16352 has been updated by shyouhei (Shyouhei Urabe= ).

Description updated

This behaviour has been there since the beginning.=C2=A0 No ruby version si= nce 0.49 has successfully dumped such long string.=C2=A0 Same thing happens= for a very big bignum, a very long array, a class that has very long class= path (Q::W::E::R::...), an object of 2**31 instance variables (which isn= 9;t impossible these days), and much much more.

The limitation is due to marshal's binary format.=C2=A0 I guess the rea= son behind this is simply because at the time the format was designed (back= in 1990s), there simply was no such thing like a 64 bit integer type.=C2= =A0 To properly reroute we have to reconsider all use of `long` in marshal = format.=C2=A0 I guess that is essentially a format change.=C2=A0 That shoul= d hurt data portability so not that easy.

Any nice idea to fix the situation?

----------------------------------------
Bug #16352: Marshal limit of=C2=A0 >=3D 2 GiB
https://bugs.ruby-lang.org/issues/16352#change-8= 2729

* Author: seoanezonjic (Pedro Seoane)
* Status: Open
* Priority: Normal
* Assignee:
* Target version:
* ruby -v: ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-l= inux]
* Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
----------------------------------------
Hi
Using a gem to handle matrix operations called Numo-array I found the follo= wing error when save large matrix:
in `dump': long too big to dump (TypeError)
Github thread: https://github.com/ruby-numo/numo-= narray/issues/144
Digging with the authors, we found the following code that reproduces the e= rror:
```
ruby -e 'Marshal.dump(" "*2**31)'
```
Executed in :
ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux]

The marshal library=C2=A0 has a limit that is checked with the SIZEOF_LONG = constant. This check is performed in this line https://github.com/ruby/ruby/blob/e7ea6= e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Marsha= l.c file. I don't understand the motivation of this limit and has a gre= at impact in libraries that need to serialize large objects as numeric matr= ix. In this case, the limit of=C2=A0 >=3D 2 GiB it's reached easily = and it blocks the ruby development in scientifical projects as cited. I fou= nd other bug related: #1560, but the Marshal problem itself was not address= ed in this case.
Thank you in advance
PEdro Seoane

--
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=3Dunsubscribe= >
<http://lists.ruby-lang.org/cgi-bin/m= ailman/options/ruby-core>

Austin Ziegler =E2=80=A2 halostatue@gmail.com =E2=80=A2 austin@halostatue.ca
http://www.ha= lostatue.ca/ =E2=80=A2 http://twitter.com/halostatue

--000000000000a395d00597cc9c33-- --===============0184473582== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline Unsubscribe: --===============0184473582==--