From: shevegen@... Date: 2019-11-21T13:59:49+00:00 Subject: [ruby-core:95905] [Ruby master Bug#16352] Marshal limit of >= 2 GiB Issue #16352 has been updated by shevegen (Robert A. Heiler). > I don't understand the motivation of this limit and has a great impact in libraries that need to serialize large objects as numeric matrix. > In this case, the limit of >= 2 GiB it's reached easily and it blocks the ruby development in scientifical projects as cited. Shyouhei already pointed out the historic reason. I believe you can quite easily convince the ruby core team that a change may be necessary in the long run (most likely past ruby 3.0) based on use cases. Matz likes to hear real world use cases, so the more information may be given the better. :) As for possibility of change, I guess the Marshal format could be kept by default, but another variant could perhaps be added where people could switch to another format - a bit like syck and psych could be used interchangably for yaml to some extent (I used syck for quite some time even after psych was added, before I transitioned into Unicode finally; I used to specify the yaml engine via e. g. YAML.engine = or something like that). ---------------------------------------- Bug #16352: Marshal limit of >= 2 GiB https://bugs.ruby-lang.org/issues/16352#change-82748 * Author: seoanezonjic (Pedro Seoane) * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux] * Backport: 2.5: UNKNOWN, 2.6: UNKNOWN ---------------------------------------- Hi Using a gem to handle matrix operations called Numo-array I found the following error when save large matrix: in `dump': long too big to dump (TypeError) Github thread: https://github.com/ruby-numo/numo-narray/issues/144 Digging with the authors, we found the following code that reproduces the error: ``` ruby -e 'Marshal.dump(" "*2**31)' ``` Executed in : ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux] The marshal library has a limit that is checked with the SIZEOF_LONG constant. This check is performed in this line https://github.com/ruby/ruby/blob/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Marshal.c file. I don't understand the motivation of this limit and has a great impact in libraries that need to serialize large objects as numeric matrix. In this case, the limit of >= 2 GiB it's reached easily and it blocks the ruby development in scientifical projects as cited. I found other bug related: #1560, but the Marshal problem itself was not addressed in this case. Thank you in advance PEdro Seoane -- https://bugs.ruby-lang.org/ Unsubscribe: