From: duerst@... Date: 2020-06-03T07:28:11+00:00 Subject: [ruby-core:98637] [Ruby master Feature#16352] Modify Marshal to dump objects larger than 2 GiB Issue #16352 has been updated by duerst (Martin D�rst). jeremyevans0 (Jeremy Evans) wrote in #note-3: > Supporting the dumping of larger objects does seem like a useful feature, but as @shyouhei mentioned, it requires a format change, which would break backwards compatibility as a marshal dump from Ruby 3 would not be restorable on Ruby 2.7. It does seem like Ruby 3 would be a good time to implement such a format change if we want to support the marshaling of larger objects. We would probably want to keep the supporting the old format so that a marshal dump from Ruby 2.7 will work in Ruby 3, and maybe consider working on a gem that you could install in older Ruby versions to support the new marshal format. It sure would be useful to be able to dump huge stuff. But how frequent is this? My guess is that it would be better to by default use the current format, and switch to the new format with an explicit option. That might give the better interoperability story. ---------------------------------------- Feature #16352: Modify Marshal to dump objects larger than 2 GiB https://bugs.ruby-lang.org/issues/16352#change-85962 * Author: seoanezonjic (Pedro Seoane) * Status: Open * Priority: Normal ---------------------------------------- Hi Using a gem to handle matrix operations called Numo-array I found the following error when save large matrix: in `dump': long too big to dump (TypeError) Github thread: https://github.com/ruby-numo/numo-narray/issues/144 Digging with the authors, we found the following code that reproduces the error: ``` ruby -e 'Marshal.dump(" "*2**31)' ``` Executed in : ruby 2.7.0dev (2019-11-12T12:03:22Z master 3816622fbe) [x86_64-linux] The marshal library has a limit that is checked with the SIZEOF_LONG constant. This check is performed in this line https://github.com/ruby/ruby/blob/e7ea6e078fecb70fbc91b04878b69f696749afac/marshal.c#L301 to 321 of the Marshal.c file. I don't understand the motivation of this limit and has a great impact in libraries that need to serialize large objects as numeric matrix. In this case, the limit of >= 2 GiB it's reached easily and it blocks the ruby development in scientifical projects as cited. I found other bug related: #1560, but the Marshal problem itself was not addressed in this case. Thank you in advance PEdro Seoane -- https://bugs.ruby-lang.org/ Unsubscribe: