From: normalperson@... Date: 2016-12-31T01:01:49+00:00 Subject: [ruby-core:78921] [Ruby trunk Bug#13085] io.c io_fwrite creates garbage Issue #13085 has been updated by Eric Wong. File 0001-io.c-io_fwrite-temporarily-freeze-string-when-writin.patch added Proposed patch to temporarily freeze string while copying ~~~ io.c (io_fwrite): temporarily freeze string when writing This avoids garbage from IO#write for [Bug #13085]. Memory usage from benchmark/bm_io_copy_stream_write.rb is reduced greatly: target 0: a (ruby 2.5.0dev (2016-12-30 trunk 57236) [x86_64-linux]) target 1: b (ruby 2.5.0dev (2016-12-30) [x86_64-linux]) Memory usage (last size) (B) name a b io_copy_stream_write 82235392.000 6651904.000 Memory consuming ratio (size) with the result of `a' (greater is better) name b io_copy_stream_write 12.363 There is also a speedup in execution time: Execution time (sec) name a b io_copy_stream_write 0.380 0.143 Speedup ratio: compare with the result of `a' (greater is better) name b io_copy_stream_write 2.651 Caveat, there is one potential race condition: If another thread calls String#freeze on the string we are currently writing; we will blindly unfreeze it during fwrite_unfreeze from ensure. However, I do not expect this to be a real-world case. Ideally, Ruby should have a way of detecting threads which are not visible to other threads. ~~~ ---------------------------------------- Bug #13085: io.c io_fwrite creates garbage https://bugs.ruby-lang.org/issues/13085#change-62335 * Author: Eric Wong * Status: Open * Priority: Normal * Assignee: * Target version: * ruby -v: * Backport: 2.2: UNKNOWN, 2.3: UNKNOWN, 2.4: UNKNOWN ---------------------------------------- Relying on rb_str_new_frozen for unconverted strings does not save memory because copy-on-write is always triggered in read-write I/O loops were subsequent IO#read calls will clobber the given write buffer. buf = ''.b while input.read(16384, buf) output.write(buf) end This generates a lot of garbage starting with Ruby 2.2 (r44471). For my use case, even IO.copy_stream generates garbage, since I wrap "write" to do Digest calculation in a single pass. I tried using rb_str_replace and reusing the string as a hidden (klass == 0) thread-local, but rb_str_replace attempts CoW optimization by creating new frozen objects, too: https://80x24.org/spew/20161229004417.12304-1-e@80x24.org/raw So, I'm not sure what to do, temporal locking seems wrong for writing strings (I guess it's for reading?). I get test_threaded_flush failures with the following: https://80x24.org/spew/20161229005701.9712-1-e@80x24.org/raw IO#syswrite has the same problem with garbage. I can use IO#write_nonblock on fast filesystems while holding GVL, I guess... ---Files-------------------------------- 0001-io.c-io_fwrite-temporarily-freeze-string-when-writin.patch (2.6 KB) -- https://bugs.ruby-lang.org/ Unsubscribe: