[ruby-core:87077] Re: [Ruby trunk Bug#14745] High memory usage when using String#replace with IO.copy_stream

[e-mail] wrote:
> I wrote:
> > Finally, I always assumed your example is a contrived case and
> > you're dealing with an interface somewhere (not StringIO) which
> > doesn't accept a destination buffer for .read.
 
> The example was simplified for reproducing purposes. The place
> where I discovered this was in
> https://github.com/rocketjob/symmetric-encryption/pull/98 (I
> eventually managed to figure out `String#replace` was causing
> the high memory usage, so I switched to `String#clear`).
> 
> In short, the `SymmetricEncryption::Reader` object wraps an IO
> object with encrypted content, and when calling `#read` it
> reads data from the underlying IO object, decrypts it and
> returns the decrypted data. So, it's not patching the lack of
> outbuf argument (because the underlying IO object *should*
> accept the outbuf argument), rather it provides an `IO#read`
> interface over incrementally decrypting IO object content.

I may be misreading(*), but it looks like @stream_cipher.update
can take a second destination arg (like IO#read and friends) and
maybe that helps...  (that appears to be OpenSSL::Cipher#update)

If that's not usable somehow, I've sometimes wondered if adding a
String#exchange! method might help:

	foo = "hello............................"
	bar = "world............................"
	foo.exchange!(bar)
	p foo # => "world............................"
	p bar # => "hello............................"

In your case, you could still use String#clear with this:

	dst.clear
	dst.exchange!(tmp)
	# tmp is now '' because dst was cleared before


For non-embed strings, it would swap the pointers and be zero-copy,
but won't suffer the new-frozen-string-behind-your-back problem
of String#replace.



(*) I'm looking at your commit 8d41efacff26f3357016d6b611bee174802fba66
    after git clone-ing (since I don't do JS-infested web UIs)

Unsubscribe: <mailto:[e-mail]?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>