From: akr@... Date: 2017-08-31T06:59:37+00:00 Subject: [ruby-core:82563] [Ruby trunk Misc#13597] Does read_nonblock call remalloc for the buffer if does it just set the size attribute Issue #13597 has been updated by akr (Akira Tanaka). We discussed this in a developer meeting, it is better to call realloc when outbuf is not given and maxlen - len is bigger than 4K where len is the length actually read. (if maxlen - len is too short, freeing memory is not worth than realloc's cost (such as acquiring a lock)) ---------------------------------------- Misc #13597: Does read_nonblock call remalloc for the buffer if does it just set the size attribute https://bugs.ruby-lang.org/issues/13597#change-66380 * Author: emilys (Emily Stolfo) * Status: Open * Priority: Normal * Assignee: ---------------------------------------- Hello I've observed that a lot of memory gets allocated and wasted when read_nonblock is called for a number of bytes much larger than is actually read from the socket. This line https://github.com/ruby/ruby/blob/0130bb01baed404c0e3c75bd5db472415a6da1d3/io.c#L2686 appears to eventually only change the heap size value here https://github.com/ruby/ruby/blob/144e06700705a3f067582682567bc77b429c4fca/string.c#L104 but does not call remalloc. I see this request to allow an offset to be passed to read_nonblock: https://bugs.ruby-lang.org/issues/11484 but until that is implemented, how do you recommend efficiently asking to read a large number of bytes from a socket? If I'm not mistaken, if I request 16000000, but only read 1000000, the buffer that has been allocated in io_read_nonblock for 16000000 doesn't seem to be resized. Would you recommend instead requesting a more predictable number of bytes, closer to the default system value (SO_RCVBUF, for example) in each call to read_nonblock? For context, this pull request against the MongoDB Ruby driver has lead me to this investigation. https://github.com/mongodb/mongo-ruby-driver/pull/864 Thank you in advance Emily -- https://bugs.ruby-lang.org/ Unsubscribe: