[ruby-core:63120] [ruby-trunk - Bug #9934] High memory usage from file_expand_path_*

Issue #9934 has been updated by Eric Wong.


 [e-mail] wrote:
 > This is the best I could come up with. Definitely not ideal since it
 > allocates another ruby object.
 
 What about using rb_str_resize?
 
 rb_str_resize may work even better if we remove the 1024 byte threshold
 and trust the realloc implementation to do the right thing.
 
 Additionally, rb_str_freeze may also be smarter and force a resize
 unconditionally, as keeping the buffer is useless for frozen strings.
 Not sure if changing rb_str_freeze this way breaks compatibility
 (but we should be able to safely resize for rb_fstring, at least).

----------------------------------------
Bug #9934: High memory usage from file_expand_path_*
https://bugs.ruby-lang.org/issues/9934#change-47181

* Author: Aman Gupta
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: trunk
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
All the file expansion routines use `EXPAND_PATH_BUFFER()` which allocates PATH_MAX bytes on the heap per invocation.
The strings returned by `File.expand_path` are never realloc'd after they are populated, so they continue using 4kb (on linux) per string.
In our rails app, 22MB of heap usage is due to expanded path name strings.

~~~
$ ruby -robjspace -e' puts ObjectSpace.dump(File.expand_path("/foo")) '
{"address":"0x007fa2b44dd6c8", "type":"STRING", "class":"0x007fa2b3f99608", "bytesize":4, "capacity":4098, "value":"/foo", "encoding":"US-ASCII", "memsize":4099, "flags":{"wb_protected":true}}
~~~

The following failing patch demonstrates the issue as well:

~~~
diff --git a/test/ruby/test_file_exhaustive.rb b/test/ruby/test_file_exhaustive.rb
index 2c945ea..49be9de 100644
--- a/test/ruby/test_file_exhaustive.rb
+++ b/test/ruby/test_file_exhaustive.rb
@@ -458,6 +458,12 @@ class TestFileExhaustive < Test::Unit::TestCase
     end
   end

+  def test_expand_path_memsize
+    require "objspace"
+    path = File.expand_path("/foo")
+    assert_equal 5, ObjectSpace.memsize_of(path)
+  end
+
   def test_expand_path_encoding
     drive = (DRIVE ? 'C:' : '')
     if Encoding.find("filesystem") == Encoding::CP1251
~~~



-- 
https://bugs.ruby-lang.org/