[ruby-core:36407] [Ruby 1.9 - Bug #3924] Performance bug (in require?)

Issue #3924 has been updated by Xavier Shay.

File require_performance.diff added

Progress update:

Didn't make much impact on the graph in prior update, but also may have been to eager to blame slow rails loading on it - rails only has ~50 entries in the load path.
I added a cache to file expansion (so it remembers the expanded path of 'active_record" without having to search for it), which gave a ~0.5s speed up on a blank rails app, and ~6s on a medium sized one. This patch is now faster than 1.9.2 in all benchmarks I have, and *way* faster than 1.9.3dev (though it has been that for a while). I suspect there are further optimizations that could be made, especially now the code is easier to follow.

I have attached a diff. It passes `make test`, `make test-all`, and a few big rails test suites. There are still some outstanding issues on which need discussion.
1) This is pretty much a rewrite for load.c, since I couldn't get my head around `rb_feature_p`. That means it's risky. It is likely possibly to add a hash-table lookup to the existing architecture, but someone else would have to volunteer to do this. On the up side, it is far easier to follow the algorithm in the new code.
2) It doesn't use the file search functions in `file.c`, so I suspect there could be some safe level issues not covered yet.
3) I have made public (non-static) one or two functions in `file.c` and `array.c`. Is this OK? Pretty sure I could rewrite to avoid doing this if necessary.
4) `rb_provide` doesn't really make sense. It was being used in `enumerator.c` but I found another method of providing backwards compatibility. Since it is in `intern.h`, it can maybe just be removed?
5) I use a proxy class for $LOADED_FEATURES to keep the loaded features hash up to date. Is this OK? Bit of a hack but I couldn't come up with a nicer way of doing it.
6) Is it OK to add a new member to the VM struct? I added new_loaded_features (to be renamed shortly).
7) Not tested on windows yet.
8) The code would be nicer if some related functions could be moved to another file (rb_locate_*), I don't know the best way to do this though.
----------------------------------------
Bug #3924: Performance bug (in require?)
http://redmine.ruby-lang.org/issues/3924

Author: Carsten Bormann
Status: Open
Priority: Normal
Assignee: 
Category: core
Target version: 1.9.3
ruby -v: ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10] 


=begin
 Running irb < /dev/null in 1.9.2 causes 3016 calls to lstat64.
 
 For instance, there is a sequence of 28 repetitions each of lstat calls to all 6 non-empty path prefixes of /opt/local/lib/ruby1.9/1.9.1/irb.rb -- a total of 170 lstats apparently just to load this file; another set of lstats then occurs later for another 18 (times 6) times.  Clearly, something is running amok in the calling sequence rb_require_safe -> realpath_rec -> lstat.
 
 Another example: Running a simple test with the baretest gem causes 17008 calls to lstat.  According to perftools.rb, 80 % of the 1.2 seconds of CPU is used in Kernel#gem_original_require (and another 12 in GC, some of which may be caused by this).
=end



-- 
http://redmine.ruby-lang.org