From: "dug (Doug E) via ruby-core" Date: 2026-11-18T22:58:27+00:00 Subject: [ruby-core:127602] [Ruby Feature#22604] Add `lstat: true` to `Find.find` to yield the `File::Stat` it already computes Issue #22604 has been reported by dug (Doug E). ---------------------------------------- Feature #22604: Add `lstat: true` to `Find.find` to yield the `File::Stat` it already computes https://bugs.ruby-lang.org/issues/22604 * Author: dug (Doug E) * Status: Open ---------------------------------------- # Abstract `Find.find` already calls `File.lstat` on every entry to decide whether to descend into it, but discards the result, so a block that needs the entry's size, inode or type has to call `File.lstat` again. I propose an `lstat: true` keyword for `Find.find` and `Pathname#find` that yields that `File::Stat` along with each path, so these walks make one `lstat(2)` call per entry instead of two. # Background When the block needs the stat, every entry costs two `lstat` calls (attached as `repro_find_lstat.rb`): ```ruby require "find" require "tmpdir" calls = 0 File.singleton_class.prepend(Module.new { define_method(:lstat) { |path| calls += 1; super(path) } }) Dir.mktmpdir do |dir| 3.times { |i| File.write(File.join(dir, "file#{i}"), "x" * i) } entries = size = 0 Find.find(dir) do |path| entries += 1 size += File.lstat(path).size end puts "#{entries} entries, #{calls} File.lstat calls" end ``` ``` $ ruby -v ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [arm64-darwin20] $ ruby repro_find_lstat.rb 4 entries, 8 File.lstat calls ``` `lib/find.rb` from master gives the same result. I expected 4 calls, one per entry, because the block's call repeats the one `Find.find` has just made. Code that walks trees this way: * Homebrew sized installed packages with `Pathname#find`, calling `lstat` on each entry for its size and `[dev, ino]`. To avoid the second call, https://github.com/Homebrew/brew/pull/24629 replaced `Pathname#find` with its own copy of the traversal, which a maintainer noted in review amounts to reimplementing Ruby internals. * fpm computes a Debian package's Installed-Size the same way ([deb.rb](https://github.com/jordansissel/fpm/blob/158b937600617636a350fe09e4f33f45bd3909b3/lib/fpm/package/deb.rb#L1091-L1095)): ```ruby Find.find(staging_path) do |path| stat = File.lstat(path) next if stat.directory? total += stat.size end ``` It also selects files for `md5sums` with `File.file?(path) && !File.symlink?(path)` ([deb.rb](https://github.com/jordansissel/fpm/blob/158b937600617636a350fe09e4f33f45bd3909b3/lib/fpm/package/deb.rb#L1281-L1287)), up to two more calls per entry for what `stat.file?` on an `lstat` result already tells. * The example in #13896 adds up file sizes inside `Find.find` with `FileTest.directory?` and `FileTest.size`. Existing ways to make one call per entry: * Write the traversal yourself with `Dir.children` and `File.lstat`, as Homebrew did. This duplicates `Find.find`, including its list of ignored errors. * Call `Dir.glob("**/*", File::FNM_DOTMATCH, base: dir)`, then `File.lstat` on each result. It is nearly as fast (see Evaluation), but it skips dotfiles unless given `File::FNM_DOTMATCH`, returns the root as `"."`, cannot prune, and has no way to raise on errors. * `Dir.scan` (#21800, new in Ruby 4.1) yields each entry's type without an `lstat`, which does not help when the caller needs the size or inode. * `FileUtils::Entry_` keeps each entry's `lstat` while it traverses, but it is internal. I found no earlier proposal for this. #7596 added the existing `ignore_error:` keyword to `Find.find`. # Proposal ```ruby Find.find(*paths, ignore_error: true, lstat: false) {|path| } # unchanged Find.find(*paths, lstat: true) {|path, stat| } # stat is File.lstat(path) Pathname#find(ignore_error: true, lstat: false) # same, with a Pathname ``` * `lstat:` defaults to `false`. Without it, nothing changes. * With `lstat: true`, the block receives each path and `File.lstat(path)`, or `nil` when that call raised an error that `ignore_error: true` rescues. It yields the same paths as without the keyword. * Without a block, it returns an Enumerator of `[path, stat]` pairs. * `Find.prune` works as before. * The keyword is named after the call that produces the stat, which also says that it does not follow symlinks. # Implementation The change to `Find.find`: ```diff - def find(*paths, ignore_error: true) # :yield: path - block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error) + def find(*paths, ignore_error: true, lstat: false) # :yield: path + block_given? or return enum_for(__method__, *paths, ignore_error: ignore_error, lstat: lstat) @@ catch(:prune) do - yield file.dup + yield file.dup unless lstat begin s = File.lstat(file) rescue Errno::ENOENT, Errno::EACCES, Errno::ENOTDIR, Errno::ELOOP, Errno::ENAMETOOLONG, Errno::EINVAL raise unless ignore_error + yield file.dup, nil if lstat next end + yield file.dup, s if lstat if s.directory? then ``` The attached `find_lstat.patch` also passes the keyword through `Pathname#find`, documents it for both methods, and adds tests to `test/test_find.rb` and `test/pathname/test_pathname.rb`. With the patch applied, both test files pass on Ruby 4.0.9. # Evaluation With `lstat: true`, the script above makes 4 `File.lstat` calls instead of 8. The table shows the time to walk two Homebrew packages and the whole Homebrew Cellar, adding up sizes and counting each hard-linked file once. It was measured on macOS (arm64) with Ruby 4.0.9 (`bench_ticket.rb`), as the median of 31 runs per package and 7 for the Cellar, in shuffled order. All four walks produce the same totals. | Walk | `ruby` (22,080 entries) | `gradle@8` (27,296) | Cellar (236,554) | |---|---:|---:|---:| | `Find.find`, then `File.lstat` in the block | 606 ms | 837 ms | 7,579 ms | | `Find.find(lstat: true)` | 424 ms | 657 ms | 4,981 ms | | `Dir.glob`, then `File.lstat` | 441 ms | 686 ms | 5,213 ms | | Homebrew's own traversal | 439 ms | 627 ms | 4,860 ms | `lstat: true` is 1.3���1.5x faster than calling `File.lstat` in the block, and within 5% of Homebrew's traversal. It saves one `lstat(2)` call per entry on any platform; how much time that saves depends on the cost of `lstat`, which I have measured only on macOS. # Discussion * `Dir.glob` comes within 5% of `lstat: true`, so the case is less about speed than about `Find.find`, the API meant for walking a tree, making callers that need the stat pay for twice as many `lstat` calls as a glob or a hand-written loop. * With `lstat: true`, the stat is taken before the block runs instead of after, so `Find.find` decides whether to descend based on the entry as it was before the block ran. With `ignore_error: false`, an error from `lstat` is raised before the entry is yielded rather than after. * This follows the points raised about #21800. The stat comes from an `lstat` made when the entry is visited, not a lazily filled object, so it is clear when it was taken. It is `lstat` rather than `stat` because that is what `Find.find` already uses. A keyword avoids checking the block's arity. * Alternatives: a separate method such as `Find.find_with_lstat`; or skipping entries whose `lstat` fails instead of yielding `nil`, which would yield fewer paths than `Find.find` does today. # Summary `Find.find` already makes one `lstat` call per entry, and `lstat: true` lets the block use the result instead of making another. It is a few lines in `lib/find.rb`, changes nothing for existing callers, and in Homebrew's case is within 5% of the hand-written traversal it would replace. -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/