Repository navigation
passthrough: test the readdir/readdirplus contracts and benchmark directory enumeration - #261
Merged
jiangliu merged 2 commits intoOct 9, 2026
Conversation
The existing coverage stops at buffer parsing (readdir_cookie_tests), the no_opendir re-seek regression (generic/676), and shallow error paths of test_dir_operations. The persistent-opendir flow that the kernel actually drives -- opendir handle, cookie resume, batch splitting, the readdirplus attribute and ownership contracts -- had no coverage at all. Add a readdir_tests module exercising that flow against a real backing directory through a persistent handle: - enumeration: a stream that does not fit one reply batch is drained completely and exactly once by resuming from the cookie of the last delivered entry, with strictly increasing non-zero cookies and a stable order across repeated enumerations of the same handle; - readdirplus contract: the dirent ino equals the accompanying Entry's attr.st_ino, the Entry matches a plain lookup of the name (inode, mode, size), and delivered entries stay mapped with exactly the kernel-owned reference until forget() drops them; - ownership: plain readdir is reference-neutral (transient per-entry lookups are forgotten again), and an entry rejected by a full reply buffer (add_entry -> 0) has its readdirplus reference released immediately instead of leaking; - error contract: an error before anything was stored propagates (exercised on a resumed stream, since a fresh one begins with the internally filtered "." and ".."), an error after a delivered entry returns Ok(()) with the partial delivery; - cookies: resuming from a mid-stream cookie repositions via lseek, a cookie beyond i64::MAX terminates at EOF through the linear scan fallback, and releasedir drops the cached cookie together with the handle (the cached cookie is a one-shot token consumed by the matching resume, which the drain helper's final EOF probe does); - plus the d_type passthrough of REG/DIR/LNK entries, the empty directory, size == 0, and the no_readdir kill switch. The setup uses a plain path-based config without inode file handles, so the suite runs unprivileged like the rest of the crate's non-privileged tests. Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
The fio comparison covers data IO and create/delete metadata, but not
directory enumeration -- the READDIR/READDIRPLUS handlers, whose sync
vs async cost profile is exactly the open question of the offload
discussion. fio has no readdir engine, so drive it with a helper
instead: bench_readdir.py walks the mountpoint with os.scandir()
(getdents64) over and over, first for a 2 second ramp whose counts are
discarded, then for ${RUNTIME} seconds measured by the wall clock that
spans the whole window. It writes fio-shaped JSON (entries/s as iops
on the read side) so bench_compare.py and the CI collect/compare steps
treat it like any other workload without any special casing.
Run it between filecreate and filedelete, when the directory holds
${NRFILES} entries: the workload is self-contained in the create/
delete pair and sees a realistic directory size. Enumeration is one
strictly sequential request chain per stream -- each getdents64
resumes from the cookie of the previous reply -- so the rate is
latency-bound and does not scale with THREADS.
Classify readdir with the metadata workloads for the looser regression
threshold: it is a per-entry metadata operation like create/delete,
and its byte metrics are only approximated reply traffic anyway. On
an unprivileged host the run reports ~210k entries/s with sync and
async at parity (a 5 second smoke run on a 20k-entry directory).
Signed-off-by: Jiang Liu <gerry@linux.alibaba.com>
jiangliu
requested review from
bergwolf,
eryugey,
imeoer and
liubogithub
as code owners
October 9, 2026 08:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two commits closing the readdir coverage gap in both the test suite and
the benchmark harness:
readdir_testsmodule (10 tests) infuse-backend-passthroughthatdrives the READDIR/READDIRPLUS handlers end to end through a
persistent
opendir()handle over a real backing directory.readdirworkload for thefio comparison in
bench_sync_async.sh, driven by the newbench_readdir.pyhelper (fio has no readdir engine).Why
The existing readdir coverage stops at getdents64 buffer parsing
(
readdir_cookie_tests), theno_opendirre-seek regression (xfstestsgeneric/676), and the shallow error paths of
test_dir_operations. Theflow the kernel actually drives — persistent directory handles,
cookie-based resume, reply-batch splitting, the readdirplus attribute
and ownership contracts — had no coverage at all.
Likewise, the benchmark comparison covers data IO and create/delete
metadata, but not directory enumeration: changes to the
READDIR/READDIRPLUS handlers had no performance signal (e.g. for
evaluating an offload of these ops to a blocking pool).
What the tests pin
readdir_enumerates_all_entries_in_batchesdirent.ino == attr.st_ino; the accompanyingEntrymatches a plainlookup()of the name (inode/mode/size)readdirplus_attrs_match_lookupreaddir_releases_lookup_referencesadd_entry -> 0) has its readdirplus reference released immediately instead of leakingreaddirplus_buffer_full_forgets_undelivered./..); after a delivered entry it returnsOk(())with the partial deliveryreaddir_error_contractreaddir_resume_from_mid_stream_cookiereaddir_reports_entry_typessize == 0does not touch the directoryreaddir_empty_directoryno_readdirkill switch reports success without enumeratingreaddir_disabled_returns_emptyreleasedirdrops the handle together with the cached cookie; the released handle stops workingreleasedir_clears_the_cookie_cacheThe setup is path-based (no inode file handles), so the whole suite runs
unprivileged.
Benchmark workload
bench_readdir.pywalks the mountpoint withos.scandir()(getdents64): a 2 s ramp whose counts are discarded, then a
${RUNTIME}s window measured by the wall clock spanning the wholerun. It emits fio-shaped JSON (entries/s as
iops, approximatedreply bytes as
bw/io_bytes), sobench_compare.pyand the CIcollect/compare steps treat it like any other workload — no special
casing anywhere.
filecreateandfiledelete, when the directoryholds
${NRFILES}entries.regression threshold.
${mode}-*.jsonglob;python3was already a CI dependency.Results
cargo test -p fuse-backend-passthrough --lib(unprivileged Linuxhost): 49 passed / 12 failed — the 12 are the known pre-existing
environmental EPERM tests that require root, unchanged by this PR.
All 10 new tests pass.
cargo fmt/cargo clippyclean in the new code.RUNTIME=5,NRFILES=20000):sync 213k / async 212k entries/s — parity, as expected: one
enumeration is a strictly sequential request chain (each
getdents64resumes from the cookie of the previous reply), so therate is latency-bound and does not scale with the worker count. The
workload is the measurement baseline for future
READDIR/READDIRPLUS changes rather than a sync-vs-async finding.