Single-pass MemoryFileSystem.find() to avoid quadratic listing

[Nas01010101]

What

Give MemoryFileSystem a single-pass find() so that listing a tree is linear in the number of stored paths instead of quadratic.

The problem

MemoryFileSystem keeps every path in one flat dict (self.store). The inherited AbstractFileSystem.find() walks the tree by calling ls() once per directory, and MemoryFileSystem.ls() scans the entire store on every call:

for p2 in tuple(self.store):
    if p2.startswith(starter):
        ...

So find() over a tree with D directories and N stored paths does O(D * N) work. For a store with many directories this dominates anything built on find().

The fix

Override find() on MemoryFileSystem with one pass over the flat store (the store already contains every path, so no per-directory recursion is needed). The override reproduces the base semantics exactly: maxdepth, withdirs (including implied ancestor directories and explicitly-created empty pseudo_dirs), detail, the file-as-path case, and inclusion of the search root itself when withdirs is set. It is additive and touches no existing code path.

This speeds up everything that delegates to find(): find, du, glob, and expand_path. walk() is left unchanged (it still calls ls() per directory and remains O(D * N)); it could get the same treatment in a follow-up if wanted.

Correctness

A new test, test_find_does_not_scan_per_directory, asserts the override never calls ls() (so the quadratic factor is gone), and test_find_matches_generic checks the override against the generic AbstractFileSystem.find() for equality across the full matrix of roots (including "", a file path, and a missing path), maxdepth in {None, 1, 2, 3}, withdirs, and detail. Comparing against the base implementation directly means this test also guards against future drift: if the generic find() semantics change, the test fails.

The original development run also diff-tested the override against the base across 13,560 randomized configurations with zero mismatches.

Performance

MemoryFileSystem.find("/data"), old generic ls()-per-directory vs the single-pass override, 50 files per directory, outputs verified identical:

files	dirs	old (ms)	new (ms)	speedup
2,500	50	6.1	1.2	5.2x
5,000	100	20.1	2.7	7.5x
10,000	200	80.1	5.4	14.7x
20,000	400	285.9	12.3	23.3x

Old time grows quadratically, new time grows linearly, so the gap widens with the number of directories.

Notes

• BSD-3, no behavioural change to the public API.
• docs/source/changelog.rst has an entry under a new Dev section; the #XXXX PR reference will be filled in once this PR has a number.

[martindurant]

Your implementation is probably fine (I haven't looked in detail yet), but is there really a usecase for >>1000 files in a memoryFS?

[Sanjays2402]

Could this snapshot self.store.items() before iterating? store is global across all MemoryFileSystem instances, so a concurrent create/delete now raises RuntimeError: dictionary changed size during iteration; ls() avoids that by iterating over tuple(self.store).

[Nas01010101]

Your implementation is probably fine (I haven't looked in detail yet), but is there really a usecase for >>1000 files in a memoryFS?

My impression was just that if you use memory:// as the backend for something chunked, like a zarr array where every chunk is its own key, a 200x200 grid is already 40k keys, and then glob() and du() pay the same cost since they go through find(). below ~1000 paths it is sub-millisecond and nobody would notice An assumption about how people could use memory:// though, I can close it tbh.

[martindurant]

like a zarr array where every chunk is its own key

zarr should never need to list a chunk directory, the chunk names are deterministic. Maybe you have a point, though, if listing the top-level directories, which would be the array containing group level.