Fix false 'absent' results from the path cache: infer absence only from a cached directory listing

[kandrosov]

Problem

In the v2608 HH_bbtautau production (Run3_2022EE, outputs on T3_US_FNALLPC),
law run AnaTupleFileTask ... --print-status 0 reported absent (2260/11066) while the storage
element held 10754 of the 11066 outputs. Ground truth (a gfal-ls of each of the 118 dataset
directories against the expected anaTupleFile_<i> set from the InputFileTask outputs) is
312 genuinely missing branches, not 2260. Probing GFALFileInterface.exists() on 1075 files
known to be on disk returned "absent" for 945 of them in 2.1 s with a single gfal-ls — the wrong
answers came from the path cache.

Root cause

Two different facts were stored as the same kind of cache entry — "path X exists" and "directory D
exists" — and exists() used the second as if it meant the cache knows D's content: an unknown
path whose parent directory was known to exist was reported absent, without any listing.

Directory entries are written and refreshed by events that carry no information about content:
_mark_absent_ancestors lists the parent (era) directory whenever a not-yet-created dataset
directory is probed, and set_exists then stamps every dataset directory as existing for another
validity period; filecopy marks a destination directory as existing. File entries, in contrast,
are only refreshed by listing that directory or by the job that writes the file. The two drift
apart, and once the file entries of D are gone while D's own entry is fresh, every file in D reads
"absent" — and that wrong answer suppresses the listing that would have repaired the cache.

The failure is therefore per-directory and independent of file age, which is what the data shows:
DYto2E_M_10to50 — all 49 files 99 h old, all known to the cache; TTtoLNu2Q — all 712 files
47–68 h old, none known.

Consequence is a wrong status and needless resubmission, not silent data loss: merge plans are
built from task inputs and AnaTupleFileListBuilderTask requires the complete AnaTupleFileTask
branch set, so a false absent blocks rather than drops files.

Solution

Make the listing itself a cached fact, shared through the cache server, so that correctness does
not cost extra storage access:

• set_exists(dir, items) — only ever called after a successful gfal-ls — publishes a marker key
  <dir>/.flaf_listed in the same batch as the entries and with the same validity, written first
  so it can never outlive the entries it covers.
• Absence is inferred only from a valid marker. A plain "directory exists" entry no longer implies
  anything about content.
• Because the marker is shared, one listing serves every process and every job; a job checking its
  own not-yet-produced output is answered from the cache without listing anything.
• File-level negatives are no longer published to the server — the marker covers them, while a
  published negative would outlive the file's creation by a job whose own cache update was lost.
• A failed gfal-ls is retried once before a directory negative is published: a single transient
  failure used to suppress a whole subtree for every client for a full validity period.
• filecopy records the destination directory as existing whenever it was not already known (not
  only when it was cached absent), so a directory created after its parent was listed is not
  reported absent.
• iter_valid() (the submit-time snapshot shipped to CRAB jobs, added in Add CRAB workflow support for remote LAW tasks #299) ships a marker only
  for directories listed by that process. A marker learned from the server asserts that the
  server's knowledge is complete, which does not hold for the subset of entries kept locally.

The result is also self-healing: if a job's cache publication is lost, its file becomes visible
again when the marker expires, instead of staying invisible for as long as something keeps
refreshing the directory entry.

Files changed

• RunKit/law_gfal.py — listing marker, absence inferred only from it, local-only file negatives,
  listing retry, filecopy directory bookkeeping, snapshot guard.
• test/test_path_cache.py — new: 10 regression tests over a fake gfal tree and a fake cache
  server.
• docs/troubleshooting.md — new section on a status check disagreeing with storage, how the
  cache is consulted, and the invalidate_regex pitfall that stored paths are normalised (a
  pattern containing davs:// never matches).

Testing

• Unit tests: 10 pass. Five of them fail on the unfixed code (the false-absent inference, the
  transient-listing-failure case, the directory-created-after-a-parent-listing case, the snapshot
  guard); the two that encode the load invariant — a second process and a job answer from the
  shared cache with zero listings — pass both before and after, so the fix does not trade
  storage load for correctness.
• Live production v2608, Run3_2022EE: --print-status 0 on the real command now reports
  absent (10754/11066), i.e. exactly the 312 genuinely missing branches (was 2260), in 2:50.
• Live, fresh process (what every job looks like) over the whole era: 22132 targets, 0 wrong
  answers, 0 gfal-ls calls, ~60 s — everything served from the shared cache, including 42
  files that had been produced by jobs after the last listing.
• mkdocs build --strict passes.

[kandrosov]

@cms-flaf-bot please test

[cms-flaf-bot]

pipeline#15554048 started

[cms-flaf-bot]

pipeline#15554048 passed

[Copilot]

Pull request overview

This PR fixes a correctness bug in FLAF's remote-file existence cache (RunKit/law_gfal.py) that caused law run ... --print-status to falsely report thousands of outputs as absent even though they were present on storage. The root cause was that a "directory D exists" cache entry was treated as if it meant "the cache knows D's complete content," so an unknown file whose parent directory happened to be cached as existing was reported absent without ever listing the directory. Because directory entries are refreshed by content-agnostic events (probing a parent, filecopy, set_exists), they drifted out of sync with the per-file entries, producing per-directory false negatives that also suppressed the listing that would have repaired the cache.

The fix introduces an explicit, shared "listing marker" (.flaf_listed) that is written only after a successful gfal-ls, so absence is inferred only from a genuine cached listing rather than from mere directory existence. This is framework-critical, distributed caching logic that affects existence checks, job-submission cache snapshots, and CRAB worker behavior across the whole analysis chain.

Changes:

• Absence is now inferred only from a valid LISTING_MARKER; a plain "directory exists" entry no longer implies anything about content. The marker is published in the same batch as (and written first, so it can never outlive) the entries it covers, and is shared through the cache server so one listing serves every process/job.
• File-level negatives are kept local only; a failed gfal-ls is retried once before publishing a directory negative; filecopy records a destination directory as existing whenever it was not already known; and iter_valid() ships a marker only for directories actually listed by the current process (snapshot guard added for the CRAB path from #299).
• Adds a 10-case regression test suite (test/test_path_cache.py) and a troubleshooting doc section, including the invalidate_regex path-normalization pitfall.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
RunKit/law_gfal.py	Core fix: listing marker key/helper, has_listing, marker-based absence inference in exists(), local-only file negatives, listdir retry, filecopy directory bookkeeping, and iter_valid snapshot guard with listed_dirs.
test/test_path_cache.py	New unit tests over a fake gfal tree and fake cache server covering false-absent regression, shared-listing load invariants, transient-failure handling, and the snapshot guard.
docs/troubleshooting.md	New section explaining the path cache, its self-healing behavior, and the invalidate_regex normalization pitfall.

I verified the marker ordering/validity invariants (marker written first on both client and server, so it expires first), the iter_valid snapshot guard (markers learned from the server are correctly excluded via listed_dirs), the filecopy is not True condition, the listdir retry path, the docs' path-normalization claim against pathCacheServer.standardize_path_name, the WLCGFileSystem/global.yaml configuration reference, and the CRAB load_entries+has_listing flow. All 10 tests were traced against the new code and are consistent. I found no concrete blocking defects.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.