Skip to content

fix(performance): retention keeps a monitor run this store never examined - #1455

Merged
Juliusolsson05 merged 10 commits into
integration/batch-2026-09-27-rfrom
fix/monitor-unexamined-run
Sep 27, 2026
Merged

Juliusolsson05 merged 10 commits into
integration/batch-2026-09-27-rfrom
fix/monitor-unexamined-run

Conversation

@Juliusolsson05

@Juliusolsson05 Juliusolsson05 commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1453.

Plan (first commit): docs/plans/2026-09-27-monitor-unexamined-run.md.

Problem

MonitorHistoryStore.maintain() deleted every run folder its in-memory maps did not know. Those maps are filled only by startup indexing. A run created later, by a second store sharing the folder (--packaging-smoke skips the single-instance lock), was never examined, read as empty, and deleted on the next maintenance pass. That is the "never seen means empty" shape the q109/q115 rule forbids.

What merges

  • examinedRuns: retention deletes a run as empty only if startup indexing examined it. A run that appeared later is unknown and waits for the next start to index it.
  • Unknown is never empty (reviews a+b), each marking the run unknown so retention leaves it:
    • an incident file that exists but cannot be read, parsed or trusted (it used to read as []);
    • a tier stat failure other than ENOENT;
    • a tier file with unparseable lines.
  • examinedRuns is forgotten when a run is deleted, so a name another store recreates is unexamined again.
  • Another live store's files are never acted on from a stale index (review b rounds 2-3): each foreign tier and incident file's size and mtime are recorded at indexing. Expiry, compaction and every incident rewrite (retention and the global incident limit) first check the file is unchanged; a changed file makes the run unknown.
  • Retention keeps any run with a file touched within the retention window (touchedSince; any failure counts as touched), so a run examined while empty and filled afterwards is kept.

Residuals

  • The capacity budget may still remove unexamined or unknown runs, as it already may for unindexedRuns (the 128 MiB ceiling wins).
  • Cross-process check-then-act (review a, round 2): another store's write can land between retention's final touchedSince() and its rm(). It needs two processes sharing the folder (--packaging-smoke only) and a write in that sub-millisecond window. Closing it would need a cross-process lock on the monitor folder. Accepted by B6 (owner proxy) as a residual; no lock.
  • Once an expired run's last file is removed, its fresh folder mtime keeps the empty folder for one more retention window.

Tests

  • Real files and the real clock, with each fixture's files and folder aged past retention (review c), so every guard is exercised on its own:
    • a run appearing after indexing is kept, examined by a restart, and removed once expired (incident file, then its folder);
    • unreadable incidents;
    • recreated after deletion;
    • an ELOOP tier link;
    • unparseable tier lines;
    • empty at examination and filled afterwards;
    • two live stores: a stale tier expiry, content appended then compaction, and a fresh incident then rewrite.
  • Mutations, each killed:
    • the examinedRuns guard dropped;
    • never populated;
    • kept after delete;
    • unknown incidents unprotected;
    • unparsed ignored;
    • no touched check.
      The ELOOP case has two guards (noted in its test).
  • Merged main (fix: one bad record no longer fails the whole set (C5 rows 8–12, #1251) #1411 incident refusals): both sets of unknown-is-protected rules are kept. Main's refused and foreign incident runs and its set-aside scan sit alongside this branch's examinedRuns, touchedSince and foreign-file fingerprints. Main's rewrite path in writeRunIncidents is behind foreignChanged too; the incident-guard and examined-guard mutations still fail on the merged code.
  • Main's fix: one bad record no longer fails the whole set (C5 rows 8–12, #1251) #1411 guards re-pinned after the merge (B6 check 2110, test-only): touchedSince kept fix: one bad record no longer fails the whole set (C5 rows 8–12, #1251) #1411's fresh, 1970-clock fixtures on its own, so three of main's guards had lost their failing tests. The set-aside run, the foreign-only run and the listing-failure run now use the real clock and aged() fixtures. Each fails without its guard: foreignIncidents.has, refusedAsideRuns.has, and the unindexed mark on a failed listing.
  • Two-store carried rows: another run's aged, unchanged file mixes an expired readable row with a newer build's row. Retention rewrites it, carries the unrecognised row and keeps the run. The test fails when the rewrite drops carried rows.
  • Gate: npx tsc -b is clean; src/main/performance passes, 13 files / 82 tests at c77a0e61.

🤖 Generated with Claude Code

Juliusolsson05 and others added 2 commits September 27, 2026 10:25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
)

A run folder created after startup indexing (a second store sharing the
folder) was absent from the index, read as empty, and deleted by the next
maintenance pass. Retention now deletes only runs it examined.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05 Juliusolsson05 added type:bug Something works wrong class:C3-silent-failure The app knows it failed and does not say sev:P3 Minor labels Sep 27, 2026
Juliusolsson05 and others added 6 commits September 27, 2026 12:05
…ted run is no longer examined (#1453, #1455 review a)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eeps a run touched within the window (#1453, #1455 review b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 review a round 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd is tested on its own (#1455 review c)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n file that changed since indexing (#1453, #1455 review b round 2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e's changed incident file (#1455 review b round 3)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05

Copy link
Copy Markdown
Owner Author

Review disposition (head 39174f1)

Reviewer Verdict Findings and disposition
a r1 FIX Unreadable incidents counted as empty; the examined set outlived a deleted run: fixed (8068967)
a r2 FIX Cross-process check-then-act window: accepted residual (below)
b r1 FIX A tier stat/parse failure counted as empty; a run examined empty and filled later: fixed (92295a1)
b r2 FIX A stale index expired, compacted or rewrote another live store's files: fixed (afb2a08; a fingerprint check on every foreign-file action)
b r3 FIX The global incident limit bypassed that check: fixed (39174f1; the check lives inside writeRunIncidents)
c r1 FIX A fake clock let touchedSince shadow every guard in tests: fixed (2880b8d; real clock with aged fixtures, so every guard is killed on its own)
c r2 MERGE-READY (2880b8d)

Accepted residual (B6, owner proxy, 2026-09-27): another store's write can land between retention's check (unchanged / touched) and its rm or rewrite. It needs two live processes sharing one data folder (only --packaging-smoke skips the single-instance lock) and a sub-millisecond window. Closing it would need a cross-process lock on the monitor folder, which is not added.

Other residuals:

  • The capacity budget may remove unexamined or unknown runs (the 128 MiB ceiling wins).
  • Once an expired run's last file is removed, the empty folder survives one more retention window.

Gate at 39174f1: npx tsc -b clean; src/main/performance 66 tests. The review cap is reached, and B6 is inspecting.

Juliusolsson05 and others added 2 commits September 27, 2026 12:59
…ed-run

Keeps both sets of unknown-is-protected rules: main's refused / foreign
incident runs and set-aside scan, and this branch's examinedRuns,
touchedSince and foreign-file fingerprints; main's rewrite path in
writeRunIncidents is behind foreignChanged too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sted alone

After merging main, touchedSince kept #1411's fresh 1970-clock fixtures on
its own, so the foreignIncidents, refusedAsideRuns and listing-failure
unindexed guards lost their failing tests. Those tests now use the real
clock and aged() fixtures; each fails without its guard. Adds the two-store
carried-rows retention test (B6 check 2110).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05
Juliusolsson05 marked this pull request as ready for review September 27, 2026 21:07
@Juliusolsson05
Juliusolsson05 changed the base branch from main to integration/batch-2026-09-27-r September 27, 2026 21:32
@Juliusolsson05
Juliusolsson05 merged commit 2e539f6 into integration/batch-2026-09-27-r Sep 27, 2026
2 checks passed
@Juliusolsson05
Juliusolsson05 deleted the fix/monitor-unexamined-run branch September 27, 2026 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

class:C3-silent-failure The app knows it failed and does not say sev:P3 Minor type:bug Something works wrong

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(performance): a second monitor store can delete a run folder the first never examined

1 participant