Skip to content

test: tolerate absent triage wake queues - #2845

Open
karotkriss wants to merge 3 commits into
kunchenguid:mainfrom
karotkriss:fm/fm-2628-triage-test
Open

test: tolerate absent triage wake queues#2845
karotkriss wants to merge 3 commits into
kunchenguid:mainfrom
karotkriss:fm/fm-2628-triage-test

Conversation

@karotkriss

@karotkriss karotkriss commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Fixes #2628

Additional commits from the validation pipeline

CI on this branch surfaced two failures unrelated to the queue-read fix described below. The validation pipeline diagnosed and fixed both, so this branch also carries:

  • tests/fm-pi-watch-extension.test.sh: the session-transition case identified its arm child by PID alone, so under CI load a recycled PID could make a dead generation look alive, or make one live process count twice. Replaced with a unique per-arm lifetime marker that the child creates and removes on exit, preserving the overlapping-generation detection the case exists to prove. This is a pre-existing, CI-only, load-sensitive flake, not a consequence of the change below.
  • A one-line follow-up removing an apostrophe from a comment introduced by that fix, which stock macOS Bash 3.2 misparses inside a quoted heredoc nested in command substitution.

tests/fm-watch-triage.test.sh is untouched by both.


Intent

Fix GitHub issue #2628: the fm-watch-triage test suite's exited-declared-pause case fails when state/.wake-queue is absent.

Reported defect: running tests/fm-watch-triage.test.sh reaches the exited-declared-pause case without state/.wake-queue existing. Two awk reads then fail with "can't open file" for exited-declared-pause/state/.wake-queue, and the suite reports 'not ok - dead-agent declared pause flooded stale wakes across six unchanged polls' with no number in the message. The reporter's stated expectation: the test should either create the bounded paused-recheck wake queue it asserts against, or handle an absent queue without producing an empty integer comparison. The reporter also scoped this as reproducing independently of the fm-batch-tooling changes, to be fixed on its own.

Root cause established by direct reproduction, not inference: awk aborts on a missing input path before its END rule runs, so 'awk ... END { print n + 0 } $state/.wake-queue' prints nothing and exits 2 instead of printing 0. The count variable collapses to the empty string, and the following '[ "$wakes" -le 1 ]' fails as a bash 'integer expression expected' error rather than a comparison. That error takes the '|| fail' branch, so an absent queue is misreported as a wake flood of an unprintable count - masking the real contract breach that the very next grep names correctly ('did not use the bounded paused recheck'). A watcher that correctly absorbs every poll and queues nothing never creates .wake-queue at all, so the file is legitimately optional here.

Chosen resolution and why: of the reporter's two options, take the second (tolerate an absent queue) rather than pre-seeding a fixture queue. Seeding a queue file would pre-empt the very thing the case measures - how many stale wakes the watcher itself enqueued - whereas absent-tolerance is already this suite's own established convention for reading a possibly-absent queue: the drain-count assertions at the end of the same file use 'awk ... "$state/.wake-queue" 2>/dev/null || echo 0', and the sibling fm-procevent suites read the queue with '2>/dev/null' for the same reason. The fix therefore reuses the existing in-repo idiom instead of introducing a new helper or abstraction.

Scope decision: the ticket names the two reads in the exited-declared-pause fixture, but the identical unguarded pair appears again later in the same test function for the live external-decision-gate fixture, whose queue an acknowledged drain can also leave behind. All four reads were fixed together, since patching only the two the ticket names would leave the same latent crash in a sibling read one screen down. Deliberately NOT changed: the awk read of state/.watch-triage.log later in the file, which looks similar but reads a file that same test creates itself with 3001 lines and so can never be absent - guarding it would be unrelated scope creep.

Behavior preserved, verified by a focused before/after harness: with the queue absent the case now reports the accurate 'did not use the bounded paused recheck' diagnostic instead of two bogus integer-comparison failures; with a genuine two-wake flood present the guard still goes RED with its real counts (2), so the assertion's discriminating power is intact and this is not a weakened test. Also confirmed the exact reported symptom reproduces before the change: the failure line prints with the count missing, verbatim as filed.

Constraints honored: test-file-only change, no production watcher or classifier behavior touched, no new dependency or abstraction added, and an explanatory comment left at the first site so a future reader knows why the queue is optional and what the unguarded form used to misreport. Full suite verified green locally (60 ok, 0 not ok) and shellcheck -x clean on the changed file.

Deliver as a pull request that closes issue #2628 and nothing else.

What Changed

  • Treat a missing optional .wake-queue as zero wakes in declared-pause and live decision-gate assertions.
  • Preserve accurate wake-count comparisons and failure diagnostics when the watcher queues nothing.
  • Document why the queue may legitimately be absent.

Risk Assessment

✅ Low: Captain, this well-bounded test-only change correctly treats an absent optional wake queue as zero while preserving detection of genuine wake floods and required queue contents.

Testing

After reviewing the base-to-target test-only diff, I ran the canonical affected watcher suite and a focused before/after CLI harness. The suite passed, the reported empty-integer failure reproduced with the base expression, the fixed expression returned zero for an absent queue and preserved the accurate contract diagnostic, and a real two-wake flood remained detectable with its correct count.

Evidence: Targeted fm-watch-triage test transcript

Source: Targeted fm-watch-triage test transcript

FM_TEST_BEGIN 2026-08-23T07:41:08Z tests/fm-watch-triage.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - signal_reason_is_actionable: benign absorbed, captain verbs and coalesced batches surfaced
ok - stale_is_terminal: terminal status surfaces, non-terminal and no-status are benign
ok - scan_captain_relevant_statuses lists only captain-relevant statuses
ok - classifier primitives: keyed decisions and activity phases, captain relevance, window-to-task, and overrides
ok - crew_is_provably_working: only working+run-step/pane is provable; idle/finished/parked/failed/unknown surface
ok - status_is_paused: only the leading paused verb matches, paused is not captain-relevant, and the two declared-wait verbs stay separable
ok - crew_absorb_class: working/paused/none from one read; crew_is_paused and crew_is_provably_working agree
ok - crew_worktree_written_since: real writes are evidence; no worktree, no anchor, quiet trees, .git churn and a mate's own home are not
ok - an empty FM_WORKTREE_WRITE_PRUNE widens the probe to the whole depth-bounded tree instead of disabling it
ok - an empty FM_WORKTREE_WRITE_PRUNE exported into the environment prunes nothing, widening the probe
ok - the worktree write probe is wall-clock bounded, and hitting the bound reads as no write evidence
ok - signal_crew_provably_working: benign only when every referenced crew is provably working
ok - a secondmate's status signal is never absorbed as provably working; crewmates are unaffected
ok - a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)
ok - a bare turn-end whose crew is provably working (busy pane) is absorbed
ok - a bare turn-end whose crew is not provably working is surfaced (the swallowed-finish fix)
ok - a no-verb working: note whose crew is idle with no running pipeline is surfaced
ok - a secondmate's status note surfaces even while its own agent is busy
ok - a self-announced close never wakes its own home, and the next real note still does
ok - captain-relevant signal is surfaced (queue + exit) and marked surfaced
ok - a stale pane sitting on a terminal status is surfaced (queue + exit)
ok - a stale terminal-looking status is overridden and absorbed while a run is actively working, then wedge-escalated
ok - provably-working non-terminal stale is absorbed on first sight, then wedge-escalated past the threshold
ok - consecutive wedge escalations on the same pane accumulate and demand deep inspection at the threshold
ok - a pane becoming active again resets the consecutive wedge-escalation counter
ok - a busy worker below the turn-age bound remains working with no escalation
ok - a busy worker with a stable pane hash still escalates once its completed-turn age reaches the bound
ok - a busy worker whose pane hash changes every poll still escalates once its completed-turn age reaches the bound
ok - touching a busy worker's completed-turn marker resets the age and prevents an old-age escalation
ok - repeated busy turn-age escalations reuse the existing escalation counter and demand deep inspection at the threshold
ok - the production default busy-turn-age bound is 3600s (5min under does not wedge, 66min over does)
ok - a busy pane under a declared pause is rechecked on the long cadence, and lifting the pause restores the wedge escalation
ok - a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)
ok - a declared pause is absorbed on first sight, then re-surfaced as a recheck past the threshold, never wedge-escalated
ok - exited declared-pause and captain-held panes use bounded pause cadence while a live decision gate still surfaces once
ok - a declared paused secondmate re-surfaces on the bounded normal-mode cadence
ok - a captain-held secondmate re-surfaces on the bounded normal-mode cadence
ok - a non-paused secondmate retains normal stale suppression
ok - a resumed secondmate clears pause and stale tracking before stale exemption
ok - unchanged stale hashes reclassify when a crew enters or leaves pause
ok - a declared pause is periodically rechecked against authoritative active-run state
ok - a paused status overridden by authoritative working preserves its wedge timer and escalates
ok - matching non-terminal stale suppressors repair missing or corrupt stale-since timers
ok - a quiet pane writing its own worktree is deferred, while one writing nothing still wedge-escalates on the unchanged schedule
ok - a write deferral re-surfaces once on the bounded pause cadence, so a churning worktree cannot stay invisible
ok - a secondmate's own home supervision churn is not crew write evidence, so a pane recording that home keeps the unchanged escalation schedule
ok - an idle-window timer repair drops a finished write-deferral chain, so the next deferral gets a fresh re-surface window
ok - both first-sight paths through a captain-relevant status drop a finished write-deferral chain with the idle window
ok - triage log capping handles wc byte counts with leading spaces
ok - a captured process-event result wakes a healthy watcher proactively, with no manual drain
ok - an unacknowledged process-event result re-drains until handling is acknowledged
ok - complete process-event queue keys map to distinct seen markers
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 9599.1787471005.3DU48E
ok - queue revalidation, proactive output, and marker commit serialize with drain
/home/cmckay/.no-mistakes/worktrees/80aee654c94f/01M0PRXSB2MWVBYH1N7VHSMK17/bin/fm-push-transition-lib.sh: line 96: echo: write error: Broken pipe
tests/wake-helpers.sh: line 286: 16538 Killed                     PATH="$dir/fakebin:$PATH" FM_HOME="$dir" FM_PROCEVENT_CLAIM_ROOT="$dir/claims" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out"
tests/wake-helpers.sh: line 286: 20239 Killed                     PATH="$dir/fakebin:$PATH" FM_HOME="$dir" FM_PROCEVENT_CLAIM_ROOT="$dir/claims" FM_CREW_STATE_BIN="$dir/fakebin/fm-crew-state.sh" FM_POLL=0.2 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out"
ok - surfacing failures replay until post-handling acknowledgement
ok - marker failure exits through the shared wake owner, releases its lock, and replays later
ok - a heartbeat with no captain-relevant change is absorbed and backs off the cadence
ok - heartbeat backstop fail-safe surfaces a captain-relevant status the per-wake path missed
ok - the liveness beacon stays fresh while the watcher absorbs benign wakes (fm-guard never false-alarms)
ok - with .afk present the watcher reverts to one-shot so the daemon owns triage (no double-triage)
ok - AFK changed paused panes hand off plain stale identities for daemon-owned pause triage
FM_TEST_END 2026-08-23T07:43:49Z tests/fm-watch-triage.test.sh exit=0 duration_ms=160198 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=160244
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=1 duration_ms=160198 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-watch-triage.test.sh duration_ms=160198
Evidence: Missing-queue before/after and genuine-flood evidence

Source: Missing-queue before/after and genuine-flood evidence

CASE 1: base expression with an absent optional queue
awk: fatal: cannot open file `/tmp/tmp.lnTFGMS0wd/state/.wake-queue' for reading: No such file or directory
awk_status=2 captured_count=<>
bash: line 14: [: : integer expected
not ok - dead-agent declared pause flooded  stale wakes across six unchanged polls

CASE 2: fixed expression with an absent optional queue
wakes=0 bare=0
not ok - dead-agent declared pause did not use the bounded paused recheck
integer_comparisons=clean

CASE 3: fixed expression with a genuine two-wake flood
wakes=2
not ok - dead-agent declared pause flooded 2 stale wakes across six unchanged polls

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Inspected git diff 8714c9a78c1b4355782fcb9ce1ccf14337478268..e328ebb9a0b8f31d2d722463bc241eea70c47a8e -- tests/fm-watch-triage.test.sh
  • Ran bin/fm-test-run.sh tests/fm-watch-triage.test.sh
  • Executed a focused shell harness reproducing the base missing-file failure, the fixed absent-queue behavior, and a genuine two-entry flood
  • Ran git status --short to confirm testing left no transient worktree changes
✅ **Document** - passed

✅ No issues found.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

…queue

The exited-declared-pause case counts queued stale wakes by handing
state/.wake-queue straight to awk. A watcher that queues nothing never
creates that file, and awk aborts on a missing path before its END rule
runs, so the count collapses to the empty string. The next comparison
then fails as an integer-expression error and surfaces as a wake flood
with no number, hiding the real contract breach the following grep names.

Read the queue the way the drain-count assertion at the end of this file
already does: silence awk's open error and default an absent queue to
zero. Applied to all four counts in this case, including the live
external-decision gate pair whose queue an acknowledged drain can also
leave behind. An absent queue now reports "did not use the bounded
paused recheck", while a genuine flood still fails with its real count.

Fixes kunchenguid#2628
@greptile-apps

greptile-apps Bot commented Aug 23, 2026

Copy link
Copy Markdown

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Reviews (3): Last reviewed commit: "no-mistakes: apply CI fixes" | Re-trigger Greptile

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

Scheduled 3:10am PT 8/23 pass. Main reconfirmed 8714c9a78c1b4355782fcb9ce1ccf14337478268. VISION.md read in full from that SHA.

VISION (inspected tests/fm-watch-triage.test.sh exited-declared-pause / live-gate awk reads, and the extra tests/fm-pi-watch-extension.test.sh session-transition lifetime-marker hunk). Per-rule: honest interface aligns (an absent optional .wake-queue must not be reported as a wake flood of an unprintable count); scripts own the mechanics aligns (reuse of this suite's own 2>/dev/null || echo 0 idiom); field incidents become coverage aligns for the named test; scope mixed — issue #2628 is the four unguarded reads, but HEAD also carries an unrelated CI-flake fix in tests/fm-pi-watch-extension.test.sh. Production watcher/classifier behavior is untouched.

Class: corrective (test-only). The named fix treats a missing optional queue as zero wakes and keeps genuine-flood counts discriminating. Extra CI-fix commits are also test-only.

Security: none. Test fixtures only. No .github files.

Overlap / HOLD: not spawn/teardown, no herdr.sh / fm-spawn.sh. Same-file overlap with #2839 on tests/fm-pi-watch-extension.test.sh (different hunks: lifetime marker vs OpenCode expect_code). Issue #2628 firstmate note: same test function as #2713 — coordinate; #2713 is still OPEN ready-for-pr. Neither is a standing hold pair.

CI / NM: HEAD a3c9584dcaf1b2a07295e51813294595da51ddcb. MERGEABLE / CLEAN, ahead 3 / behind 0. Body no-mistakes-pipeline-attestation:v1 names e328ebb9a0b8f31d2d722463bc241eea70c47a8e (the first commit), not THIS HEAD. Two later no-mistakes: apply CI fixes commits (09a0cd39, a3c9584d) moved HEAD without a matching attestation. Body-compliance FAILURE without matching attestation = NM red, even though GitHub "PR must be raised via no-mistakes" shows SUCCESS (runs 32628496212, 32629332941). CI run 32628496185 all SUCCESS. Greptile SUCCESS — not a gate.

Workflows: already approved (CI completed SUCCESS on this HEAD). Run IDs: 32628496185 (CI), 32628496212 (Require no-mistakes), 32629332941 (Require no-mistakes). No pending first-time-fork approval.

Land-eligible rec: NO (NM attestation does not match THIS HEAD; extra out-of-issue test file). Captain-flag NOW: no. Author can re-run no-mistakes so the body attestation matches a3c9584d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fm-watch-triage declared-pause test fails when .wake-queue is absent

2 participants