Skip to content

fix(server): stop no-longer-true events from waking the watcher - #108

Merged
nullStack65 merged 3 commits into
release/native-0.1.10from
feat/native-0.1.10-wake-filter
Nov 19, 2026
Merged

nullStack65 merged 3 commits into
release/native-0.1.10from
feat/native-0.1.10-wake-filter

Conversation

@nullStack65

Copy link
Copy Markdown
Owner

Problem. The watcher thread (Lead's Metathread) was woken by alerts that were already false when it read them. Each thread event was judged once, when it fired, then queued into the watcher behind whatever turn it was running. Observed on 2026-11-19: a lane-ended alert for a lane that had already restarted itself, and alerts that sat in the Metathread's queue (runs 206 and 207 queued behind run 205) until long after the lane moved on. Each costs a full-context coordinator turn that learns nothing.

Fix. Two deterministic checks before any model wakes, both in the existing outbox (no new service):

  1. Judge again at delivery. threadEventStillHolds (in ThreadEventAlerts.ts) re-runs judgeThread on the lane's current shell, as of the time the event fired (so the 15-minute recent-run window does not expire while delivery waits), and keeps the alert only if the same finding, with the same signature, still holds. Dropped: the lane restarted or a newer run finished, was settled or archived, registered a PR watch or background wait, merged its PR, had its question answered, or showed new activity since a stuck alert. This extends the existing currentAlert staleness check, which only covered newer runs and archived threads.
  2. Hold while the watcher is mid-turn. busyWatcherHold holds watcher deliveries while the watcher thread is preparing/queued/starting/running, rechecking every 30 s (the flush interval), for at most 10 minutes. A queued message waits for that turn anyway, so the latency cost is at most one flush; the gain is that check 1 runs when the watcher can act. Webhook deliveries are not held.

Drops log watcher alert no longer true; dropped without waking anyone with type and kind, which is the count for the before/after measurement.

Tests. Table rows for each drop reason and the keep cases (ThreadEventAlerts.test.ts), plus the hold table (ModelAlertReactor.test.ts): 80 passing. check:changed against origin/release/native-0.1.10: format, lint, graphql and typecheck pass. knip reports only the 4 unused upstream scripts already present on this base (committed with --no-verify for that, as #98 did). AgentContextBudget.test.ts fails identically on the base head 1a634c53 (mcpToolListJsonBytes 53985 vs 53783), unrelated to this change.

Part of the agent event and messaging lane (tracking: nullStack65/closura-agent-config#873).

Work by Claude Opus 5.5 in Claude Code (T3 lane on primary-r720).

🤖 Generated with Claude Code

Deploy: fleet

nullStack65 and others added 2 commits November 19, 2026 18:19
A lane-ended, stuck, promise or needs-person alert was judged once, when it
fired, and then queued into the watcher thread behind whatever turn it was
running. By the time the watcher read it the lane had often restarted,
settled, registered a wait or had its question answered, and the coordinator
spent a full-context turn learning nothing.

Delivery now judges each thread event again, as of the time it fired, and
drops it when that finding no longer holds. While the watcher is mid-turn,
watcher deliveries are held (rechecked every 30 s, at most 10 minutes) so the
check runs when the watcher can act, instead of when the message is queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… hold

A peer read that dies (not just fails) now delivers as if the watcher were idle.
The delegated-completion tests now expect the second local alert to wait for the
watcher's queued turn, and the remote stub answers the status read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author

Bot review (round 1, Claude Fable 5.2):

Independent review of this pull request (the watcher fix that re-judges thread events at delivery and holds alerts while the watcher is mid-turn), at head 37a8a4c13ae5324f717b7a90b0701d2b338aebb6 against release/native-0.1.10 at 1a634c53. This is an automated review, not a human approval.

Verdict (my judgment): nothing here blocks the merge. I found no case where the re-judge delivers something it should have dropped, and no case where a still-true alert is dropped in the ordinary path. The two findings worth acting on are 1 (the hold misses the watcher's most common busy state) and 2 (remote read cost); both are a few lines in the same place, so I would fix them in this pull request or a direct follow-up. The rest is low severity.

What I ran: vp test run on ModelAlertReactor.test.ts and ThreadEventAlerts.test.ts in my own clone at the head: 2 files, 80 tests, all pass. I also ran a throwaway probe test (deleted, never committed) against threadEventStillHolds; its results are quoted below. I did not run a server or an end-to-end delivery.

Findings, most severe first

1. Medium, correctness gap: a watcher whose run is waiting is not held, but a queued alert is blocked behind a waiting run exactly as behind a running one.
apps/server/src/orchestration-v2/ModelAlertReactor.ts:457 leaves waiting out of BUSY_WATCHER_STATUSES, and ModelAlertReactor.test.ts:492 locks that in ("a watcher waiting on its own background work" → no hold).

  • A turn that ends with run-owned work still open is stored as waiting (RunExecutionService.ts:821).
  • isActiveRun counts waiting (CommandPolicy.ts:120), and the alert is sent with queue_after_active / mode: "queue" (ModelAlertReactor.ts:636, :645), so it queues behind that run (Orchestrator.ts:7498).
  • The queued run cannot start while any run is waiting: startNextQueuedRun returns early on isActiveRun (Orchestrator.ts:2448-2455), and the canStartQueuedRun query excludes 'waiting' (ProjectionStore.ts:6251).

Failure scenario: the Lead thread ends its turn waiting on delegates or a background command, so its run is waiting. A lane fails; the lane-ended alert is judged true, is not held, and queues behind the waiting run. Two minutes later the lane restarts itself. When Lead's wait ends, the queued alert runs and Lead reads a stale alert. That is the failure this pull request sets out to remove, and "waiting on delegates" is where a coordinating watcher spends much of its time.

Suggested fix, which differs by path:

  • Remote watcher: hold while the thread read's activeRunId is set. It is computed with the same isActiveRun predicate the queue uses (latestActiveRun, ThreadManagementService.ts:603-609, used by threadDetail in OrchestratorMcpService.ts), so add it to WatcherThreadStatus (ModelAlertReactor.ts:727). Do not simply add waiting to the status set here: over this read, status is also waiting for a thread with no active run whose background work holds completion (threadStatus, OrchestratorMcpService.ts:934-942), where a message would start at once.
  • Local watcher: add waiting to the status set. The local shell's status is the latest run's own status, and its activeRunId cannot be used because it excludes a waiting run (isInterruptibleRunForShell, ProjectionStore.ts:1993-1995).

Limits of my claim: read from code, not run end-to-end. Also, a wait can outlast the 10-minute cap, so this narrows the gap and does not close it.

2. Medium, cost: every watcher delivery to a remote watcher now makes a t3_thread_read peer call, and a held entry repeats it every 30 seconds.
ModelAlertReactor.ts:842-861, called at :949.

  • The remote watcher is the default case (Lead's thread on the laptop, alerts from the other machines), so this is one extra peer round trip before every send, where before only usage-limit alerts read the watcher.
  • A held entry re-reads on each recheck: up to about 20 reads per entry over the 10-minute cap. Entries are read independently (concurrency: 4, :985), so five alerts held behind one long watcher turn cost about 100 reads of the same thread for one answer.
  • limit: 1, runLimit: 1 trims the response, but the peer still loads the readable thread (shell, runs, provider turns, runtime requests) before paging.
  • A usage-limit alert whose holdUntil returns null reads the same thread twice in one deliver (:867-914, then :949).
  • The read and the send share the single 30-second deliver timeout (:963).

Suggested fix: read the watcher's state once per flush (it is the same watcher for every entry) and pass it to both holdUntil and busyWatcherHold. That is smaller than the current code, since the two peers.call blocks collapse into one.

3. Low, correctness: a drop at delivery keeps the dedupe signature, so an alert dropped for a state that then reverts is never sent for that run.
recordThreadEvents clears reported only when a trigger that evaluates the kind finds nothing; lane-ended is evaluated only on a run ending, and its signature is the run id. One concrete path is the snooze check: hasWakeHandle compares snoozedUntil with nowMs (ThreadEventAlerts.ts:106-107), and the re-judge passes the fire time as nowMs (:355). One timestamp serves two purposes that want different clocks: the recent-run window (fire time is right) and the snooze (now is right).

Probe result: a lane that completed with an open pull request fires lane-ended; a 5-minute snooze is set afterwards; judged at a delivery 20 minutes later, threadEventStillHolds returns false although the snooze has expired. The alert is dropped and the lane stays idle with an open pull request and nothing to wake it, with no further alert. The same shape applies to a watch or background task registered and removed between fire and delivery. It needs a delayed delivery (a hold, retries or an offline watcher), so it is rare. Smallest fix: pass the real current time for the wake-handle check only.

4. Low, cost and duplication in currentAlert. ModelAlertReactor.ts:788-798.

  • It reads the same thread shell a second time (first read inside stillCurrent, :763) and reads settings again.
  • judgeContext runs the full getOpenRemoteDelegatedTasks scan (:813) per entry per attempt, including for needs-person, whose count is unused: the inline mapping passes "run-ended" for it, so the request/inbox short-circuit at :808-810 never applies. The sweep was written to scan once for all threads; held entries now rescan every 30 seconds each.
  • The inline mapping (stuck → sweep, else run-ended) is a second copy of REJUDGE_TRIGGER (ThreadEventAlerts.ts:330-335) and already disagrees with it for two kinds. It is harmless today. Exporting one mapping and using it in both places is smaller and removes the needless scan.

5. Low, maintainability: the signature is recovered by rebuilding the id string. ThreadEventAlerts.ts:360-365 must stay in step with the format at :308; a format change would silently drop every queued event, since a mismatch reads as "no longer true". A comment at :308 pointing at the rebuild, or a round-trip test (present today only indirectly), would guard it. Related and unverified: outbox entries written by a build older than the commit that added the timestamp to the id (227b12e067, already in the base) would now be dropped on upgrade. I do not know whether any such entries exist; I expect none.

6. Low, behaviour the description and doc understate.

  • "At most one flush" of added latency holds only when the watcher goes idle. A watcher with back-to-back turns holds every alert the full 10 minutes and then sends it behind the queue anyway, so a needs-person alert reaches Lead later than before this change. The hold also applies to model alerts and usage-limit incidents, not only thread events. I think the trade is right; the doc should say it.
  • A wake-triggered flush shortly before a hold is due restarts the 30-second timer (:1317-1325), so a recheck can take close to 60 seconds, not 30.
  • When several held alerts come due in one flush, up to four are sent together; the first starts a watcher turn and the others queue behind it, judged at send time. Bounded to one watcher turn.
  • docs/operations/watcher-thread.md:13 says a lane that "was replaced" is dropped. For thread events the replaced-by-a-sibling check never runs (about is undefined for them at :784-786), so only a restart of the same lane is detected. Either drop "or was replaced" or wire the check.

Do the tests exercise the behaviour?

Partly. The two pure helpers are covered; the wiring is not.

  • Nothing tests deliver or currentAlert: the holdUntil ?? busyWatcherHold order, the remote status read and its null-on-error path, the inline trigger mapping, or the drop log. A fake peer with one held-then-sent and one dropped delivery would cover most of it.
  • ThreadEventAlerts.test.ts:385-390, "still holds when delivery waited past the recent-run window", does not exercise the window. The function takes no delivery time, so its first assertion is the same as the "nothing changed" row at :339. A real version would re-judge a lane whose run ended more than 15 minutes before a later clock. The second assertion (unparseable time keeps the alert) is useful.
  • needs-person and promise-without-wake have only drop rows (:367, :374), as does lane-ended with an open pull request. A threadEventStillHolds that always returned false for those kinds would pass. My probe confirms the keep cases do hold today (request, inbox, promise, open pull request, and the stuck orphan-wait signature), so this is missing coverage, not a bug: add those five rows.

Excess complexity

Not over-built, in my view. Re-using judgeThread for the delivery check is the right small design, and the hold is about 20 lines. The smaller version of the same change is: one trigger mapping (finding 4), one watcher read per flush shared with holdUntil (finding 2), and hold on activeRunId in place of a status set (finding 1).

Checked and found sound

  • Judging at the fire time keeps a true alert through a long wait: the windows are measured from occurredAt, which is taken after the original judgment.
  • Restart, newer run, settle, archive, merge, answered request and new activity all drop, and each either re-fires on its own trigger or is correctly final. A dropped stuck alert re-fires through the sweep.
  • holdUntil keeps precedence and skips the extra read when it holds. Retries, backoff and the 7-day outbox expiry are unchanged by a hold; a hold does not count as a failed attempt.
  • An unreadable remote watcher means no hold, so an offline peer cannot starve alerts; the send then fails and retries as before.
  • Not passing usageLimitReported at the re-judge only makes it more permissive.

Unverified: end-to-end delivery against a live remote watcher; finding 1 is traced through the code, not reproduced on a running server.

- A watcher whose run is waiting also queues a sent alert, so it is held too.
- One watcher read per flush is shared by every due alert and by the usage-limit hold.
- currentAlert re-judges with the shared REJUDGE_TRIGGER map.
- Keep rows for needs-person, promise-without-wake and lane-ended with an open PR.
- Doc states the hold covers every alert kind and the 30-60 s recheck.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nullStack65

Copy link
Copy Markdown
Owner Author

Review fixes for the round-1 bot review above (head 357fdab3216564d9cbfb20c534835e6da33ecb94, on top of bb200804, which fixed the two DelegatedCompletionDelivery tests and made a failed peer read never block an alert).

  1. Fixed: a waiting watcher is now held. waiting joins the busy set. The test row at ModelAlertReactor.test.ts now expects a hold. As noted in the review, a wait longer than the 10-minute cap still delivers, by design.
  2. Fixed: one watcher read per flush. flush reads the watcher once (Effect.cached) and shares that view with every due alert and with the usage-limit hold, which no longer does its own read. An unreadable watcher is treated as idle.
  3. Not fixed here (low): a drop keeps the dedupe signature, and the snooze check is judged at fire time. Listed as a follow-up.
  4. Partly fixed: currentAlert now uses the exported REJUDGE_TRIGGER, so the two kinds that disagreed now match. The second shell read and the remote-task scan per entry remain (follow-up).
  5. Not fixed here (low): signature recovered by rebuilding the id prefix. Follow-up.
  6. Fixed: docs/operations/watcher-thread.md drops the "replaced" claim and states that the hold covers every alert kind, that a dropped event stays reported, and that a recheck takes about 30–60 s.

Tests: keep rows were added for needs-person, promise-without-wake and lane-ended with an open PR. The three suites (ThreadEventAlerts, ModelAlertReactor, DelegatedCompletionDelivery) pass, 149 tests. Server typecheck is clean. No wiring test for deliver yet; not verified against a live remote watcher.

@nullStack65
nullStack65 merged commit 79162c0 into release/native-0.1.10 Nov 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant