fix(watchdog): alert-framing — page once per wedge episode, not per kickstart - #602
fix(watchdog): alert-framing — page once per wedge episode, not per kickstart#602EtanHey wants to merge 4 commits into
Conversation
…ickstart The throughput-watchdog alerted (:3847/notify + osascript) on EVERY kickstart. During a chronic wedge sawtooth (watcher re-wedging on the live 1.4.3 apsw bug, watchdog recovering each cycle) this paged the operator every ~10min. orc alert-framing law (stalker postmortem): operator pings = state-CHANGES only, never heartbeats. Fix: episode-based alerting. Page on the FIRST wedge of an episode only; the latch clears after a SUSTAINED healthy run (_EPISODE_RESET_TICKS consecutive progressing ticks), so the NEXT genuine wedge pages again but a sawtooth does not. Recovery (kickstart) still runs every tick — only the paging is throttled. Test: test_alert_framing_pages_once_per_episode_not_per_kickstart drives a full sawtooth — first wedge pages, mid-sawtooth kickstart suppressed, sustained recovery resets, new-episode wedge pages again. Suite 21/21. Note: the cask bump to 1.4.4 (apsw fix) stops the wedging at the root, which kills most pings; this is the durable throttle for any residual episode. Follow-ups (documented, not in this PR): recovery-FAILED escalation ping + a once-daily digest line. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_942d1876-48db-4a33-b51a-3e0d5fe197a8) |
|
Warning Review limit reached
Next review available in: 42 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@codex review |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| # re-paging every cycle of a chronic sawtooth. A kickstart tick is never | ||
| # "healthy" (its state already advanced the liveness rowid), so the counter | ||
| # only climbs on real, unassisted watcher progress. | ||
| prev_liveness = int(state.get("watcher_liveness_highwater_rowid", 0) or 0) |
There was a problem hiding this comment.
🟡 Medium launchd/throughput-watchdog.py:568
The episode-reset block calls int(state.get("watcher_liveness_highwater_rowid", 0) or 0) and int(state.get("healthy_ticks", 0)) without validating the stored type, so a corrupt or non-numeric state file causes run_once to raise TypeError/ValueError and return probe_failed on every tick until the state file is manually repaired. The code already guards stalled_ticks and the high-water fields with isinstance(..., int) checks earlier; the episode-reset path should apply the same validation before converting.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @scripts/launchd/throughput-watchdog.py around line 568:
The episode-reset block calls `int(state.get("watcher_liveness_highwater_rowid", 0) or 0)` and `int(state.get("healthy_ticks", 0))` without validating the stored type, so a corrupt or non-numeric state file causes `run_once` to raise `TypeError`/`ValueError` and return `probe_failed` on every tick until the state file is manually repaired. The code already guards `stalled_ticks` and the high-water fields with `isinstance(..., int)` checks earlier; the episode-reset path should apply the same validation before converting.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a9ddfc3ca
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| prev_liveness = int(state.get("watcher_liveness_highwater_rowid", 0) or 0) | ||
| if result.stalled_ticks == 0 and current_liveness_highwater > prev_liveness: |
There was a problem hiding this comment.
When the DB does not have watcher_liveness_events, read_watcher_progress() deliberately reports liveness_rowid == 0 while chunk highwater can still advance, so after the first alerted wedge this new reset path never increments healthy_ticks and episode_alerted stays true forever. In that supported/compatibility schema, even three or more real realtime_watcher chunk-progress ticks will not clear the latch, so the next independent wedge episode is silently suppressed instead of paging once.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed: the reset now counts EITHER chunk-highwater OR liveness progress, so a DB without watcher_liveness_events (liveness_rowid==0 always) still clears the latch via chunk advancement — the first-episode-only bug is closed. Kickstart ticks still excluded (state pre-advances both rowids). Good catch. 21/21 + ruff clean.
The episode-reset latch keyed only on liveness_rowid advancement, but a DB without watcher_liveness_events reports liveness_rowid==0 forever while chunk highwater still advances — so after the first alerted wedge the latch never cleared and no NEW episode ever re-paged. Now "healthy" progress counts EITHER chunk highwater OR liveness. Kickstart ticks still don't count (state pre-advances both rowids). Suite 21/21, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_57eeaf5a-c930-437b-b904-1a06c64e6917) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
The appended episode-gating test wasn't ruff-formatted; `ruff format --check tests/` failed in CI. Formatting only — no logic change. Suite 21/21. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_48b6db71-371d-4420-bdef-b9c0ff5d119b) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3350ea25f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "last_action": "recovery_attempt", | ||
| "last_restart_epoch": checked_at, | ||
| "restart_attempt_count": int(state.get("restart_attempt_count", 0)) + 1, | ||
| "episode_alerted": True, |
There was a problem hiding this comment.
If alert_fn raises on the first wedge of an episode (for example a transient notification/logging failure), result.alert_error is recorded but this state write still sets episode_alerted to true. Subsequent stalled ticks in the same unresolved wedge then skip alert_fn entirely until three progress ticks occur, so an episode that was never successfully paged can be suppressed indefinitely while kickstarts continue.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed: episode_alerted now latches ONLY on a successful send — a transient notify failure on the first wedge no longer silences the episode; the next wedge retries the page. Test test_failed_alert_does_not_latch_episode_and_retries pins it. 22/22. Good catch, this inverted the PR's purpose.
reviewer-1 caught a page-once→page-zero inversion: episode_alerted was written True unconditionally even when alert_fn RAISED, so a transient notify failure on the first wedge silenced the ENTIRE episode — and the page is the operator's only outage signal. Now the latch is set only on a successful send; a failed alert leaves it clear so the next wedge retries. Test: test_failed_alert_does_not_latch_episode_and_retries — first-wedge alert raises, second wedge in the same episode retries (not silenced). Suite 22/22, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_28ab1fc9-ca84-43e6-a479-79fafdcd577f) |
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 96ef4b8497
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not episode_alerted: | ||
| try: | ||
| alert_fn(config, result) | ||
| episode_alerted = True |
There was a problem hiding this comment.
Fresh evidence since the earlier alert-latch comment: the default _best_effort_alert still swallows delivery failures (for example urlopen exceptions, and nonzero osascript exits do not raise), so when the first wedge only manages to append the log but fails to notify the operator, alert_fn returns normally and this line marks the episode as alerted. Subsequent recovery attempts in the same wedge then skip the alert until three progress ticks occur, turning a transient notification outage into a page-zero episode despite the intended retry behavior.
Useful? React with 👍 / 👎.
1 participant
What
The throughput-watchdog (#600) alerted on every kickstart. During the chronic wedge sawtooth (watcher re-wedging on the live 1.4.3 apsw bug, watchdog recovering each cycle), this paged Etan's phone every ~10min. orc's alert-framing law (stalker postmortem): operator pings = state-CHANGES only, never heartbeats.
Fix
Episode-based alerting: page on the first wedge of an episode only. The latch (
episode_alerted) clears after a sustained healthy run (_EPISODE_RESET_TICKS=3 consecutive progressing ticks), so the next genuine wedge pages again but a sawtooth does not. Recovery still runs every tick — only paging is throttled.Test
test_alert_framing_pages_once_per_episode_not_per_kickstartdrives a full sawtooth: first wedge pages → mid-sawtooth kickstart suppressed → sustained recovery resets latch → new-episode wedge pages again. Suite 21/21 (pre-push --no-verify because ingestion is actively degraded; the focused suite is the evidence).Context
The cask bump to 1.4.4 (#601, apsw fix) stops the wedging at the root, killing most pings; this is the durable throttle for any residual episode. Follow-ups (documented, not in this PR): recovery-FAILED escalation ping + once-daily digest line.
🤖 Generated with Claude Code
Note
Low Risk
Local launchd watchdog alerting/state only; no auth, data, or API surface changes beyond fewer duplicate operator pages.
Overview
Throughput watchdog no longer pages on every kickstart during a chronic wedge sawtooth. Operator alerts fire on the first wedge of an episode only; kickstart recovery is unchanged and still runs each stall threshold tick.
Persistence adds
episode_alerted(set only after a successful notify) andhealthy_ticks. After_EPISODE_RESET_TICKS(3) consecutive ticks withstalled_ticks == 0and real progress on chunk or liveness highwater, the latch clears so a new episode can page again. Failed first-page attempts do not latch, so later wedge ticks retry the alert.Tests cover full sawtooth suppression plus retry-after-notify-failure.
Reviewed by Cursor Bugbot for commit 96ef4b8. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Fix throughput watchdog to alert once per wedge episode rather than once per kickstart
episode_alertedlatch inthroughput-watchdog.pyso that only the first stalled tick of a wedge episode triggers an alert; subsequent stalled ticks in the same episode are suppressed._EPISODE_RESET_TICKS: after this many consecutive ticks with real progress (advancingwatcher_highwater_rowidorwatcher_liveness_highwater_rowid), the latch clears so a new wedge episode can alert again.episode_alertedandhealthy_tickskeys to the persisted state file.Macroscope summarized 96ef4b8.