Repository navigation
feat(server): one Lead alert per usage-limit incident, auto-resume at reset, refuse launches on an exhausted provider - #100
Merged
nullStack65 merged 5 commits intoNov 19, 2026
Conversation
…lf, refuse launches on an exhausted provider Group usage-limit alerts by (environment, provider account) into one queued incident that lists threads and the reset time, merging late members instead of queuing more turns. Threads resume themselves at the reset (auto-resume now on by default); Lead hears only about a thread that does not come back. Drop alerts at delivery when the thread has a newer live run, is settled or archived, moved provider, or was replaced by its parent. Hold delivery while the watcher thread is out on the same account. Refuse delegate_task and thread launches on a provider known to be out of usage. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
….1.10-grouped-limit-alerts
…alerts Narrow the queued incident outside the map callback, keep the incident literal type, read the exhaustion time through Clock and format it with DateTime, and fix three test typings (alert runId union, readonly expected kinds, Clock in the launch refusal test). Model: Claude Sonnet 5.5 Task: fix Agent: claude Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ive-0.1.10-grouped-limit-alerts
- A leftover digest gets its own id, so its delivery key is not a replay of the first. - A dropped incident keeps members that joined after its snapshot. - A watcher hold lasts only while the reset it reports is ahead, and a remote watcher no longer holds on any failure. - A sibling replaces a delegated child only when it is running or finished and not archived or deleted. - Auto-resume is not expected for a child of a closed parent or a thread snoozed past the grace period. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nullStack65
changed the base branch from
feat/native-0.1.10-lead-wake-events
to
release/native-0.1.10
November 19, 2026 06:50
This was referenced Nov 19, 2026
nullStack65
added a commit
that referenced
this pull request
Nov 19, 2026
* Run ARC CI when a PR is retargeted onto a release branch PR #100 was retargeted to release/native-0.1.10 after all its commits were pushed, so the pull_request trigger (default types, no edited) never ran ARC and the regression merged unchecked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep bot-triggered edit runs from cancelling the real ARC run Review round for this PR (Fable 5.2, non-blocking findings 1 and 2): an edit by a non-founder actor such as CodeRabbit started a run that cancelled the in-progress founder run and then skipped its only job. PR runs now also group by triggering actor. ready_for_review is dropped because the job has no draft condition, so it only added a rerun of the same head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: nullStack65 <nullStack65@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Kameron Smith <kameron.smith@closura.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kameron: "how to avoid 30 of these each time?" One account hitting its usage limit made ~20 threads each queue their own "Model alert ... usage limit" turn in Lead, all already stale. Later, 6 helpers started on an exhausted Codex each failed in under 2 s and alerted, 4 of them already replaced. Stacked on #98 (retarget to
release/native-0.1.10when it merges).Pattern survey (Alertmanager and similar):
group_by(one notification per label set),group_wait/group_interval(collect a storm, then rate-limit repeats), inhibition (a known cause silences its symptoms), and resolving an alert when its source recovers. This applies each, using #96's outbox rather than a second system.usage-limit-incidentper (environment, provider account): thread count, a numbered thread list (25 shown), reset time. Late threads merge into the queued item; the first waits 30 s (group_wait) for the storm, and a group that just reported waits 5 min (group_interval) before stragglers go out as a follow-up digest.autoResumeLimitedThreadsis now on by default (Closura comment). The existingUsageLimitRecoveryWorkerresumes each thread at the reset; Lead hears nothing unless a thread is still stopped 30 min past its reset, then one "did not come back" incident. A failed resume (a resumed run that hits the limit again) also reports.t3_thread_readtool, no contract change.delegate_task,t3_thread_launchand delegated-task dispatch refuse withProvider "X" is out of usage until <reset>; pick another provider.and start no run. Messages to existing threads are not blocked. Memory is in-process and relearned on the first failure after a restart. This belongs in the T3 fork: it is where launches and the failure signal both live. Magpie's own quota knowledge could later feedProviderExhaustion.mark, but nothing here needs it.Tests: 20 threads hitting one limit queue exactly one Lead-facing item and deliver one message; 20 with auto-resume ahead queue none; not-resumed threads make one separate incident; late members and stragglers; held and dropped items; staleness rules; and 6 launches on an exhausted provider are each refused with that message and create no thread.
Deploy: fleet
🤖 Generated with Claude Code