Skip to content

feat(server): one Lead alert per usage-limit incident, auto-resume at reset, refuse launches on an exhausted provider - #100

Merged
nullStack65 merged 5 commits into
release/native-0.1.10from
feat/native-0.1.10-grouped-limit-alerts
Nov 19, 2026
Merged

nullStack65 merged 5 commits into
release/native-0.1.10from
feat/native-0.1.10-grouped-limit-alerts

Conversation

@nullStack65

Copy link
Copy Markdown
Owner

Kameron: "how to avoid 30 of these each time?" One account hitting its usage limit made ~20 threads each queue their own "Model alert ... usage limit" turn in Lead, all already stale. Later, 6 helpers started on an exhausted Codex each failed in under 2 s and alerted, 4 of them already replaced. Stacked on #98 (retarget to release/native-0.1.10 when it merges).

Pattern survey (Alertmanager and similar): group_by (one notification per label set), group_wait/group_interval (collect a storm, then rate-limit repeats), inhibition (a known cause silences its symptoms), and resolving an alert when its source recovers. This applies each, using #96's outbox rather than a second system.

  1. Group by cause. Usage-limit alerts become one usage-limit-incident per (environment, provider account): thread count, a numbered thread list (25 shown), reset time. Late threads merge into the queued item; the first waits 30 s (group_wait) for the storm, and a group that just reported waits 5 min (group_interval) before stragglers go out as a follow-up digest.
  2. Auto-resume on reset. autoResumeLimitedThreads is now on by default (Closura comment). The existing UsageLimitRecoveryWorker resumes each thread at the reset; Lead hears nothing unless a thread is still stopped 30 min past its reset, then one "did not come back" incident. A failed resume (a resumed run that hits the limit again) also reports.
  3. Drop stale alerts at delivery. Re-read each thread when sending: newer run that is running or completed, thread settled, archived, deleted, moved to another provider, or its parent started a later child with the same task title. Stale members leave an incident; an empty one is dropped.
  4. Merge, don't append. Described in 1; also applies to held items.
  5. Hold during a shared outage. If the watcher thread has itself failed on the same provider account, delivery waits (no attempt counted) and members keep merging, so Lead gets one digest when it is back. Works for a remote watcher using the existing t3_thread_read tool, no contract change.
  6. Stop it at the source. The server remembers which provider instances stopped at a usage limit and until when. delegate_task, t3_thread_launch and delegated-task dispatch refuse with Provider "X" is out of usage until <reset>; pick another provider. and start no run. Messages to existing threads are not blocked. Memory is in-process and relearned on the first failure after a restart. This belongs in the T3 fork: it is where launches and the failure signal both live. Magpie's own quota knowledge could later feed ProviderExhaustion.mark, but nothing here needs it.

Tests: 20 threads hitting one limit queue exactly one Lead-facing item and deliver one message; 20 with auto-resume ahead queue none; not-resumed threads make one separate incident; late members and stragglers; held and dropped items; staleness rules; and 6 launches on an exhausted provider are each refused with that message and create no thread.

Deploy: fleet

🤖 Generated with Claude Code

nullStack65 and others added 5 commits November 19, 2026 06:17
…lf, refuse launches on an exhausted provider

Group usage-limit alerts by (environment, provider account) into one queued
incident that lists threads and the reset time, merging late members instead
of queuing more turns. Threads resume themselves at the reset (auto-resume now
on by default); Lead hears only about a thread that does not come back. Drop
alerts at delivery when the thread has a newer live run, is settled or
archived, moved provider, or was replaced by its parent. Hold delivery while
the watcher thread is out on the same account. Refuse delegate_task and
thread launches on a provider known to be out of usage.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…alerts

Narrow the queued incident outside the map callback, keep the incident
literal type, read the exhaustion time through Clock and format it with
DateTime, and fix three test typings (alert runId union, readonly expected
kinds, Clock in the launch refusal test).

Model: Claude Sonnet 5.5
Task: fix
Agent: claude

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- A leftover digest gets its own id, so its delivery key is not a replay of the first.
- A dropped incident keeps members that joined after its snapshot.
- A watcher hold lasts only while the reset it reports is ahead, and a remote watcher no longer holds on any failure.
- A sibling replaces a delegated child only when it is running or finished and not archived or deleted.
- Auto-resume is not expected for a child of a closed parent or a thread snoozed past the grace period.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@nullStack65
nullStack65 changed the base branch from feat/native-0.1.10-lead-wake-events to release/native-0.1.10 November 19, 2026 06:50
@nullStack65
nullStack65 merged commit c771298 into release/native-0.1.10 Nov 19, 2026
2 checks passed
nullStack65 added a commit that referenced this pull request Nov 19, 2026
* Run ARC CI when a PR is retargeted onto a release branch

PR #100 was retargeted to release/native-0.1.10 after all its commits were
pushed, so the pull_request trigger (default types, no edited) never ran ARC
and the regression merged unchecked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep bot-triggered edit runs from cancelling the real ARC run

Review round for this PR (Fable 5.2, non-blocking findings 1 and 2): an edit
by a non-founder actor such as CodeRabbit started a run that cancelled the
in-progress founder run and then skipped its only job. PR runs now also group
by triggering actor. ready_for_review is dropped because the job has no draft
condition, so it only added a rerun of the same head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: nullStack65 <nullStack65@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kameron Smith <kameron.smith@closura.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant