Parent: #674
Objective
Prevent worker/process explosion and duplicated local inference when many issues or subissues arrive at once. Admit work according to observed capacity, preserve fairness and share only safe equivalent in-flight inference.
Implementation steps
- Model separate limits for runnable tasks, active workers, inference requests, backend slots and memory budget.
- Add bounded queues with explicit saturation behavior.
- Implement weighted fair queueing across repositories/sessions/priorities with aging.
- Derive an equivalence key from model/backend identity, stable-prefix generation, canonical request hash, tool-registry generation and safety scope.
- Coalesce only side-effect-free inference requests with identical authority/context boundaries.
- Never deduplicate tool execution, mutations, approvals or user-private contexts.
- Fan out the same immutable completion/receipt to equivalent waiters with separate correlation ids.
- Propagate cancellation correctly: one waiter cancel must not cancel shared work needed by others.
- Add retry budgets with jitter and circuit breaking; prevent retry storms.
- Emit queue depth, wait time, admission decision, dedup hit/miss, fairness and saturation metrics.
Tests
- Queue capacity and deterministic rejection/defer behavior.
- Fairness/starvation simulation with hot and cold sessions.
- Dedup positive/negative matrix including one-byte prompt, authority and tool-generation differences.
- Cancellation with one/all shared waiters.
- Backend outage/circuit breaker and bounded recovery.
- Load/property tests proving queue and worker counts never exceed caps.
- Security test proving different users/scopes cannot share inference.
- E2E burst workload comparing duplicate calls and wall time.
Acceptance criteria
Parent: #674
Objective
Prevent worker/process explosion and duplicated local inference when many issues or subissues arrive at once. Admit work according to observed capacity, preserve fairness and share only safe equivalent in-flight inference.
Implementation steps
Tests
Acceptance criteria