Skip to content

[Scheduler] Add fair admission control, bounded backpressure and equivalent-work inference deduplication #676

Description

@wesleysimplicio

Parent: #674

Objective

Prevent worker/process explosion and duplicated local inference when many issues or subissues arrive at once. Admit work according to observed capacity, preserve fairness and share only safe equivalent in-flight inference.

Implementation steps

  1. Model separate limits for runnable tasks, active workers, inference requests, backend slots and memory budget.
  2. Add bounded queues with explicit saturation behavior.
  3. Implement weighted fair queueing across repositories/sessions/priorities with aging.
  4. Derive an equivalence key from model/backend identity, stable-prefix generation, canonical request hash, tool-registry generation and safety scope.
  5. Coalesce only side-effect-free inference requests with identical authority/context boundaries.
  6. Never deduplicate tool execution, mutations, approvals or user-private contexts.
  7. Fan out the same immutable completion/receipt to equivalent waiters with separate correlation ids.
  8. Propagate cancellation correctly: one waiter cancel must not cancel shared work needed by others.
  9. Add retry budgets with jitter and circuit breaking; prevent retry storms.
  10. Emit queue depth, wait time, admission decision, dedup hit/miss, fairness and saturation metrics.

Tests

  • Queue capacity and deterministic rejection/defer behavior.
  • Fairness/starvation simulation with hot and cold sessions.
  • Dedup positive/negative matrix including one-byte prompt, authority and tool-generation differences.
  • Cancellation with one/all shared waiters.
  • Backend outage/circuit breaker and bounded recovery.
  • Load/property tests proving queue and worker counts never exceed caps.
  • Security test proving different users/scopes cannot share inference.
  • E2E burst workload comparing duplicate calls and wall time.

Acceptance criteria

  • No unbounded queue or automatic worker multiplication exists.
  • Equivalent inference is coalesced only before side effects.
  • Security/authority scope is part of the equivalence key.
  • Every waiter receives its own trace correlation and truthful shared-execution reference.
  • Fairness tests demonstrate bounded starvation.
  • Saturation degrades through explicit defer/block/fallback states.
  • Retry storms are mechanically prevented.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions