Skip to content

Sidecar preflight: base-probe success never validates the real serving token budget #1454

Description

@seonghobae

Background

Devin Review, on PR #1452 (ADR-0005's implementation), verified against current code:
scripts/ci/contextual_orchestrator_review_launcher.py's _preflight_review_agents admits a
candidate into the serving pool the moment its cheap REVIEW_PREFLIGHT_BASE_TOKENS = 16 base probe
returns any non-empty content. Escalation to REVIEW_PREFLIGHT_ESCALATED_TOKENS (REVIEW_MAX_OUTPUT_TOKENS,
4096 — the real budget main()'s serving ModelClient uses for actual review traffic) only fires on
evidence of failure (empty content matching the budget-too-small signature). A candidate that
succeeds at 16 tokens is never probed at 4096 before being admitted.

Why this is a real gap, not just theoretical

ADR-0005's own Research section (docs/adr/0005-sidecar-preflight-token-budget.md, axis 2, around line
81) already documents that provider completion-token ceilings are a real, evidenced, per-model quantity
separate from reasoning-overhead (axis 1): "Some providers reject a request outright if max_tokens
exceeds what that specific model supports."
Axis 1 (reasoning overhead) is what the escalation path
corrects for. Axis 2 (a hard completion ceiling strictly between 16 and 4096) is not corrected for on
the success path: if a candidate's real ceiling is, say, 500 tokens, it happily returns real content at a
16-token probe, gets admitted, and then a real review request at 4096 tokens can be rejected by the
provider outright.

Layer 2 (the shell script's separate virtual-pool smoke request) does exercise the real 4096-token budget
— but only against the ONE candidate the virtual pool router happens to select for that single smoke
request, not against every individually-admitted Layer 1 candidate. With up to 12 admitted candidates,
this leaves most of the pool's success-path candidates unvalidated at the real serving budget.

Known mitigation (partial, not a fix)

contextual_orchestrator.orchestrator.TaskOrchestrator/ModelClient already implements per-request
failover to the next capability-matched agent plus a per-agent circuit breaker in production. A candidate
that is actually incompatible with the real budget should fail over at request time rather than silently
serve garbage — but this still costs latency/retries on real review traffic and could trip a circuit
breaker on an otherwise-healthy candidate that was simply preflighted at the wrong budget.

Options considered (not evaluated to a decision — needs its own design pass)

  1. Probe the real 4096-token budget for every candidate, not just on the failure/escalation path — but
    this risks reintroducing exactly the original bug ADR-0005 exists to fix (a uniformly-large probe
    budget caused the 120s-timeout regression that motivated this whole redesign).
  2. Block on real per-model max_output_tokens/context_window discovery data (tracked separately,
    ContextualWisdomLab/contextual-orchestrator#927) so admission can be decided from known ceilings
    instead of an empirical probe at all.
  3. Something narrower — e.g. a second, cheap-but-larger confirmation probe only for candidates that will
    actually be selected into the served catalog, bounded so it doesn't reintroduce the timeout regression.

Ask

Design a fix (or an explicitly accepted, documented residual-risk decision, per this org's convergence
convention) for the success-path gap. Not blocking PR #1452's fix of the 7 verified Devin Review findings
already in that PR — flagged there as architecturally significant enough to need its own pass rather than
a guessed patch.

Cross-ref: ADR-0005 (docs/adr/0005-sidecar-preflight-token-budget.md), PR #1452, PR #1449,
ContextualWisdomLab/contextual-orchestrator#927.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions