Gateways that pick a model at request time: the model is unknown before the call and unreported after it

[susheem-k]

TokenOps assumes the model identifier is known before a call is made. For stacks that route through a gateway which picks the model at request time, it is not — and the actual choice never reaches TokenOps at all, in either direction.

---

1. Context

TokenOps caps spend for a whole agent workflow and enforces it before every model call. Money is computed by a PriceFn — (provider, model, Usage) -> Micros — so every governance decision depends on knowing which model is being called.

Two policies consume that, at opposite ends of a call:

Policy	When	Needs the model for
pre_call_worst_case	Before dispatch	Pricing the projected input + capped output, to decide whether the call could exceed the budget
cost_budget	After dispatch	Pricing the actual Usage observed, to decide whether the budget is now exhausted

The model string flows from the caller, through wrap_complete, into both:

# src/tokenops/control/integration.py — inside governed()
request = CallRequest(
    attr=attr,
    provider=provider,
    model=m,                                   # <- supplied by the caller
    estimated_input_tokens=estimate(messages),
    max_output_tokens=controls.call.max_output_tokens,
)
governor.pre_call(request)

After the call, the crossing hook records an Observation, taking the model from the dispatch arguments:

# src/tokenops/control/crossing.py:153
model = str(state.get("model") or (gov.model if gov else "") or "")

state here is the dispatch call's arguments. So the model recorded post-call is the model that was passed in, not the model that answered.

---

2. Problem

A gateway — OpenRouter auto-routing, a LiteLLM proxy with fallbacks, a Bedrock or Vertex router, a Portkey config — accepts an alias (auto, a routing profile, a virtual key) and chooses a concrete model at request time. Selection can depend on load, price, availability, or a fallback chain after a 429.

That breaks the assumption in both directions.

Before the call: nothing to price

pre_call_worst_case receives the alias. Two outcomes, both wrong:

• The alias has no rate. Pricing raises, the policy TRIPs, and the run halts before its first call — the failure documented in Pricing table is a hardcoded 5-model dict: resolve rates from a per-run map or the control plane #128, now unavoidable rather than fixable by adding a rate.
• The alias has a rate. The number is fiction. It prices one model while the gateway may serve any of several, so the pre-emptive guarantee is not a guarantee.

There is no third option today: the policy has no way to express "the model is not yet known".

After the call: the answer never arrives

Post-call billing could in principle recover, since the gateway knows what it used. It does not, because the dispatch contract has nowhere to put it:

# src/tokenops/providers/types.py
@dataclass
class ModelResponse:
    content: str
    input_tokens: int = 0
    output_tokens: int = 0

No model field. Combined with crossing.py:153 reading the model from the request arguments, a gateway that returns model: "anthropic/claude-sonnet-4-5" in its response body has no way to tell TokenOps. The ledger bills the alias, and per-model attribution in the dashboard is wrong for every gateway user.

Some gateways go further and return a computed cost directly, which TokenOps has no way to accept either — it always prices from its own table.

Blast radius

Any deployment fronting models with a router. That is a common production shape: teams adopt a gateway precisely to get failover and cost routing, which are the same features that make the model unpredictable. cost_budget still functions as a backstop if an alias rate exists, but attribution is wrong and pre_call_worst_case is unusable.

---

3. Proposed approach

Sketch only — this needs design agreement before implementation.

Let the model be unknown until the response. Treat "resolved at request time" as a first-class case rather than an error, so the pre-call path can degrade deliberately instead of tripping.

Let dispatch report what actually answered. Add an optional model (and possibly cost_micros) to ModelResponse, and prefer it over the request argument in crossing.py:153 when present. This is the smallest change with the largest effect: it fixes post-call attribution and billing accuracy on its own, independently of anything pre-call.

Then choose a pre-call strategy. Options worth weighing:

Option	Behaviour	Trade
Skip pre_call_worst_case for gateway runs	Rely on cost_budget as backstop	Budget can overshoot by one call
Price a declared ceiling	Caller names the most expensive model the gateway may return	Preserves the pre-emptive guarantee; needs the caller to know the routing set
Accept gateway-reported cost	Bypass the rate table when the gateway provides a cost	Trusts the gateway's arithmetic; no pre-call help

The ceiling option is the only one that keeps pre_call_worst_case meaningful, and it composes with #128's rate map — the ceiling is just another key.

---

4. Tradeoffs and non-scope

• Not a rate-table problem. Pricing table is a hardcoded 5-model dict: resolve rates from a per-run map or the control plane #128 makes rates configurable, and nothing in it helps here: the issue is not that the model's price is unknown, but that the model is unknown. The two are independent and Pricing table is a hardcoded 5-model dict: resolve rates from a per-run map or the control plane #128 should not wait for this.
• Gateway-reported cost means trusting the gateway. Accepting a cost the gateway computed removes TokenOps' independent arithmetic. Reasonable for attribution, questionable as the sole basis for enforcement; likely wants to be opt-in.
• Streaming is out of scope — wrap_stream and time-windowed accrual are Multimodal stream governance: implement wrap_stream and time-windowed accrual (spec) #110.
• Failure postures are out of scope — whether an unresolvable model should fail open or closed belongs to Failure-mode groups and per-group behaviour flags (plane unavailable, pricing unavailable, policy error) #129's taxonomy.

5. Acceptance

• A run whose model is chosen by a gateway is governable without a fabricated alias rate
• The model that actually answered is what gets attributed in the ledger and dashboard
• pre_call_worst_case either has a defined behaviour for unknown models or is documented as inapplicable
• A gateway-reported cost can be used where available, and its use is visible in the run record
• Non-gateway runs are byte-for-byte unaffected

---

Related: #128 (rate resolution — assumes the model is known, and says so), #129 (failure-mode postures), #110 (streaming governance), #57 (output-token prediction for pre_call_worst_case).

Code references verified against main at v0.2.1.

[susheem-k]

Related: #137 — Usage.cached is never populated, so cached tokens bill at the full input rate (measured 3× over-billing on a real agent run). Relevant here because it is the same class of defect — `ModelResponse` cannot carry what the provider response already knows.