Failure-mode groups and per-group behaviour flags (plane unavailable, pricing unavailable, policy error)

[susheem-k]

Summary

TokenOps applies one global failure posture — fail closed (control/core.py:18: "an unknown price or unhandled action blocks; it never silently allows"). That is the right default, but it is currently uniform, implicit, and untyped: an operator cannot choose a different posture for different classes of failure, and a caller cannot tell why a run stopped.

The work here is two-part, and the first part is the substance: agree the failure-mode groups, then attach behaviour flags to them. Filing the taxonomy as part of the issue rather than jumping to flags.

Why this is not just ergonomics

Two failures observed in one integration (mini-SWE-agent), each with a different correct posture, both currently handled the same way or worse:

1. Unpriced model — halts, but signals like a budget stop. A model missing from the rate table raises, and the caller sees:

Halt: run already halted; refusing further calls

That message describes the symptom (ledger.admit found the halt flag) rather than the cause (no price for the model). It is byte-for-byte what a legitimate exhausted budget looks like. Diagnosis took materially longer than it should have — the only tell was that a $0.02 budget halting before call 1 was arithmetically impossible. See #128.

2. Control plane unreachable — raw transport error escapes. With TOKENOPS_URL pointing at a dead port:

httpx.ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it

This propagates straight out of tokenops_run. Two consequences:

• It is an Exception, not a Halt. Halt deliberately extends BaseException (core.py:235-239) precisely so "a framework's broad except Exception for flaky APIs cannot swallow the breaker" — but the plane-unavailable path has no such protection and is swallowed by exactly those handlers. In mini's benchmark runner the instance was simply recorded as failed with exit_status: ConnectError, alongside ordinary agent errors.
• There is no posture choice. For a pre-production run an operator may well prefer "plane down → run ungoverned and warn loudly" over "plane down → nothing runs". Today that is not expressible.

Proposed failure-mode groups

Strawman for discussion — the grouping is the thing to settle:

#	Group	Examples	Plausible default
1	Plane availability	register/join run, ledger read/write, governance config fetch, connect/timeout/5xx	fail-closed (configurable)
2	Pricing / metering	model not in rate table, provider returned no usage, token counts absent or zero	fail-closed
3	Policy evaluation	detector or actuator raises, budget id references a missing budget, unknown policy name in config	fail-closed on config errors; degrade on runtime errors
4	Actuator application	retry exhausted, compaction hook missing, model_override target unavailable, CANCEL unimplemented (#67)	degrade + record
5	Telemetry / boundary	Chronicle hook not installed, crossing observation fails, span emit fails	fail-open (never block a run on telemetry)

Group 5 is the clearest case for a different default than group 1 — losing an observation should not stop a run, whereas losing the ledger arguably should.

Proposed behaviour flags

Per group, one of:

• fail_closed — raise/halt, refuse to proceed
• fail_open — proceed ungoverned, emit a structured warning
• degrade — proceed with a reduced guarantee (e.g. local ledger while the plane is down, reconciled later), emit a warning

Sketch:

governance:
  on_failure:
    plane_unavailable: fail_closed     # fail_open | degrade
    pricing_unavailable: fail_closed
    policy_error: degrade
    actuator_error: degrade
    telemetry_error: fail_open

Defaults must preserve today's behaviour so this is not a breaking change.

Signalling requirement

Whatever the postures, the cause must be distinguishable at the call site. Today every stop is a Halt with a free-text string. Suggested minimum: a machine-readable reason code on Halt (e.g. Halt.reason: Literal["budget_exhausted", "step_cap", "unpriced_model", "plane_unavailable", ...]), and a distinct exception type — or at least a BaseException subclass — for infrastructure failures so they cannot be swallowed by framework except Exception handlers the way ConnectError currently is.

A caller should be able to write:

except Halt as h:
    if h.reason == "budget_exhausted":
        ...      # expected, business event: record partial result
    else:
        ...      # infrastructure problem: alert, do not count as a governed stop

Acceptance criteria

• Failure-mode groups agreed and documented (this is the gating decision).
• Each group has a configurable posture; defaults match current behaviour.
• Infrastructure failures are distinguishable from budget stops by type or reason code, not by string matching.
• Plane-unavailable failures cannot be silently swallowed by a broad except Exception.
• Unknown-price halts name the model and the cause, not just run already halted.
• Docs page describing the groups, the defaults, and when to change them.

Related: #128 (unpriced models — the concrete case behind group 2), #109 (fail-closed when a run has no governed dispatch — a specific instance of group 1/3), #118 (plane as a hard HTTP dependency — motivates group 1 postures), #67 (CANCEL unimplemented — group 4).