§6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable

[Premmethuku75]

Found while checking §6.4 against infrastructure measurements from an agent evaluation I ran; the numbers are below.

The gap

§6.4 requires: "Every step that proceeds without a decision MUST be recorded as an audit event, so the bypass is visible rather than silent." I went looking for where that event can conformantly live, and there is no answer:

• The SessionContext chain can't hold it. context-entry.json defines step_type as "the method name from the originating request envelope," and a fail-open proceed is not a method. Structurally worse: the chain is written by the Guardian (§8), and the defining feature of a decision failure is that the Guardian produced nothing for this step.
• The Trace vocabulary can't hold it. The acs.decision span event enumerates exactly the five dispositions, and the OCSF severity mapping covers only those five. "Proceeded without a decision" is not representable.
• sessionEnd has no field for it. Its summary object carries outcome, step_count, errors[]. No failure classes, no counts.

So the spec contains a MUST that no deployment can satisfy in a way another party could consume. It has also already solved this exact problem once, at the handshake: §4.1 says a session started unguarded "MUST be recorded in the deployment's own audit log — there is no Guardian to receive the event — and SHOULD be surfaced on Trace events when the deployment claims ACS-Trace." That pattern stops at the handshake and never reaches per-step decision failures, which is where the volume is.

This bears on #32. The proposal there to flip the default rests on "the audit event already records every fail-open proceed, so the choice remains observable." As of v0.1.1 that event has no defined shape, so the premise is asserted rather than specified. The default question and this issue are separable: either answer to #32 needs the event to exist first.

Why the rate matters: measured data

§6.4 prices the trade qualitatively ("an adversary who can disrupt the channel converts control into audit"), and the handshake schema prices timeout latency per millisecond. Nothing anywhere lets a deployment or an auditor learn what fraction of steps resolved by posture instead of decision. From what I measured, that fraction is not an edge case on realistic infrastructure.

Over ten weeks I evaluated an LLM agent on a self-hosted serving stack (~120B open-weight model, 4-bit quantized, behind a reverse proxy; data could not leave the network). A production Guardian under §12.2 has the same shape: an LLM evaluator receiving full hook payloads, and agentResponse, toolCallResult, and knowledgeRetrieval all carry content. What I measured:

• Same 86 KB payload, 10 sends, 2 s apart: 4 of 10 returned 504 at the proxy's ~60 s upstream timeout; 6 of 10 returned 200 in 25–47 s. The stack was healthy at ~46 KB bodies; the failing calls were 83–86 KB. Payload size, not load, was the controlling variable.
• Effective per-step failure rate: 16% with a single retry, 2.56% after retry-with-backoff (3 attempts, 10/20/40 s).
• In one evaluation run, 3 "final answers" were verbatim proxy error pages (504 Gateway Time-out ... nginx), recorded as if the model had answered. Until traced, the substrate's failure was indistinguishable from a decision.

Small sample, single environment, stated as such. Even the mitigated rate is ~1 in 40 governed actions resolving by posture — under the default, proceeding ungoverned — with no conformant way to see the aggregate. Unmitigated it is 1 in 6. And because the rate is payload-size-dependent, it is attacker-improvable without compromising anything: inflate the payload → cross the proxy's tolerance → timeout → posture proceeds. Same bypass shape as the -32600 oversize path #32 documents, reached through the timeout half of §6.4 instead of the refusal half.

Proposed direction (wire-minimal, per SPEC_REVIEW_PRINCIPLES.md, Principle 3)

1. Define the decision-failure audit event. A named Trace event class alongside acs.decision — e.g. acs.decision.failure — carrying: failure class (timeout / transport / error_without_decision), the §17.1 error code when one exists (adopting @astrogilda's point in §6.4 conflates Guardian refusal with decision failure; refusals are attacker-reachable and should fail closed #32 that the class must travel as a code, not prose), negotiated timeout, elapsed ms, payload-size bucket, and the posture applied. Emitted by the Observed Agent, since the Guardian by definition cannot.
2. Extend §4.1's language to per-step failures: recorded in the deployment's own audit log; SHOULD be surfaced on Trace when ACS-Trace is claimed.
3. Add a session-cumulative count to sessionEnd.summary (decision_failures by class), so the per-session rate is computable from artifacts the spec already defines.

Payload-size bucketing is the one field I'd argue for from data rather than symmetry: it is the dimension along which the failure rate actually moved, and the one an anomaly detector needs to tell organic failure from induced failure.

Out of scope here: whether proceed is the right default. That is #32's live question; this issue is the observability floor under either answer.

References

• docs/spec/instrument/specification.md §6.4, §4.1, §8, §13 (the ping caveat — "a successful ping does not prove the enforcement path is healthy" — already concedes this path is otherwise invisible)
• docs/spec/trace/events.md (five-disposition acs.decision vocabulary)
• specification/v0.1.0/context-entry.json, specification/v0.1.0/hooks/session-end.json
• Related: §6.4 conflates Guardian refusal with decision failure; refusals are attacker-reachable and should fail closed #32 (refusal vs failure taxonomy; default-flip proposal), Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36 (trace self-reporting)

Prem Methuku

[ossumpossum]

This is the right kind of finding, and the measured data is what makes the case. I have hit the same failure class from the offensive-evaluation side, which corroborates the "indistinguishable from a decision" point from a different direction.

Running LLM agents against targets through a harness, I repeatedly saw transport and delivery failures (malformed or rejected deliveries, truncated exchanges) recorded by the harness as if they were the model's output, silently understating capability until I separated the delivery layer from the model's actual answer. That is the same shape as your three "final answers" that were verbatim nginx 504 pages: the substrate failed, and nothing downstream could tell that apart from a real decision.

Your payload-size mechanism sharpens it in a direction I had not measured but which follows directly: an adversary who inflates a payload to cross the proxy's tolerance produces a timeout indistinguishable from organic failure. That is exactly why the acs.decision.failure event and the payload-size bucket are worth specifying rather than leaving implicit, since without them the fail-open event cannot separate induced from organic failure, which is the one distinction an anomaly detector needs. +1 on all three proposed changes; the session-cumulative count in sessionEnd.summary makes the per-session rate computable from artifacts the spec already defines.

[Premmethuku75]

Thanks, corroboration from the attack side is the cross-check this needed. We hit the same failure class from opposite directions and both saw the substrate answering for the model.

Your induced-vs-organic point firms up something I left soft. I proposed the payload-size bucket as a field worth having, but if this event is meant to feed anomaly detection, it can't be optional, a stream that can't tell induced from organic failure answers the wrong question. So: payload-size bucket and the class code (per @astrogilda #32) both at MUST, everything else can stay optional.

[astrogilda]

The three-place search settles where the event can live. It leaves one constraint open, on the emitter, and that constraint decides whether the count is measurable.

Your event is emitted by the Observed Agent, because the Guardian by definition cannot. So the record that a governance decision failed is written by the party the governance was watching, using a code it selects for itself. That holds up under organic failure and breaks under induced failure, which is the case payload-size bucketing exists to detect. An agent that benefits from proceeding unguarded can emit timeout for a step where nothing timed out, omit the event, or emit nothing at all, and none of those is distinguishable downstream from a healthy deployment that had no failures. A zero in decision_failures and an unemitted event look identical.

This is the same shape as the #36 trace argument, one layer down. There the question was whether emission is evidence; here the emitter is the party under scrutiny and the subject is its own non-compliance.

A fourth change makes the count checkable. Without it, the MUST records a failure that only its perpetrator can vouch for.

Give the session a monotonic step counter that the Observed Agent cannot skip, and have the Guardian sign its own decision receipts with a sequence number. Then the count is derivable by subtraction: total steps, minus the steps that carry a signed Guardian receipt. Nobody has to be trusted to report it. An agent that suppresses failure events now has to suppress steps, which changes the step count, which the counter records. The self-reported class code then answers what kind of failure, and the counter answers how many, so no number depends on the agent's honesty about its own count.

This makes two records checkable. First, sessionEnd.summary.decision_failures becomes a claim a consumer can check against the receipts. If they disagree, that disagreement is itself the finding, and it is the one an anomaly detector should fire on first.

Second, the section 4.1 extension needs the same counter. An unguarded session recorded only in the deployment's own audit log has the deployment attesting to its own bypass, and a sequence gap is visible to a party who was not there where a missing log line leaves no trace of itself.

On the ping caveat in section 13, that a successful ping does not prove the enforcement path is healthy. A signed receipt per decision is the thing that would, since it is produced by the enforcement path doing its actual work, where a ping is answered by a side channel that stays up when the path is dead.

Codes for the failure classes, and one more distinction inside the enumeration: whether the Guardian was reached and did not answer, or was never reached. Those have the same effect on the step and different security meanings, so a deployment carrying only one code for both reads a slow Guardian and an absent one the same way.

[ossumpossum]

+1 to deriving the count rather than trusting it. Moving from "the emitter reports its failures" to "the failures are whatever is left after the signed receipts" is the same self-report-to-verifiable move as #36, and it is the right one.

Two conditions decide whether the subtraction actually holds, and both point back at #36:

1. The total has to come from an authenticated source, not the agent. Count-by-subtraction is total steps minus signed-receipt steps, but if the Observed Agent supplies the total it just truncates the tail: drop the last N steps and their receipts together, and the failures in that tail never show up in the subtraction. That is the same truncation case your run-end seal defends against in Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36. The session needs an authenticated maximum sequence number (a sealed step count at session end) for the subtraction to be sound. Un-skippable is not the same as un-truncatable.

2. The receipt signing scope has to bind the sequence number, canonically. "Signed with a sequence number" is only sound if the index is inside the signature and over a canonical form; otherwise a receipt can be replayed onto a different step, and two verifiers can disagree on which steps are covered. That is the defined-scope and canonical-serialization point I raised on the corpus in Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36: the receipt is evidence of "step k was decided" only if k is inside the signature, byte-exact.

With those two, monotonic-counter-plus-signed-receipts gives a count nobody has to be trusted to report, which is what the MUST needs.

[astrogilda]

Both conditions are right and the first one is a hole in what I proposed, not a refinement of it.

I made the counter un-skippable and left the total in the agent's hands, and those are not the same property. Drop the last N steps with their receipts and the subtraction returns zero for a tail that never gets counted, which is worse than a self-reported number because it looks derived. Un-skippable is not un-truncatable, and I had the first and claimed the second.

So the total has to be sealed, and the run-end seal from #36 already does it, so this needs no new mechanism. A sealed maximum sequence number at session end makes the tail non-droppable, and forgery cost is what does it, not ignorance. The seal is written at session end, so the producer does know by then which steps it would like to lose; what it cannot do is present a smaller count without forging a signature over the sealed one. The arming record and the end seal bracket the run, and the counter is what makes the bracket say something about each step inside it.

Your second condition is the sharper of the two for anyone implementing it. The index has to be inside the signature and over a canonical form, or a receipt is a receipt for nothing in particular: replay it onto a different step and two verifiers reach different covered sets from the same bytes. That is the signing-scope pin I put on #18, and it is the same defect underneath, so it wants one fix and not three.

Which makes the three issues one mechanism, and the count is the part that proves it works. #36 gives the bracket, #18 gives the scope the signature covers, #37 gives the quantity you can derive once the other two hold. Any one of them alone leaves a number somebody has to be trusted for.

Two things to state normatively, not leave to implementers.

The seal has to cover the count, not sit beside it, or a producer emits an honest seal and then reports a smaller total beside it.

And a session whose seal is absent has to read as unmeasurable, never as clean. If a missing seal reads as zero failures, truncating the entire session becomes the cheapest attack in the set, and it would sit inside the same fail-closed default #36 already argues for.

[Premmethuku75]

On the last normative point, a missing seal reading as unmeasurable rather than clean, my data backs it from the organic side: silent non-emission is the default state of instrumentation, not just an attack. In the same evaluation, my own harness under-emitted its recovery events 3 of 4 times, recovery happened in a code path the instrumentation didn't cover. An earlier audit found only 2 of 13 claimed runs had left logs at all, which forced a rebuild so every run writes evidence before it counts. Honest deployments will under-emit like this constantly. So the derived count isn't only defeating a lying agent, it catches the ordinary bugs that produce the same silence. That's why absent-reads-as-unmeasurable belongs in spec text: it'll fire far more often on bugs than on adversaries.