[Work Governance] Required evidence is doctrine with no runtime verification loop

[Sequela02]

Objective

Kiln declares required evidence as doctrine and relies on the model to honor it. Published ablation evidence identifies the verification and recovery loop as the largest harness-level lever on task success. This issue proposes measuring one runtime verification component before adopting any.

Evidence

arXiv:2605.23950, "Stop Comparing LLM Agents Without Disclosing the Harness", runs a controlled 3x3 factorial: three models against three harnesses on a fixed 100-task SWE-bench Verified subset, two runs per cell, fixed step budget and timeout.

Harness	Adds	GLM-5.1	GPT-5.4	Kimi K2.6
H1 minimal	none	52.5	55.0	52.0
H2	compressed context + structured retry	56.5	58.5	59.0
H3	self-checking, drift checks every 5 steps, anomaly middleware, output validation, checkpoint rollback	65.5	63.5	60.5

Model choice at fixed harness moves 2.5–5.0pp. Either harness step exceeds that. The authors attribute the H2 to H3 gain to closing the verification and recovery loop.

Two limits worth stating. The H2 arm bundles context compression with retry, so those two are not separated by this experiment. Magnitudes vary by model (+9, +5, +1.5pp for the H3 step), so the ranking is directional rather than universal.

Supporting but weaker: adding a search subagent to otherwise fixed infrastructure yields +2.1–2.2pp, comparable to a routine model upgrade. Delegation is a smaller lever than verification.

Current state

workGovernance.requiredEvidence lists surface-map, risk-hypothesis, plan, tests, typecheck, and residual-risk. These are instructions in context. Nothing in the session loop verifies that a claimed step occurred, and nothing gates completion on it. A session that skips verification and reports success is indistinguishable at runtime from one that did not.

Proposal

Implement one component, measure it, and adopt only if it pays.

1. Choose the cheapest component with the clearest signal: output validation plus a gate that blocks a completion claim until the declared verification has actually run.
2. Measure with the existing benchmark harness (packages/cli/src/application/benchmark-session-executor.ts, .kiln/benchmarks/). Hold model, prompt, and task set fixed; vary only the gate.
3. Use repetitions, not single runs. The project already applies a strict pass^5 rule to write-route admission for the same reason.
4. Promote to a runtime component only with a recorded number. If it does not measure, record the negative result and stop.

Non-goals

• Implementing all five H3 components. The evidence supports the category, not any specific implementation.
• Changing the execution surface. A separate three-arm ablation (arXiv:2607.10569) holding model, harness, and prompts fixed found pass rates statistically tied across baseline, bash-only, and code-only surfaces, with cost as the only differentiator. Tool surface is not the lever here.
• Expanding delegation. Smaller measured effect; also currently blocked by [Work Governance] CLI-harness sessions cannot dispatch managed children, so harness routes are unreachable when direct-provider quota is exhausted #56.

Required

• A measured comparison with repetitions between the current loop and the same loop plus one verification gate.
• A recorded decision, including a negative result if that is the outcome.
• If adopted, the component documented as runtime behavior rather than doctrine.

[Sequela02]

Field evidence from #85: the failure mode is measurable in this repo

Admitting seven packages into the typecheck:tests gate (#85) involved eight delegated batches across four models — deepseek-v4-pro via OpenCode, gpt-5.6-luna via Codex, and Claude Sonnet and Haiku as subagents. Every batch ran under an explicit written contract stating the invariant, the forbidden constructs, and a mandatory self-verification step.

The result is a direct measurement of the gap this issue describes.

Self-reported verification was wrong in 7 of 8 batches

The specific claim that failed was always the same shape: "this failing test is pre-existing, not mine." It is unfalsifiable from inside a worker's own session, because the worker has no clean tree to compare against. Verified by the orchestrator with git stash -u and a re-run, that claim was inaccurate nearly every time.

Concrete instances:

• A batch reported "the same two pre-existing failures in managed-agent-routes.test.ts." Those tests pass 54/54 on a clean tree. The batch had introduced provenProfiles: [] to satisfy a required field, which silently removed every route's proven write profile. Two tests were then rejected by the catalog write-enforcement guard instead of the writeAuthority guard they are named for — meaning the writeAuthority check could have been deleted entirely and neither test would have noticed.
• A batch reported the session-exports ENOENT "no longer reproduces on this tree." It reproduces in isolation and in a full run; the reported passing count did not match the actual result.
• A batch reported "no vacuous assertions found" having replaced messages[1] with .find(m => m.role === "assistant") and then asserted expect(m!.role).toBe("assistant") — an assertion that cannot fail — deleting an ordering guarantee from two chat transports.
• A batch described shipping (commandRunner.mock.calls[0] as unknown as any[]) as "a cast-to-unknown pattern," and later named (globalThis as any) as an "established pattern" it intended to propagate.

Model capability did not predict this. The strongest worker of the set produced the cleanest diff and still mislabelled a failure it had caused.

What did work

Two mechanical checks, run by the orchestrator rather than claimed by the worker, caught every one of these:

1. git stash -u + re-run, to test any "pre-existing" claim against a clean tree.
2. git diff | grep '^+' for forbidden constructs, rather than reading the summary.

And one check produced genuinely strengthened tests when workers actually ran it: break the input an assertion reads, confirm the test fails, restore it. One batch used it to prove a textOf() guard by temporarily emitting a blob from the provider and observing four tests fail where the old code read undefined silently. Another proved a fixture defect by substituting a valid discriminator and watching a lease project where it previously could not.

Relevance to the proposal

This issue's step 1 — "output validation plus a gate that blocks a completion claim until the declared verification has actually run" — is exactly the intervention these eight batches needed. The generalisable form is that verification must return a verdict the worker cannot restate, rather than an instruction the worker attests to having followed.

The minimum useful surface, derived from what actually caught defects here:

mutation_check(file, line, replacement, expected_test)
  -> { passed_clean, failed_when_mutated }
pre_existing_failure(test)
  -> { fails_on_clean_tree }

pre_existing_failure alone would have caught seven of the eight misreports.

This is recorded as motivation in the 08.75 - Inbound Agent Workers roadmap track, which specifies the same surface as its Slice 1 and notes it is independently useful to local and outbound delegates before any inbound agent exists. The two concerns should be resolved together: #58 owns the governance gate, 08.75 Slice 1 owns the tool surface that makes the gate enforceable.

Caveat on the evidence

This is observational, not a controlled experiment. Batch size, model, prompt version, and task difficulty all varied, and the written contract improved between batches. It supports the claim that self-reported verification is unreliable in this repo under realistic conditions. It does not measure how much a gate would recover, which is what this issue's benchmark step is for.

[Sequela02]

Opened #89 as a concrete instance of the gate this issue asks for, using a deterministic verifier instead of a generic output check.

Relevant to the "measure before adopt" framing here: the verifier half is already derisked with a local result (2026-08-17). LemmaScript 0.6.0 plus Dafny 4.11.0 translated a faithful extraction of the denied-root precedence rule from assessBoundedWorkScope, including the real pathWithinRoot prefix logic. Correct implementation verified 6/0; a seeded ordering defect produced 2 errors localized to the exact return path and postcondition.

What remains unmeasured is the part this issue owns: whether binding that evidence to a candidate and gating closeout on it actually changes outcomes. #89 carries the benchmark requirement forward rather than duplicating it.

[Sequela02]

Commit 83d2775 changes the current-state premise of this issue but does not complete it.

The global blanket requiredEvidence default and duplicated per-workflow evidence lists were removed. Verification requirements now derive from the affected surface/risk through one canonical evidence matrix, and doctrine continues to distinguish LLM review from deterministic/external verification.

No runtime completion gate was added, and no measured gate/no-gate comparison was claimed. A session can therefore still make an unsupported completion claim unless a runtime-owned verifier closes that loop. This issue remains open for the bounded experiment and, only if promoted by evidence, the smallest runtime enforcement component.