needs-runtime-test open questions don't gate findings: the same run said "source alone cannot determine which" and then shipped one candidate as strong inference / fix before porting

[TheAmericanMaker]

Summary

A phase can ship a finding as strong inference with action fix before porting while an open
question registered in the same run states that source alone cannot settle that exact point.
Nothing cross-checks the two, so the run contradicts itself and the confident version is the one
that reaches the reader.

Reproduction

Ran pipeline-full-with-deep-audit.yaml over a Windows-only local LLM frontend
(ElodineOfficial/GobboNet at 5524fd4), then verified the findings by executing the product.

1. defect-scan-mechanical registers q-logit-bias-root-cause, kind: needs-runtime-test,
   description ending: "the breakage is either upstream (llama-server ignoring/not supporting
   logit_bias on the pinned build), a request-shape mismatch, or a tokenizer mismatch. Source
   alone cannot determine which."
2. The same phase routes the shape hypothesis onward as mech-CF3.
3. defect-scan-semantic Pass 5 CodeCartographer for [Pi] #2 then closes mech-CF3 by asserting the request-shape
   candidate: the client sends logit_bias as a map where /v1/chat/completions expects an array
   of {id, bias}, labelled strong inference, action fix before porting, with a one-line
   reshape as the recommended fix.
4. Both artifacts ship. q-logit-bias-root-cause is still listed unresolved in status.yaml
   while the finding that presumes it resolved carries an actionable label.

Expected behavior

If a run registers an open question whose stated reason is that source cannot settle a point, no
finding in that run should assert one of that question's candidates with an action label implying
it is settled. Either the question closes with evidence, or the finding inherits the question's
uncertainty.

Concretely, one of:

• Block fix before porting on any finding that closes a carry-forward derived from an unresolved
  needs-runtime-test question, and force a weaker action such as needs-runtime-test or
  port differently.
• Add an evidence level below strong inference for claims about external-system behavior, since
  server-side payload parsing, API contracts and engine-version behavior are not observable from
  client source at any read depth.
• At minimum, have validation flag the contradiction so the orchestrator sees it before completion.

Actual behavior

Both findings shipped confidently and both were wrong when executed. The confident label survived;
the hedge did not travel with it.

Case 1, and this one was harmful. Runtime testing against the pinned engine (b9294,
temperature 0, three deterministic runs per cell) inverted the finding:

logit_bias shape	Result on /v1/chat/completions
{"13753": -20} (what the client sends)	suppresses the token, works
[{"id":13753,"bias":-20}] (the recommended fix)	silently ignored, HTTP 200, no parse complaint, output byte-identical to no bias

Re-verified on b10509, 1,215 builds later: same result. The recommended one-line reshape would
have converted a working feature into the one shape the engine ignores, with no error surfacing.
The real cause was a tokenizer-level ceiling that the audited code already documented in a comment
the scan had read.

Case 2, same family. defect-scan-mechanical recorded in its coverage limits that
"the encoded search-proxy command (launch.bat:1608) was not fully decoded." defect-scan-semantic
Pass 4 #1 then asserted, as observed fact, that "the search proxy forwards verbatim to
https://ollama.com/api" and concluded the project's privacy claim was "false as written."
Decoding and running that component showed it builds a fresh header hashtable and drops browser
headers, so the claim was substantially true. An upstream phase's declared coverage gap did not
stop a downstream phase asserting a conclusion inside it.

Environment

• Surface: MCP server
• Pipeline: pipeline-full-with-deep-audit.yaml
• Phase that failed: defect-scan-semantic (both cases); contributing gap in defect-scan-mechanical
• LLM provider / model: Claude, Opus 5
• Node version: v22.22.3
• CodeCartographer version: v0.16.0
• OS: Fedora Linux 7.1.8 (audited target is Windows-only; runtime verification ran on both)

Additional context

The two findings the run ranked highest in its own summaries were the two that were wrong. What
survived verification was the mechanical material: missing request-body size caps, an over-broad
static file handler, a credential prefix written to a browser console, wildcard CORS. Inventory and
omission-spotting held up well. Every claim about what another process does with a payload did not,
and both were stated in the same register as the verifiable ones.

Three smaller instances of the same habit, from the same run, all plausible-looking specifics
stated flatly and all wrong: the engine zip recorded as ~300 MB when it is 32.8 MB; a
logit_bias example strength of -5 when the product's default is -20; and "the call site" for a
reshape when there are two.

Suggested reading if useful: the runtime write-ups are on a public branch,
TheAmericanMaker/GobboNet @ audit/logit-bias-runtime-result, under
.codecarto/findings/runtime-egress-verification/ (TEST-PLAN-logit-bias.md,
TEST-RESULT-logit-bias.md, egress-verification.md).

[TheAmericanMaker]

Investigation

Full write-up: docs/plans/2026-08-21-issue-122-unverifiable-claims.md on claude/github-issue-122-jgwjih. No code changed yet — the two vocabulary names below want a decision first.

Six independent gaps, each separately sufficient

1. The pre-porting action vocabulary has no "not settled yet" value. findings/defect-scan/SKILL.md §Action Classification gives maintenance pipelines four actions including investigate ("needs runtime testing or deeper analysis to confirm"). Pre-porting pipelines get three: fix before porting, port differently, leave behind — all three assert the diagnosis is settled. The evidence ladder already admits open question, but the Action cell still has to be filled and every available value overstates. The vocabulary forces the contradiction this issue reports.

2. The evidence ladder has no rung for a claim about a system outside the analyzed tree. This is the crux, and it's why re-reading the SKILL wouldn't have caught it. strong inference is defined as "highly likely based on multiple code observations" — and that was literally satisfied: the client source does send a map, and the OpenAI-compatible spec does document an array. The label was not a misuse of the definition as written. The ladder is under-specified, not misapplied. passes/05-api-contract-violations.md actively invites this class of claim ("APIs documented to return a specific shape…") and never mentions that the analyzed code isn't the thing implementing the contract.

3. The three defect templates are the only phase templates with no ## Open Questions section. architecture-map.md, behavioral-contracts.md, protocols-and-state.md, reverse-engineering-bundle.md all carry one. mechanical-defects.md, semantic-defects.md, defect-report.md do not. So a defect scan's needs-runtime-test question exists only in scratch/handoffs/<phase>.yaml → workflow/status.yaml; the report a human reads, and the report the next phase reads as a required read, carries the finding with no adjacent hedge. This is mechanically why "the confident label survived; the hedge did not travel with it."

4. Nothing verifies a claimed closure. applyHandoff (core/status.ts:321) removes carry_forward and open_questions entries by id unconditionally. validatePhaseOutput reads only the ## Validation table, never the findings tables. Closing mech-CF3 while q-logit-bias-root-cause stays unresolved isn't merely undetected — there is no code path that could detect it.

5. A declared coverage gap does not constrain downstream phases — Case 2. ## Coverage and limits is required in every primary output (enforced by tests/pipeline-invariants.test.mjs:166) and the mechanical phase used it correctly, but that ledger is carried nowhere: not into the next phase's prompt, not into status.yaml, not into validation. The existing "Contradiction sweep" duty (core/prompts.ts:67) compares the incoming phase's required reads against completed phases' owner_notes only — a declared blind spot is not an owner note, so the sweep as written could not have seen it.

6. No cite-or-hedge rule for quantities. Nothing requires a number in a finding to cite where it was read, so the 32.8 MB→~300 MB / -20→-5 / "the call site" family has nothing aimed at it.

Why the existing safeguards missed it

Worth flagging: the re-triage duty points toward this failure, not away from it. It asks whether a needs-runtime-test label has become answerable by reading — this run effectively answered yes and closed it by inference. The duty was written for the inverse defect (the four-phase mislabel recorded in references/orchestration.md) and has no counterpart: "and when the answer is genuinely no, no finding in this phase may assert one of that question's candidates with a settled action."

Validation couldn't help either: both reports' ## Validation tables were honestly filled and did PASS. Criterion 5 is "Findings are marked with evidence levels" — satisfied. No criterion reads what the levels say. Completion refuses only FAIL/MISSING.

Recommendation — three stages

Stage 1 (no code, reaches all three surfaces). Add a verify at runtime action and propagate it through the whole chain — defect SKILLs → bundle Defect Synthesis disposition → spec Spike List + post_pipeline kind: spike → pipeline completion criteria. A label that porting flattens back into three is worthless, so this is ~12 files. Add the evidence rung for external-system claims, with a hard rule that it can never pair with fix before porting. Add ## Open Questions to the three defect templates, with a Derived-findings column so the question and the findings touching it sit in one document. Add the missing counterpart duty to GUIDE.md §Roles and references/orchestration.md. Plus the quantity-citation rule.

Stage 2 (core; Pi + MCP). The findings tables are header-identical across all three defect templates, so an evidence/action cross-check is a deterministic parse rather than a heuristic. Gate the flat contradiction (external-behavior or open question evidence paired with fix before porting), warn on the softer ones, via a new warnings[] on ValidationResult plus one summary line — precedent is the non-gating secondary-outputs NOTE:.

Stage 3 (schema). Optional derives_from on carry-forward entries so closing one whose question is still open refuses completion; evidence on open_question_closures; and the coverage-gap ledger parsed and rendered into the next phase's prompt (the duty the Contradiction sweep is missing). This is also the unchecked "closure sweep" item already in docs/plans/post-0.11-evidence-roadmap.md.

Two constraints shaped the staging: a gating FAIL stops the --auto loop (extensions/codecarto/auto-runner.ts:294), so no heuristic check may gate; and the hard gate wants a version-gate on scaffoldVersion, since an in-flight workspace had no honest action to choose. docs/ROADMAP.md §Validation hardening already plans "automated structural pre-checks… evidence-tag coverage" — Stage 2 is that line item aimed at this defect.

Decisions needed before implementation

1. Evidence rung name — external-behavior claim (says what it's about) or unverifiable from source (says why it matters)? It lands in six pass files, five SKILLs, three templates and four docs, so worth deciding once.
2. Action name — verify at runtime reads as an instruction and parallels the maintenance set's investigate; needs-runtime-test reuses the existing open_questions.kind value exactly (one vocabulary instead of two) at the cost of reading oddly in an Action column.
3. Does the Stage 2 check gate or only warn on a current scaffold?
4. Is derives_from worth the handoff-schema addition, given it depends on the upstream phase filling it in? (Additive/optional, so no ABI break — but it's still schema.)

Thanks for the runtime write-ups on audit/logit-bias-runtime-result — the b9294/b10509 re-verification is what makes gap 2 diagnosable as a vocabulary defect rather than a model slip.

---

Generated by Claude Code