Skip to content

eval: add offline v5.3.1 version-audit clarification candidate - #88

Merged
Air000000 merged 1 commit into
mainfrom
codex/version-audit-clarification
Oct 17, 2026
Merged

Air000000 merged 1 commit into
mainfrom
codex/version-audit-clarification

Conversation

@Air000000

Copy link
Copy Markdown
Owner

Why

PR #87's real-provider flow passed, but the model audited an explicit fixed version as NOT_REQUIRED and separately called version coverage SUPPORTED while acknowledging the requested version was not covered. These are semantic model errors, not invalid JSON or citation locators. The existing parser is deliberately not a semantic oracle.

Minimal candidate, not a verified fix

  • Add an offline-only v5.3.1 system-prompt suffix clarifying constraint satisfaction vs merely documenting source scope.
  • Explicit static versions still require auditing; excluding the requested version is not evidence supporting it; visible compatibility remains valid without exact-string matching.
  • Ask for consistency with version/time reasoning without turning unrelated feature gaps into version failures.
  • Reuse the existing payload, schema, status vocabulary and strict locator gate unchanged. No keyword/regex semantic classifier, no output relabeling, no extra model/correction loop.
  • Do not include NovaVPN/AcornExpense fixture names or their expected answers in the new suffix.

Evidence and limitations

  • Six handwritten fictional boundary traces test construction and gate wiring ONLY. Two additional structurally valid but semantically wrong traces explicitly demonstrate that the parser still cannot repair semantic errors.
  • Focused candidate + original v5.3 + Hybrid tests: 64 passed.
  • Complete offline suite: 640 passed in 96.91s, using temporary DB/dummy credentials with external Python sockets and private project data reads blocked.
  • Ruff and git diff --check passed.

No real model calls this turn. No API/runtime/policy-id change, no persistent config or ticket creation. Frozen v5.3 sources, paid acceptance script and historical reports are untouched; NO_PROMOTION / FAIL remains unchanged. This is preparatory candidate code, NOT evidence of improved model behavior. A separately authorized small classification-only paired check should precede any runtime adoption, without repeating embedding/rerank/generation.

@Air000000
Air000000 merged commit 4d6f17c into main Oct 17, 2026
1 check passed
@Air000000
Air000000 deleted the codex/version-audit-clarification branch October 17, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant