Skip to content

eval: record bounded version-audit paired check, no runtime promotion - #89

Merged
Air000000 merged 2 commits into
mainfrom
codex/version-audit-paired-check
Oct 18, 2026
Merged

Air000000 merged 2 commits into
mainfrom
codex/version-audit-paired-check

Conversation

@Air000000

Copy link
Copy Markdown
Owner

Observed result — partial audit improvement, unchanged decisions

One explicitly authorized classification-only comparison on six fixed fictional development cases, using the same question/source/parameters per pair and alternating the first arm:

Metric v5.3 v5.3.1
version/time facet matches local expectation 3/6 5/6
overall gate decision matches local expectation 6/6 6/6

The candidate improves the version-mismatch and compatible-range audits. Both arms still incorrectly mark version/time NOT_REQUIRED when the source covers the explicitly requested version but a separate encryption-mode detail is missing. Outputs and expectations are preserved, not relabeled. This tiny known-fixture development check is not independent acceptance or evidence of improved answer/refusal accuracy. No runtime adoption or historical NO_PROMOTION / FAIL changes.

Execution and scope

  • Protocol committed before real calls: 214eb8d.
  • Exactly 12 Singapore qwen3.5-plus-2026-04-20 attempts, all HTTP 200, no retries/redirects; thinking off, temperature 0, max output 2048, timeout 30s.
  • 27481 reported tokens (12994 baseline / 14487 candidate), 119304 request-body bytes. ¥1 was a conservative authorized estimate, not an invoice.
  • Native httpx hooks bind every outbound body/endpoint/order to the frozen plan; hard limits 12 attempts, 24 KiB/body, 192 KiB total. Expected labels remain local. Network/provider/parse failure stops the run; semantic mismatches remain normal recorded results.
  • No embedding, rerank, answer generation, business material, TechQA/DEV, keys in model bodies, hidden reasoning, or provider error-body retention. No extra run or prompt edits after observation.

Verification

  • Final wire-guard tests: 16 passed; full guarded offline suite: 656 passed in 66.40s.
  • Tests cover paired clean payloads/order, plan-hash binding, fake complete flow, timeout/connect/HTTP/JSON/body/truncation stop, no retries, data tampering, size/call caps, endpoint scope, and redaction.
  • Ruff, report source SHA256/counts/usage/paired metrics and git diff --check passed.

Only fixed probe + tests + sanitized report + README. Existing API, refusal policy IDs, frozen prompts and historical reports are unchanged; switches stay default-off. Next prioritize deployment/persistence acceptance rather than automatically expanding audit-only paid tuning.

@Air000000
Air000000 merged commit 6b56b3b into main Oct 18, 2026
1 check passed
@Air000000
Air000000 deleted the codex/version-audit-paired-check branch October 18, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant