feat(audit): okf audit — corpus-level trust, freshness and lifecycle query by jchable · Pull Request #49 · jchable/okf4net

jchable · 2026-08-22T12:14:58Z

What

Adds okf audit, a corpus-level query over a bundle's OKF v0.2 trust (§5.3), lifecycle (§5.4) and staleness (§5.5) signals. It answers, in one command, the question v0.2 makes askable but ships no query surface for:

which of my concepts are past their stale_after date and have never been verified by a human?

okf audit bundles/acme_retail --stale --trust unverified,machine-confirmed

Four surfaces, one computation:

src/OKF4net/Audit.cs — ConceptAudit (the shared query), AuditQuery, AuditFinding, AuditReport, and AuditVocabulary, the single spelling of the trust/status/freshness vocabularies so no two layers can drift.
src/OKF4net.Cli/ — the okf audit verb. Without a filter flag it selects exactly what --stale selects and prints a summary plus the worklist; with any filter flag it prints one line per concept, so the output pipes. --json always emits the full document.
tests/fixtures/golden/audit-v02.{out,json} — byte-exact goldens, hand-authored and verified against the spec text (no upstream CLI to capture), pinned with --as-of so they cannot drift with the calendar.
src/OKF4net.Agents/ — the read-only okf_audit tool, also served over MCP.

Counts always describe the whole bundle while findings describes the selection: audit is a worklist, not an inventory.

Design

Spec and plan are in the diff: docs/superpowers/specs/2026-08-21-okf-audit-design.md and docs/superpowers/plans/2026-08-21-okf-audit.md.

Deliberate decisions worth reviewing:

audit is not validate. validate answers "is this bundle conformant?" and exits non-zero when it isn't; audit reports and always exits 0. No existing validate/info output changes, so no existing golden moves.
The CLI and agent renderers are separate on purpose. The CLI's bytes are golden-locked and must not move because an agent-facing string was tuned. Only the computation and the vocabulary are shared.
CLI input parses strictly, frontmatter parsing stays permissive. Lifecycle.From resolves an unknown status to stable (§5.4, §11); --status retired fails. A typo silently absorbed into stable would make a worklist lie.
StalePolicy and AuditQuery.StaleOnly coexist. They answer opposite questions: "should I surface this to a consumer?" versus "is this on my worklist?".

Verification

1041 tests, 0 failures (baseline 987), clean build under TreatWarningsAsErrors, dotnet format --verify-no-changes clean.
No existing fixture modified; no PackageReference added to the zero-dependency projects.
Exercised by hand end to end, not only through tests: both text modes and --json on bundles/acme_retail, the error paths, and a real okf-mcp stdio session calling tools/call for okf_audit.
AgentIntegrationTests now drives okf_audit through the framework's real function-invoking pipeline, which is what covers the JSON argument binding; the test was mutation-checked (drop the trust argument → it fails).

Also in here

The samples/acme-retail-agent sample already exposed the new tool (it filters GetTools() by WriteToolNames), but nothing told the agent a corpus-level question was answerable in one call. Adds that to the system instructions plus a "Questions worth asking" section.

That work surfaced a pre-existing break: a dependabot bump moved OKF4net.Agents to Microsoft.Agents.AI 1.17.0 while the sample's own pin stayed at 1.15.0, which NU1605 rejects as a downgrade — so the sample could not be built at all. Realigned. Nothing caught it because samples/ is outside OKF4net.sln and outside CI.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TDM9Aozt9YciNJ3oncrFxG

Adds the design for a new CLI verb, core query type and agent tool that answer "which concepts are past stale_after and were never verified by a human?" across a whole bundle -- the question OKF v0.2 makes askable but ships no query surface for. Three units: ConceptAudit (core, shared computation), the `okf audit` verb (text + --json), and the read-only okf_audit agent/MCP tool. No new frontmatter fields: everything derives from v0.2 sections 5.3-5.5. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Seven defects found by re-reading the spec against the code: - AuditQuery is a record struct whose generated equality compares the trust set by reference; the type advertised a value equality it does not have. Documented as a remark, not relied on. - No way to select the whole corpus: without filters the verb reports the stale worklist, so --json findings are not the corpus. Documented the three-tier idiom and why no --all is added. - audit-v02.exitcode would be a golden for a constant (exit is always 0); dropped, asserted inline like Info_output_matches_golden. - The set of mode-switching flags was never enumerated: --as-of and --json must not switch to query mode, which the golden depends on. - query.trust JSON ordering was unspecified; pinned to ladder order so the document is reproducible with several tiers. - The position of the audit line in the usage text was unpinned while test 24 checks it; pinned right after validate. - CLAUDE.md and ROADMAP.md are concurrently modified by another session: noted as end-of-branch work to avoid a rebase conflict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

All five verified against the code before accepting: - DateOnly has no (s, format, provider, out) overload, so the parsing contract as written would not compile. Pinned the full five-argument call, matching Lifecycle.From. - --status had two incompatible semantics: scalar in AuditQuery, the JSON and the tool, but described as a deduplicated list in the parsing rules. Settled on scalar; --trust is the only multi-valued flag. - The "unreadable files" test described an impossible state: I/O, permission and non-UTF-8 failures throw BundleLoadException and abort the load, while ParseErrors only collects DocumentParseException and ConceptIdException. Reframed around an invalid-frontmatter document. - The agent tool would have fallen back to SystemClock, bypassing the existing UtcNow/Today seam that ReadConcept and Search already use, making its output depend on the run date. It now passes Today through a private IOkfClock adapter, with a test pinning today == stale_after. - "No fixture created or modified" contradicted the new golden files; stated precisely: no fixture bundle and no existing golden change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…umptions The plan covers the spec in six tasks (core query, CLI text modes, JSON, goldens, agent tool, docs), each TDD with real test and implementation code rather than descriptions. Writing it surfaced that the spec was written against the wrong base. FlagValue and Positional's valuedFlags parameter do not exist on origin/main -- both come from the unmerged viewer branch, which added them for `render --out`, and the spec cited that as existing precedent. The plan now opens with Task 0 adding them, copied verbatim from that branch so the eventual merge conflict resolves to identical code, and the spec no longer claims a precedent this base does not have. Same for the subcommand count: six here, not seven. Self-review also caught, in the plan itself: a nullable-narrowing pattern that would not compile under warnings-as-errors, a cap test that would have counted the "concepts:" header as a finding, two unused vocabulary members, and a nested code fence that broke its own block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

All four verified against the code first: - The tool bypassed RunTool, the guard every bundle-loading tool uses to turn OkfException into an "Error: ..." string. A bundle deleted after construction would have thrown out of a function tool instead of returning an error. Now wrapped, with a test that deletes the directory before the first load. - CmdAudit resolved the positional before validating flag values, so `okf audit --as-of` reported "missing <bundle>" instead of naming the unvalued flag. Reordered, with a Theory over all four valued flags. - Both renderers spelled the trust/status labels as literals, which is precisely the drift AuditVocabulary was introduced to prevent. They now read every label from it; only the display ORDER stays local (the report shows the strongest tier first). - Task 1's interface block still advertised TrustTierNames/StatusNames, removed from the implementation during the previous review. Also folded in two design contracts the audit showed were missing from the spec rather than from the plan: why StalePolicy and AuditQuery.StaleOnly coexist (restitution vs worklist -- opposite questions), and why CLI input parses strictly while frontmatter parsing stays permissive (a typo absorbed into "stable" would make a worklist lie). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

origin/dev carries the merged viewer work (PR #48), so FlagValue and Positional's valuedFlags parameter now exist: Task 0 added them and becomes a no-op, exactly as the plan anticipated. Removed it, restored the spec's `render --out` precedent, and corrected the subcommand count (seven here, eight with audit). Baseline after the merge: 987 tests, 0 failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

… test case

Adds `okf audit`, wired to the ConceptAudit engine shipped earlier: report mode (bundle-wide trust/status/staleness summary plus a worklist), query mode (--stale/--trust/--status/--type select a bare concept-line listing), and --json (source-generated, AOT-safe) for both. Implemented as one unit per the task-2-3 brief's controller ruling, which forbids the intermediate --json stub the original two-task split called for.

Hand-authored against the v0.2 spec (audit has no upstream reference binary), verified by running okf4net's own audit against tests/fixtures/okf_v02 before saving. Both goldens matched the predicted text exactly. The JSON parity test's Windows-only backslash normalization needed a fix along the way: r.Out is serialized JSON, where a native backslash separator is escaped to the two-character sequence `\` in the JSON text itself. The originally drafted `r.Out.Replace('\', '/')` replaced each of those two characters individually, turning one path separator into "//" instead of "/". Matching on the two-character escaped sequence instead of the bare char fixes it.

Two CLI-level review gaps closed: - Audit_report_mode_prints_none_when_nothing_is_stale: report mode's "needs attention: none" branch (WriteAuditReport's empty-Findings early return) had no test exercising it. - Audit_finding_line_reports_no_stale_after_when_the_field_is_absent: FormatAuditFinding's "no-stale-after" freshness label (emitted when a concept has no stale_after at all, as metrics/legacy does) was never asserted. Also corrects JsonOutput's class doc comment, stale since WriteAudit was added: it still credited only validate/info.

Adds a PinnedClock IOkfClock adapter over the existing UtcNow/Today seam and a RenderAudit helper (deliberately separate from the CLI's golden-locked renderer) so okf_audit surfaces ConceptAudit's trust/freshness/lifecycle signals to agents, capped at 20 findings and never throwing (RunTool guard). Not added to WriteToolNames: it is read-only. Updates the existing tests that hard-code OkfBundleTools' tool count/order and the MCP server's read-only-mode tool count now that an 11th unconditional tool exists, and syncs the two READMEs' tool tables/counts accordingly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…lary spellings - RenderAudit now takes staleOnly and only headings the worklist "needs attention" when the selection IS the stale worklist; otherwise it uses the neutral "selected" heading. Previously okf_audit(stale: false) on a perfectly healthy bundle printed "needs attention (N):" over concepts that were not stale at all -- a factual misstatement the agent would relay verbatim. - Move the freshness token ("stale <date>"/"fresh <date>"/"no-stale-after") into AuditVocabulary.Freshness, and the "--trust"/trust comma-list grammar into AuditVocabulary.TryParseTrustTiers, so neither is spelled out twice between the CLI renderer and the agent tool renderer. The CLI's message on an unknown trust tier is unchanged (still pinned by a CliTests assertion); the tool still returns its own usage message. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDM9Aozt9YciNJ3oncrFxG

IsFiltered's doc read as "did the caller pass a filter flag?", but it is true for the CLI's own report-mode query (new AuditQuery(StaleOnly: true)), which no flag produced. Reword it to say precisely what the property answers -- whether the query constrains the selection -- without touching its behavior; All and IsFiltered both stay as mandated by the design spec (§3.1). Pin both ends with a unit test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDM9Aozt9YciNJ3oncrFxG

…ters Only --status deprecated (empty result) and a positional-order [Theory] that ignores stdout covered these two flags before this, so transposing Type and Status in ParseAuditQuery would leave the whole suite green. Add assertions on tests/fixtures/okf_v02 (2 concepts, both type: Metric, both resolving to status: stable): --type Metric selects both lines, --type metric (lowercase) selects none -- pinning the documented ordinal, case-sensitive rule at the CLI boundary -- and --status stable selects both lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDM9Aozt9YciNJ3oncrFxG

- README.md: GetTools() returns eleven tools unconditionally (was "ten"), plus a twelfth -- okf_run_computation -- conditional on an attestation orchestrator, matching the table 18 lines below which already said eleven/twelfth. Also quote the CLI's own --help wording for the audit verb line ("Report trust, freshness and lifecycle across the bundle") instead of paraphrasing it, matching every other verb line in that block. - src/OKF4net.Agents/README.md: "The nine tools" was stale even before this branch and is now three tools behind (okf_audit, okf_get_computation, and the conditional okf_run_computation). Correct the count and list to match GetTools(). - CLAUDE.md: add audit to the CLI verb list, and add one line under src/OKF4net/ pointing at ConceptAudit as the single shared corpus-level query behind okf audit and okf_audit -- the same "do not fork this" guidance already given for ConceptSearch. - ROADMAP.md: record that okf audit shipped, in the same style as the "static render shipped" bundle-viewer entry. - CHANGELOG.md: bold the okf audit entry to match its feature-scale neighbors (**okf render ...**, **okf validate/okf info gain a --json flag**), and remove the blank line that had made the Added list loose around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TDM9Aozt9YciNJ3oncrFxG

Every other okf_audit test calls the C# method directly, so nothing covered the framework's JSON argument binding: a model passes `stale` as a boolean and `trust` as a comma-separated string, and the binding has to reach the method's parameters for the filters to take effect. The bundle is built inline with the UtcNow seam pinned, so one concept is stale AND unverified while another is equally stale but human-reviewed. A binding failure that dropped the trust argument would list both — verified by mutating the script's trust argument to null and watching the test fail, then restoring it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

okf_audit was already reaching the sample's agent — Program.cs filters GetTools() by WriteToolNames, so a new read-only tool is exposed with no code change. What was missing is that nothing told the agent, or the reader, that a corpus-level question is now answerable in one call: a model asked "what is stale?" would browse concept by concept, which is the very behaviour the audit surface exists to replace. Adds one sentence to the system instructions, a "Questions worth asking" section splitting retrieval questions from corpus-level ones, and a note in "What it does" on how the two use the tools differently. The section is honest about the demo: nothing in this bundle is stale yet, so the trust question is the one that bites today, and the bundle starts reporting seven stale concepts on 2027-01-01. Also realigns the sample's Microsoft.Agents.AI pin from 1.15.0 to 1.17.0. That break predates this branch: a dependabot bump moved OKF4net.Agents to 1.17.0 and the sample's own pin stayed put, which NU1605 rejects as a downgrade. Nothing caught it because samples/ is outside OKF4net.sln and outside CI — the sample could not be built at all before this commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Fix CLI option parsing around consumed values and the -- separator; update the plan’s stale expected heading.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a corpus-level okf audit query for trust, lifecycle, and staleness across CLI, Agents/MCP, documentation, and samples.

Changes:

Adds shared audit computation and vocabulary handling.
Adds CLI text/JSON output with filters and golden fixtures.
Exposes the read-only okf_audit tool with tests and documentation.

File summaries

File	Reviewed scope / final note
`tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs`	MCP registration coverage
`tests/OKF4net.Tests/GoldenParityTests.cs`	Golden output parity
`tests/OKF4net.Tests/CliTests.cs`	CLI behavior and output
`tests/OKF4net.Tests/AuditTests.cs`	Core audit logic
`tests/OKF4net.Tests/Agents/OkfBundleToolsTests.cs`	Read-only tool filtering
`tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs`	Agent audit behavior
`tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs`	Tool exposure
`tests/OKF4net.Tests/Agents/AgentIntegrationTests.cs`	End-to-end agent binding
`tests/fixtures/README.md`	Fixture documentation
`tests/fixtures/golden/audit-v02.out`	Text golden output
`tests/fixtures/golden/audit-v02.json`	JSON golden output
`src/OKF4net/Audit.cs`	Shared audit model and computation
`src/OKF4net.Mcp/README.md`	MCP documentation
`src/OKF4net.Cli/OkfCli.cs`	Moderate finding: option parsing must skip consumed values and honor `--`
`src/OKF4net.Cli/JsonOutput.cs`	Audit JSON serialization
`src/OKF4net.Agents/README.md`	Agent documentation
`src/OKF4net.Agents/OkfBundleTools.cs`	Agent audit tool
`samples/acme-retail-agent/src/AcmeRetailAgent/Program.cs`	Sample agent instructions
`samples/acme-retail-agent/src/AcmeRetailAgent/AcmeRetailAgent.csproj`	Dependency alignment
`samples/acme-retail-agent/README.md`	Sample usage guidance
`ROADMAP.md`	Roadmap update
`README.md`	Project usage documentation
`docs/superpowers/specs/2026-08-21-okf-audit-design.md`	Audit design specification
`docs/superpowers/plans/2026-08-21-okf-audit.md`	Nit: update expected heading to `selected (2):`
`CLAUDE.md`	Architecture guidance
`CHANGELOG.md`	Changelog entry

Review details

Suppressed comments (7)

docs/superpowers/specs/2026-08-21-okf-audit-design.md:311

The French word present should be accented as présent here.

JSON doit pouvoir distinguer « champ absent » de « champ present mais illisible »

docs/superpowers/specs/2026-08-21-okf-audit-design.md:556

The French verb assère is misspelled; it should be assure.

la surface de fixtures. Le test l'assère en ligne (`Assert.Equal(0, r.Code)`),

samples/acme-retail-agent/README.md:67

Because okf_audit defaults stale to true, a human-verification query otherwise lists only concepts that are both stale and unverified. skills/run-on-bq has no stale_after, so the example as written does not isolate it; please state that this question must be run with stale: false (or show the exact prompt/tool arguments).

- *Which concepts have never been verified by a human?* — the interesting
  one today: eight of the nine concepts carry a `human:` verifier, so this
  isolates `skills/run-on-bq`.

src/OKF4net.Agents/OkfBundleTools.cs:478

The function metadata still describes the output as concepts “needing attention” even when callers pass stale: false or another non-staleness filter; those results can include fresh concepts and are intentionally headed selected. Since this description is what the model sees when deciding how to use the tool, please describe these as concepts selected by the filters instead.

    [Description("Audit the bundle's trust, freshness and lifecycle signals: counts by trust tier and status, plus the concepts needing attention. Filter with stale/trust/status/type.")]

src/OKF4net.Agents/OkfBundleTools.cs:231

Adding okf_audit here leaves the public web/ reference out of sync: the Home CLI table and docs/Cli.tsx still describe seven commands, while Home/Agents/MCP still describe ten tools and omit the audit surface. Please update those public pages (or explicitly defer the feature from the site) so the shipped API and its documentation do not contradict each other.

            AIFunctionFactory.Create(Audit, "okf_audit"),

tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs:31

The exposure suite now lists okf_audit, but it does not assert the new function's generated schema. The end-to-end test only binds stale and trust, so a regression that drops or renames the optional status/type parameters (or changes their requiredness) would still pass; add the same schema/optional-parameter coverage used for the other exposed tools.

        "okf_audit",

tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs:95

The MCP tests only prove that okf_audit appears in the tool list; they never invoke it through CallToolAsync. Because this tool adds multiple optional arguments and the MCP adapter performs its own schema/argument conversion, a binding or invocation regression could pass while the Agent-level test remains green. Add an MCP call using the audit filters and assert its returned finding.

                    "okf_append_log", "okf_audit", "okf_browse", "okf_changes_since", "okf_get_computation",

Files reviewed: 26/26 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+
+        var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false);
+
+        Assert.Contains("needs attention (2):", text);


+        }
+
+        return new AuditQuery(
+            HasFlag(args, "--stale"),


ncitnea and others added 20 commits August 21, 2026 21:02

Merge remote-tracking branch 'origin/dev' into worktree-okf-audit

8223e0b

feat(audit): add ConceptAudit, the shared corpus-level query

53a82e0

fix(audit): use component-wise ConceptId ordering, add discriminating…

0567704

… test case

docs(audit): document the okf audit verb and okf_audit tool

bea824f

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot AI lite review requested due to automatic review settings August 22, 2026 12:14

Copilot started reviewing on behalf of jchable August 22, 2026 12:15 View session

jchable merged commit 3fe16f7 into dev Aug 22, 2026
7 checks passed

Copilot AI reviewed Aug 22, 2026

View reviewed changes

Comment thread docs/superpowers/plans/2026-08-21-okf-audit.md

var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false);

Assert.Contains("needs attention (2):", text);

Comment thread src/OKF4net.Cli/OkfCli.cs

}

return new AuditQuery(

HasFlag(args, "--stale"),

jchable deleted the worktree-okf-audit branch August 22, 2026 18:06

Navigation Menu

feat(audit): okf audit — corpus-level trust, freshness and lifecycle query - #49

feat(audit): okf audit — corpus-level trust, freshness and lifecycle query#49
jchable merged 20 commits into
devfrom
worktree-okf-audit

jchable commented Aug 22, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false);

		Assert.Contains("needs attention (2):", text);

Conversation

jchable commented Aug 22, 2026

What

Design

Verification

Also in here

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (7)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants