diff --git a/CHANGELOG.md b/CHANGELOG.md index 800e89c..0bf11da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ and this project adheres to ### Added +- **`okf audit`** — a corpus-level query over a bundle's trust (§5.3), lifecycle + (§5.4) and staleness (§5.5) signals: counts plus a filterable worklist, with + `--stale`, `--trust`, `--status`, `--type`, `--as-of` and `--json`. Backed by + the new `ConceptAudit` in the core library and exposed to agents as the + read-only `okf_audit` tool. - **`okf render --out `** generates a self-contained, browsable HTML site from a bundle: one page per concept (frontmatter table + rendered body), a generated index, navigable cross-links with broken links flagged, diff --git a/CLAUDE.md b/CLAUDE.md index cbb3bc9..4225381 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,10 +33,11 @@ Requires .NET SDK 10.0+. CI (ci.yml) runs build+test on Linux/Windows/macOS, `do - **`src/OKF4net/`** — the library. One file per spec concern, following the OKF reference implementation's structure: `ConceptId` (§2), `Bundle` (§3, permissive loading — parse failures go into `Bundle.ParseErrors`, never abort), `OkfDocument`/`Frontmatter` (§4), `Links.cs`/`LinkScanner` (§6, legacy citations §13.1), `IndexGenerator` (§8), `ChangeLog` (§9), `Validate.cs`/`BundleValidator` (§11). The README has the full spec-section → type mapping table. - `ConceptSearch` — the single shared full-text scorer (title x3, tags/description x2, body x1) used by both `OKF4net.Agents` (`okf_search`/context provider) and `OKF4net.Catalog` (`OkfBundleKnowledgeSource`, `FileMemoryStore`); do not fork a second scorer in either consumer. + - `Audit.cs` — `ConceptAudit`, the single shared corpus-level query behind both `okf audit` and the `okf_audit` tool; the two renderers are deliberately separate (the CLI's bytes are golden-locked), but the computation and the `AuditVocabulary` labels must not be forked. - `Yaml/` — the documented YAML *subset* (scalars, lists, shallow maps, block/flow, `|`/`>`); it deliberately rejects anchors/tags/multi-docs with clear errors. `Frontmatter` wraps an order-preserving `YamlMapping` with typed getters rather than a fixed DTO, so unknown producer keys survive round-trips. - `Internal/LfLines.cs` — the single shared line splitter (splits on `\n` only, stripping a preceding `\r`). Use it anywhere `\n`-based line splitting matters; do not reintroduce private copies. - `Internal/ReparsePoints.cs` — internal symlink/junction detection; `OKF4net.Catalog` is granted `InternalsVisibleTo` so it can reuse this seam rather than duplicating a platform-specific implementation. -- **`src/OKF4net.Cli/`** — the `okf` binary (`validate`/`info`/`index`/`graph`/`parse`/`fmt`/`render`), published Native AOT (`PublishAot`, `InvariantGlobalization`). All logic lives in `OkfCli.Run(args, out, err)` so tests invoke it in-process without spawning a process. +- **`src/OKF4net.Cli/`** — the `okf` binary (`validate`/`audit`/`info`/`index`/`graph`/`parse`/`fmt`/`render`), published Native AOT (`PublishAot`, `InvariantGlobalization`). All logic lives in `OkfCli.Run(args, out, err)` so tests invoke it in-process without spawning a process. - **`src/OKF4net.Attestation/`** — zero-dep §10 attested-computation orchestration, referencing only `OKF4net`. Defines the host-plugged contracts (`IParameterBinder`, `IComputationExecutor`, `IAttester`, resolved per concept's `runtime` field through `IAttestationRuntimeRegistry`) and the value types that flow between them (`BoundComputation`, `Receipt`, `AttestationVerdict`, `AttestationContext`, `AttestationOutcome`); `AttestationOrchestrator.RunAsync` drives one run end to end (resolve → bind → execute → receipt-shape check → attest → gate on verdict + `stale_after`), errors-as-data, never writing a verdict back to the bundle (§10.6). Referenced by `OKF4net.Agents` to back `okf_run_computation`. - **`src/OKF4net.Agents/`** — Microsoft Agent Framework layer exposing OKF bundle operations as function tools (e.g. `OkfBundleTools`) plus `OkfContextProvider`, an `AIContextProvider` that auto-injects budget-bounded bundle context and captures deterministic per-day memory concepts; the only project depending on `Microsoft.Agents.AI`. - **`src/OKF4net.Catalog/`** — knowledge-catalog model and logic, referencing only `OKF4net` (BCL otherwise; zero `PackageReference`). Depended on by `OKF4net.Catalog.Hosting`. Each manifest source carries a `role` (`SourceRole`): `Knowledge` (read-only, searched by `IKnowledgeResolver`) or `Memory` (writable, scoped by a required `tier` — `session`/`user`/`tenant`, all three backed by `FileMemoryStore`, fed by `IMemoryStore`, never searched by the resolver); any other `role` string in `catalog.json` is rejected (`CatalogDiagnosticCode.IllegalRole`). diff --git a/README.md b/README.md index 870a0cd..8e5fda2 100644 --- a/README.md +++ b/README.md @@ -176,6 +176,7 @@ On any OS, build from source — see [Building & testing](#building--testing). ``` okf validate Check a bundle against OKF v0.2 conformance (§11) +okf audit Report trust, freshness and lifecycle across the bundle okf info Summarize a bundle (concepts, types, links, version) okf index (Re)generate every index.md in the bundle okf graph Print the cross-link graph (--dot for Graphviz DOT) @@ -192,6 +193,21 @@ okf validate ./bundles/ga4 okf graph ./bundles/ga4 --dot | dot -Tsvg > graph.svg ``` +`okf audit ` reports trust, freshness and lifecycle across a whole +bundle: counts per trust tier (§5.3) and status (§5.4), plus the worklist of +stale concepts (§5.5). Filter it to ask corpus-level questions — + +```sh +# Which concepts are past stale_after and were never verified by a human? +okf audit bundles/acme_retail --stale --trust unverified,machine-confirmed +``` + +Without filter flags it selects exactly what `--stale` selects and prints the +summary form; with any filter flag it prints one line per matching concept, so +the output pipes. `--json` always emits the full document. Note the counts +always cover the whole bundle while `findings` covers the selection: `audit` is +a worklist, not an inventory (use `okf info --json` for that). + Generate a browsable HTML site from a bundle: ```sh @@ -213,8 +229,8 @@ machine. Full command reference with real output samples: `src/OKF4net.Agents/` exposes bundle operations as function tools for the [Microsoft Agent Framework](https://github.com/microsoft/agent-framework): `OkfBundleTools` wraps one bundle root and its `GetTools()` method returns -ten ready-to-use `AITool`s unconditionally, which `AsAIAgent` turns into an -agent's tool list, plus an eleventh — `okf_run_computation` — only when the +eleven ready-to-use `AITool`s unconditionally, which `AsAIAgent` turns into an +agent's tool list, plus a twelfth — `okf_run_computation` — only when the tool set is constructed with an `OKF4net.Attestation` orchestrator wired in (see [Attested computation](#attested-computation-okf4netattestation)). @@ -231,9 +247,9 @@ var response = await agent.RunAsync("Search the bundle for concepts about refund Console.WriteLine(response.Text); ``` -The ten unconditional tools, plus the eleventh conditional on an attestation -orchestrator being wired (read → browse → graph → search → write → append → -regenerate → validate → changes-since → get-computation → run-computation): +The eleven unconditional tools, plus the twelfth conditional on an attestation +orchestrator being wired (read → browse → graph → search → audit → write → +append → regenerate → validate → changes-since → get-computation → run-computation): | Tool | Description | |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| @@ -241,6 +257,7 @@ regenerate → validate → changes-since → get-computation → run-computatio | `okf_browse` | Browse the bundle via its index files (progressive disclosure). Without a path, lists the bundle root. | | `okf_graph` | Inspect the cross-link graph. With a concept id: its outgoing links, backlinks and broken links. Without: bundle-wide stats. | | `okf_search` | Full-text search across concept titles, descriptions, tags and bodies. Returns matching concept ids ranked by relevance. | +| `okf_audit` | Audit the bundle's trust, freshness and lifecycle signals (§5.3–§5.5): counts by trust tier and status, plus the concepts needing attention. Read-only. | | `okf_write_concept` | Create or update a concept document. The frontmatter must contain non-empty type, title and description (producer-grade validation is enforced before writing). | | `okf_append_log` | Append an entry to the bundle root log.md under today's date (ISO). Note: log.md is re-rendered through the strict §9 model, so non-conforming prose or comments in a hand-authored log.md are not preserved. | | `okf_regenerate_indexes` | Regenerate every index.md in the bundle (progressive-disclosure listings). Run after adding or changing concepts. | @@ -507,6 +524,7 @@ This table is also published as the | §4 Concept documents | `OKF4net.OkfDocument`, `OKF4net.Frontmatter` | | §4.2 Body headings | `OkfDocument.Computation()` (fenced `# Computation` heading) | | §5 Provenance, trust, and lifecycle | `Frontmatter.Sources`/`Generated`/`Verified`/`TrustTier`/`Status`/`StaleAfter`, `Actor`/`Trust`/`Provenance`/`Lifecycle` | +| §5.3–§5.5 | `ConceptAudit`, `AuditQuery`, `AuditReport` — corpus-level trust/freshness query behind `okf audit` and `okf_audit` | | §6 Cross-linking and paths | `OKF4net.LinkScanner`, `Bundle.LinksFrom` / `Bundle.Backlinks` | | §6.2 Path-valued fields | `OkfDocument.FrontmatterResources()`, `Bundle.TryResolveResource` / `Bundle.ReadResourceText` | | §7 Actor convention | `OKF4net.Actor.Parse` — `human:`/`process:`/`/` | diff --git a/ROADMAP.md b/ROADMAP.md index cd0cba3..fabf3a0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -14,6 +14,12 @@ are the concrete entry points. ## Next +- **`okf audit` shipped** — a corpus-level query over a bundle's trust (§5.3), + lifecycle (§5.4) and staleness (§5.5) signals: counts plus a filterable + worklist, across the CLI verb and the read-only `okf_audit` agent tool, + backed by the shared `ConceptAudit`/`AuditVocabulary` model in `OKF4net`. + Motivated by ["OKF v0.2 Quietly Admits the Folder Has a Ceiling"](https://medium.com/@davidroliver/okf-v0-2-quietly-admits-the-folder-has-a-ceiling-the-way-up-is-a-library-25fa54e872f9) + — see [its design spec](docs/superpowers/specs/2026-08-21-okf-audit-design.md). - More `OKF4net.Agents` samples with Microsoft Agent Framework — the first, `samples/acme-retail-agent`, shipped in 0.4.0; more welcome. - `OKF4net.Catalog` samples: `samples/catalog-explorer` (multi-source diff --git a/docs/superpowers/plans/2026-08-21-okf-audit.md b/docs/superpowers/plans/2026-08-21-okf-audit.md new file mode 100644 index 0000000..b063301 --- /dev/null +++ b/docs/superpowers/plans/2026-08-21-okf-audit.md @@ -0,0 +1,1638 @@ +# `okf audit` Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ajouter une requête corpus-level sur les signaux de confiance et de fraîcheur d'un bundle OKF — « quels concepts ont dépassé `stale_after` sans avoir jamais été vérifiés par un humain ? » — exposée en type de cœur, en verbe CLI et en tool agent. + +**Architecture:** Un seul calcul (`ConceptAudit` dans `src/OKF4net/`), trois consommateurs qui rendent chacun leur propre présentation : le verbe CLI `okf audit` (texte + `--json`), et le tool agent `okf_audit`. Aucun champ frontmatter nouveau : tout dérive de `Frontmatter.TrustTier` (§5.3) et `Frontmatter.Lifecycle` (§5.4/§5.5), déjà présents. + +**Tech Stack:** C# / net10.0, xunit, `System.Text.Json` source-generated (Native AOT), zéro dépendance tierce. + +**Spec:** [docs/superpowers/specs/2026-08-21-okf-audit-design.md](../specs/2026-08-21-okf-audit-design.md) + +## Global Constraints + +- **Zéro dépendance tierce** dans `src/OKF4net/` et `src/OKF4net.Cli/` : BCL uniquement, aucun `PackageReference` ajouté. +- **Tout nouveau fichier source** commence par `// SPDX-License-Identifier: LGPL-3.0-or-later`. +- **Namespaces file-scoped**, XML doc obligatoire sur toute API publique, nullable activé, `TreatWarningsAsErrors` — un warning casse le build. +- **Native AOT** : toute sérialisation JSON passe par `CliJsonContext` (source-generated) ; tout formatage/parsing de date utilise `CultureInfo.InvariantCulture` (le CLI est publié avec `InvariantGlobalization`). +- **Fixtures** : ne jamais modifier un fichier existant sous `tests/fixtures/`. Ce plan n'ajoute que deux goldens neufs, écrits à la main. +- **Aucune sortie existante ne change** : `validate`, `info`, `graph`, `parse`, `fmt`, `render` produisent les mêmes octets qu'avant. Si un golden existant casse, c'est une régression à corriger côté code. +- **Vérification** : `dotnet build OKF4net.sln` (warnings = erreurs) et `dotnet test OKF4net.sln` doivent être verts avant chaque commit. Baseline au départ de cette branche : **987 tests, 0 échec**. +- **Format** : `dotnet format OKF4net.sln` avant le dernier commit (la CI lance `--verify-no-changes`). + +## Base de la branche + +`origin/dev` a été mergé dans cette branche le 2026-08-21 : le travail viewer +(PR #48) y est, donc le CLI a **sept** sous-commandes (`validate`, `info`, +`index`, `graph`, `parse`, `fmt`, `render`) et le projet `OKF4net.Viewer` existe. + +Ce merge a rendu caduque la Task 0 de la version précédente de ce plan, qui +ajoutait `FlagValue` et le paramètre `valuedFlags` de `Positional` : **les deux +existent désormais**, introduits par la branche viewer pour `render --out`. Ils +sont à utiliser tels quels, sans les modifier — la spec §4.1 les décrit comme le +précédent à suivre, ce qu'ils sont redevenus. + +Baseline après merge : **987 tests, 0 échec**. + +## Écarts assumés par rapport à la spec + +Deux points où ce plan précise la spec plutôt que de la suivre à la lettre — à signaler en revue : + +1. **`AuditVocabulary` (Task 1) n'était pas dans le croquis d'API de la spec.** La spec exige « un seul vocabulaire partout (entrée CLI, texte, JSON, tool agent) ». Sans type partagé, cette exigence se traduirait par des littéraux `"human-reviewed"` dupliqués dans `OKF4net.Cli` et `OKF4net.Agents`, libres de diverger sans qu'aucun test ne le voie. `AuditVocabulary` est le mécanisme qui rend l'exigence réelle. +2. **`PinnedClock` reste privé, dupliqué dans le CLI et dans les Agents** (4 lignes chacun), comme le demande la spec §5. Un `PinnedClock` public dans le cœur supprimerait les deux copies (et la troisième, `FixedClock`, côté tests) mais élargirait la surface publique hors périmètre. Noté comme amélioration séparée possible, non faite ici. + +## Structure des fichiers + +| Fichier | Rôle | Task | +|---|---|---| +| `src/OKF4net/Audit.cs` (créé) | `AuditQuery`, `AuditFinding`, `AuditReport`, `ConceptAudit`, `AuditVocabulary` — le calcul et le vocabulaire partagés | 1 | +| `tests/OKF4net.Tests/AuditTests.cs` (créé) | Tests unitaires du cœur | 1 | +| `src/OKF4net.Cli/OkfCli.cs` (modifié) | Usage, dispatch, `CmdAudit`, parsing des flags, rendu texte | 2 | +| `tests/OKF4net.Tests/CliTests.cs` (modifié) | Tests CLI texte + erreurs + parsing | 2 | +| `src/OKF4net.Cli/JsonOutput.cs` (modifié) | Records JSON, `WriteAudit`, enregistrement dans `CliJsonContext` | 3 | +| `tests/fixtures/golden/audit-v02.out` (créé) | Golden texte | 4 | +| `tests/fixtures/golden/audit-v02.json` (créé) | Golden JSON | 4 | +| `tests/OKF4net.Tests/GoldenParityTests.cs` (modifié) | Comparaison byte-exact + provenance | 4 | +| `tests/fixtures/README.md` (modifié) | Provenance des deux goldens | 4 | +| `src/OKF4net.Agents/OkfBundleTools.cs` (modifié) | Tool `okf_audit` + adaptateur d'horloge | 5 | +| `tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs` (créé) | Tests du tool | 5 | +| `README.md`, `CHANGELOG.md` (modifiés) | Documentation | 6 | + +`ROADMAP.md` et `CLAUDE.md` sont **délibérément exclus** : ils sont modifiés en parallèle par une autre session (spec §8, « Coordination »). Ils seront traités en fin de branche, après merge de l'autre travail. + +--- + +### Task 1: Le calcul partagé (`ConceptAudit`) + +**Files:** +- Create: `src/OKF4net/Audit.cs` +- Test: `tests/OKF4net.Tests/AuditTests.cs` + +**Interfaces:** +- Consumes: `Bundle.Concepts` (`IReadOnlyList`, chaque `Concept` porte `Id`, `Path`, `Document`), `Frontmatter.TrustTier`, `Frontmatter.Lifecycle`, `Frontmatter.Type`, `Frontmatter.Title`, `Lifecycle.IsStale(DateOnly)`, `Lifecycle.StaleAfter`, `Lifecycle.StaleAfterRaw`, `Lifecycle.Status`, `IOkfClock`/`SystemClock`. +- Produces: `ConceptAudit.Run(Bundle, AuditQuery, IOkfClock?) → AuditReport` ; `AuditQuery(bool StaleOnly, IReadOnlySet? Trust, ConceptStatus? Status, string? Type)` avec `AuditQuery.All` et `IsFiltered` ; `AuditFinding(ConceptId Id, string Path, string? Type, string? Title, TrustTier Trust, Lifecycle Lifecycle, bool IsStale)` ; `AuditReport` avec `AsOf`, `ConceptCount`, `TrustCounts`, `StatusCounts`, `StaleCount`, `Findings` ; `AuditVocabulary.Name(TrustTier)`, `Name(ConceptStatus)`, `TryParseTrustTier`, `TryParseStatus`, `TrustTiersInOrder`, `StatusesInOrder`. + +- [ ] **Step 1: Écrire les tests unitaires qui échouent** + +Créer `tests/OKF4net.Tests/AuditTests.cs` : + +```csharp +// SPDX-License-Identifier: LGPL-3.0-or-later +namespace OKF4net.Tests; + +/// +/// Unit tests for : tier and status counting, +/// the §5.5 staleness boundary, predicate composition and ordering. +/// Every test pins the date with so nothing here +/// depends on the day the suite runs. +/// +public class AuditTests +{ + private static readonly DateOnly Today = new(2026, 8, 21); + + private static Bundle Load(TempDir tmp) => Bundle.Load(tmp.Path); + + private static AuditReport Audit(TempDir tmp, AuditQuery query = default) + => ConceptAudit.Run(Load(tmp), query, new FixedClock(Today)); + + [Fact] + public void Counts_the_three_trust_tiers() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\nverified:\n - { by: process:nightly, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("c.md", "---\ntype: Metric\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(3, report.ConceptCount); + Assert.Equal(1, report.TrustCounts[TrustTier.HumanReviewed]); + Assert.Equal(1, report.TrustCounts[TrustTier.MachineConfirmed]); + Assert.Equal(1, report.TrustCounts[TrustTier.Unverified]); + } + + [Fact] + public void Unknown_status_counts_as_stable() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstatus: retired\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\nstatus: draft\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(1, report.StatusCounts[ConceptStatus.Stable]); + Assert.Equal(1, report.StatusCounts[ConceptStatus.Draft]); + Assert.Equal(0, report.StatusCounts[ConceptStatus.Deprecated]); + } + + [Theory] + [InlineData("2026-08-21", true)] // §5.5: today >= stale_after -- the exact boundary IS stale. + [InlineData("2026-08-22", false)] + public void Staleness_boundary_follows_section_5_5(string staleAfter, bool expectedStale) + { + using var tmp = new TempDir(); + tmp.Write("a.md", $"---\ntype: Metric\nstale_after: {staleAfter}\n---\n"); + + // The default query filters nothing, so the single concept is always returned. + var report = Audit(tmp); + + Assert.Equal(expectedStale, report.Findings.Single().IsStale); + Assert.Equal(expectedStale ? 1 : 0, report.StaleCount); + } + + [Fact] + public void Malformed_or_absent_stale_after_is_never_stale() + { + using var tmp = new TempDir(); + tmp.Write("bad.md", "---\ntype: Metric\nstale_after: not-a-date\n---\n"); + tmp.Write("none.md", "---\ntype: Metric\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(0, report.StaleCount); + Assert.Empty(ConceptAudit.Run(Load(tmp), new AuditQuery(StaleOnly: true), new FixedClock(Today)).Findings); + Assert.True(report.Findings.Single(f => f.Id.ToString() == "bad").Lifecycle.StaleAfterMalformed); + } + + [Fact] + public void Predicates_compose_with_and() + { + using var tmp = new TempDir(); + tmp.Write("stale-unverified.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + tmp.Write("stale-human.md", "---\ntype: Metric\nstale_after: 2026-01-01\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("fresh-unverified.md", "---\ntype: Metric\nstale_after: 2099-01-01\n---\n"); + + var query = new AuditQuery( + StaleOnly: true, + Trust: new HashSet { TrustTier.Unverified }); + + var findings = ConceptAudit.Run(Load(tmp), query, new FixedClock(Today)).Findings; + + Assert.Equal(["stale-unverified"], findings.Select(f => f.Id.ToString())); + } + + [Fact] + public void Findings_are_sorted_by_concept_id_ordinal() + { + using var tmp = new TempDir(); + tmp.Write("zeta.md", "---\ntype: Metric\n---\n"); + tmp.Write("alpha.md", "---\ntype: Metric\n---\n"); + tmp.Write("mid/beta.md", "---\ntype: Metric\n---\n"); + + var findings = Audit(tmp).Findings.Select(f => f.Id.ToString()).ToList(); + + Assert.Equal(["alpha", "mid/beta", "zeta"], findings); + } + + [Fact] + public void Counts_cover_the_whole_bundle_even_when_the_query_filters() + { + using var tmp = new TempDir(); + tmp.Write("stale.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + tmp.Write("fresh.md", "---\ntype: Metric\n---\n"); + + var report = ConceptAudit.Run(Load(tmp), new AuditQuery(StaleOnly: true), new FixedClock(Today)); + + Assert.Single(report.Findings); + Assert.Equal(2, report.ConceptCount); + Assert.Equal(2, report.TrustCounts[TrustTier.Unverified]); + Assert.Equal(1, report.StaleCount); + } + + [Fact] + public void Empty_bundle_yields_zeroed_counts_and_no_findings() + { + using var tmp = new TempDir(); + + var report = Audit(tmp); + + Assert.Equal(0, report.ConceptCount); + Assert.Empty(report.Findings); + Assert.Equal(0, report.TrustCounts[TrustTier.HumanReviewed]); + Assert.Equal(0, report.StatusCounts[ConceptStatus.Deprecated]); + } + + /// + /// A document whose frontmatter cannot be parsed lands in + /// Bundle.ParseErrors (permissive loading) and is not a concept, so + /// it must not reach any counter. Note this is a *parse* failure -- a truly + /// unreadable file (I/O, permissions, non-UTF-8) throws + /// BundleLoadException and never reaches . + /// + [Fact] + public void Unparseable_documents_are_excluded_from_every_count() + { + using var tmp = new TempDir(); + tmp.Write("ok.md", "---\ntype: Metric\n---\n"); + tmp.Write("broken.md", "---\ntype: Metric\n"); // unterminated frontmatter block + + var bundle = Load(tmp); + Assert.Single(bundle.ParseErrors); + + var report = ConceptAudit.Run(bundle, default, new FixedClock(Today)); + + Assert.Equal(1, report.ConceptCount); + Assert.Single(report.Findings); + } + + [Fact] + public void Null_clock_falls_back_to_today_in_utc() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + + var report = ConceptAudit.Run(Load(tmp)); + + Assert.Equal(DateOnly.FromDateTime(DateTime.UtcNow.Date), report.AsOf); + } + + [Fact] + public void Type_filter_is_exact_and_ordinal() + { + using var tmp = new TempDir(); + tmp.Write("metric.md", "---\ntype: Metric\n---\n"); + tmp.Write("lower.md", "---\ntype: metric\n---\n"); + tmp.Write("untyped.md", "---\ntitle: No type here\n---\n"); + + var findings = ConceptAudit.Run(Load(tmp), new AuditQuery(Type: "Metric"), new FixedClock(Today)).Findings; + + Assert.Equal(["metric"], findings.Select(f => f.Id.ToString())); + } + + [Fact] + public void Vocabulary_names_round_trip() + { + Assert.Equal("human-reviewed", AuditVocabulary.Name(TrustTier.HumanReviewed)); + Assert.Equal("machine-confirmed", AuditVocabulary.Name(TrustTier.MachineConfirmed)); + Assert.Equal("unverified", AuditVocabulary.Name(TrustTier.Unverified)); + Assert.Equal("draft", AuditVocabulary.Name(ConceptStatus.Draft)); + + Assert.True(AuditVocabulary.TryParseTrustTier("machine-confirmed", out var tier)); + Assert.Equal(TrustTier.MachineConfirmed, tier); + Assert.False(AuditVocabulary.TryParseTrustTier("machine", out _)); + + Assert.True(AuditVocabulary.TryParseStatus("deprecated", out var status)); + Assert.Equal(ConceptStatus.Deprecated, status); + Assert.False(AuditVocabulary.TryParseStatus("retired", out _)); + } +} +``` + +- [ ] **Step 2: Lancer les tests pour vérifier qu'ils échouent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~AuditTests"` +Expected: échec de compilation — `ConceptAudit`, `AuditQuery`, `AuditReport`, `AuditVocabulary` n'existent pas. + +- [ ] **Step 3: Implémenter `src/OKF4net/Audit.cs`** + +```csharp +// SPDX-License-Identifier: LGPL-3.0-or-later +namespace OKF4net; + +/// +/// The selection predicates of an audit (§5.3–§5.5). Predicates combine with +/// AND; default selects every concept. +/// +/// +/// The generated equality compares by reference (the +/// behaviour of EqualityComparer<IReadOnlySet<T>>.Default), so two +/// logically identical queries may compare unequal. Do not rely on it, and do +/// not use an as a dictionary key: the record struct is +/// for with and ToString, not for its equality. +/// +/// Keep only concepts past their stale_after date. +/// Keep only concepts whose derived tier is in this set; null keeps every tier. +/// Keep only concepts with this lifecycle status; null keeps every status. +/// Keep only concepts whose frontmatter type matches exactly (ordinal); null keeps every type. +public readonly record struct AuditQuery( + bool StaleOnly = false, + IReadOnlySet? Trust = null, + ConceptStatus? Status = null, + string? Type = null) +{ + /// The query that keeps every concept. + public static AuditQuery All => default; + + /// True as soon as one predicate is set. + public bool IsFiltered => StaleOnly || Trust is not null || Status is not null || Type is not null; +} + +/// One concept selected by an audit, with its signals already derived. +/// The concept id (§2). +/// The concept's file path, as built from the bundle root. +/// The frontmatter type, or null when absent. +/// The frontmatter title, or null when absent. +/// The derived trust tier (§5.3). +/// The lifecycle fields (§5.4/§5.5). +/// Whether the concept is stale as of the report's . +public readonly record struct AuditFinding( + ConceptId Id, + string Path, + string? Type, + string? Title, + TrustTier Trust, + Lifecycle Lifecycle, + bool IsStale); + +/// +/// The result of an audit: counts over the whole bundle, plus the concepts the +/// query selected. The counts never narrow with the query -- the denominator +/// stays stable while moves. +/// +public sealed class AuditReport +{ + internal AuditReport( + DateOnly asOf, + int conceptCount, + IReadOnlyDictionary trustCounts, + IReadOnlyDictionary statusCounts, + int staleCount, + IReadOnlyList findings) + { + AsOf = asOf; + ConceptCount = conceptCount; + TrustCounts = trustCounts; + StatusCounts = statusCounts; + StaleCount = staleCount; + Findings = findings; + } + + /// The observation date staleness was evaluated against. + public DateOnly AsOf { get; } + + /// The number of concepts in the bundle (not in the selection). + public int ConceptCount { get; } + + /// Concept counts per trust tier over the whole bundle; all three keys are always present. + public IReadOnlyDictionary TrustCounts { get; } + + /// Concept counts per lifecycle status over the whole bundle; all three keys are always present. + public IReadOnlyDictionary StatusCounts { get; } + + /// The number of stale concepts in the whole bundle. + public int StaleCount { get; } + + /// The selected concepts, sorted by concept id (ordinal). + public IReadOnlyList Findings { get; } +} + +/// +/// The single spelling of the audit vocabularies, shared by every surface (CLI +/// input, CLI text, JSON, agent tool) so no two layers can drift apart. +/// +public static class AuditVocabulary +{ + /// + /// The trust tiers, weakest to strongest -- the canonical order. JSON + /// serializes in this order; the text report walks it in reverse, showing + /// the strongest tier first. + /// + public static IReadOnlyList TrustTiersInOrder { get; } = + [TrustTier.Unverified, TrustTier.MachineConfirmed, TrustTier.HumanReviewed]; + + /// The lifecycle statuses in §5.4 order -- the order every surface displays them in. + public static IReadOnlyList StatusesInOrder { get; } = + [ConceptStatus.Draft, ConceptStatus.Stable, ConceptStatus.Deprecated]; + + /// The wire/display name of a trust tier. + public static string Name(TrustTier tier) => tier switch + { + TrustTier.HumanReviewed => "human-reviewed", + TrustTier.MachineConfirmed => "machine-confirmed", + _ => "unverified", + }; + + /// The wire/display name of a lifecycle status. + public static string Name(ConceptStatus status) => status switch + { + ConceptStatus.Draft => "draft", + ConceptStatus.Deprecated => "deprecated", + _ => "stable", + }; + + /// Parses a trust tier name (exact, ordinal). Unlike frontmatter parsing, an unknown name fails rather than defaulting. + public static bool TryParseTrustTier(string text, out TrustTier tier) + { + switch (text) + { + case "unverified": tier = TrustTier.Unverified; return true; + case "machine-confirmed": tier = TrustTier.MachineConfirmed; return true; + case "human-reviewed": tier = TrustTier.HumanReviewed; return true; + default: tier = TrustTier.Unverified; return false; + } + } + + /// Parses a lifecycle status name (exact, ordinal). Unlike , an unknown name fails rather than resolving to stable. + public static bool TryParseStatus(string text, out ConceptStatus status) + { + switch (text) + { + case "draft": status = ConceptStatus.Draft; return true; + case "stable": status = ConceptStatus.Stable; return true; + case "deprecated": status = ConceptStatus.Deprecated; return true; + default: status = ConceptStatus.Stable; return false; + } + } +} + +/// +/// Queries a bundle's §5.3–§5.5 signals. Reads nothing from disk, writes +/// nothing, and never throws on data: an unparseable document is already absent +/// from , and a malformed stale_after simply +/// reads as "not stale" (the validator owns that diagnostic). +/// +public static class ConceptAudit +{ + /// Runs over . + /// The loaded bundle. + /// The selection predicates; default selects everything. + /// Supplies "today" for staleness (§5.5); defaults to . + public static AuditReport Run(Bundle bundle, AuditQuery query = default, IOkfClock? clock = null) + { + var asOf = (clock ?? new SystemClock()).Today; + + var trustCounts = new Dictionary + { + [TrustTier.Unverified] = 0, + [TrustTier.MachineConfirmed] = 0, + [TrustTier.HumanReviewed] = 0, + }; + + var statusCounts = new Dictionary + { + [ConceptStatus.Draft] = 0, + [ConceptStatus.Stable] = 0, + [ConceptStatus.Deprecated] = 0, + }; + + var staleCount = 0; + var findings = new List(); + + foreach (var concept in bundle.Concepts) + { + var frontmatter = concept.Document.Frontmatter; + var tier = frontmatter.TrustTier; + var lifecycle = frontmatter.Lifecycle; + var isStale = lifecycle.IsStale(asOf); + + trustCounts[tier]++; + statusCounts[lifecycle.Status]++; + if (isStale) + { + staleCount++; + } + + if (query.StaleOnly && !isStale) + { + continue; + } + + if (query.Trust is { } tiers && !tiers.Contains(tier)) + { + continue; + } + + if (query.Status is { } status && lifecycle.Status != status) + { + continue; + } + + if (query.Type is { } type && !string.Equals(frontmatter.Type, type, StringComparison.Ordinal)) + { + continue; + } + + findings.Add(new AuditFinding( + concept.Id, + concept.Path, + frontmatter.Type, + frontmatter.Title, + tier, + lifecycle, + isStale)); + } + + findings.Sort(static (a, b) => string.CompareOrdinal(a.Id.ToString(), b.Id.ToString())); + + return new AuditReport(asOf, bundle.Count, trustCounts, statusCounts, staleCount, findings); + } +} +``` + +- [ ] **Step 4: Lancer les tests pour vérifier qu'ils passent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~AuditTests"` +Expected: PASS — 12 méthodes de test, 13 cas exécutés (la `[Theory]` en compte +deux). Puis `dotnet test OKF4net.sln` en entier : 987 tests de base + les +nouveaux, 0 échec. + +- [ ] **Step 5: Commit** + +```bash +git add src/OKF4net/Audit.cs tests/OKF4net.Tests/AuditTests.cs +git commit -m "feat(audit): add ConceptAudit, the shared corpus-level query" +``` + +--- + +### Task 2: Le verbe CLI `okf audit` (sortie texte) + +**Files:** +- Modify: `src/OKF4net.Cli/OkfCli.cs` (constante `Usage`, `switch` de `Run`, nouvelles méthodes privées) +- Test: `tests/OKF4net.Tests/CliTests.cs` + +**Interfaces:** +- Consumes: Task 1 (`ConceptAudit.Run`, `AuditQuery`, `AuditReport`, `AuditFinding`, `AuditVocabulary`), plus les helpers existants `Positional(args, what, valuedFlags)`, `HasFlag`, `FlagValue`, `Load(path)`, `CliOperationException`. +- Produces: le verbe `audit` et son format texte, dont les Tasks 3 et 4 dépendent. + +- [ ] **Step 1: Écrire les tests qui échouent** + +Ajouter à `tests/OKF4net.Tests/CliTests.cs` (dans la classe `CliTests`) : + +```csharp + private static readonly string V02BundlePath = + Path.Combine(TestPaths.RepoRoot(), "tests", "fixtures", "okf_v02"); + + [Fact] + public void Audit_report_mode_prints_summary_and_worklist() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + + Assert.Equal(0, r.Code); + Assert.Contains("as of: 2099-06-01\n", r.Out); + Assert.Contains("concepts: 2\n", r.Out); + Assert.Contains(" 1 human-reviewed\n", r.Out); + Assert.Contains(" 1 unverified\n", r.Out); + Assert.Contains(" 2 stable\n", r.Out); + Assert.Contains("stale: 1 of 2 past stale_after\n", r.Out); + Assert.Contains("needs attention (1):\n", r.Out); + Assert.Contains(" metrics/dau stale 2099-01-01 human-reviewed stable\n", r.Out); + } + + [Fact] + public void Audit_query_mode_prints_bare_lines_only() + { + var r = Run("audit", V02BundlePath, "--stale", "--as-of", "2099-06-01"); + + Assert.Equal(0, r.Code); + Assert.Equal("metrics/dau stale 2099-01-01 human-reviewed stable\n", r.Out); + } + + [Fact] + public void Audit_without_flags_selects_the_same_set_as_stale() + { + var report = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + var query = Run("audit", V02BundlePath, "--stale", "--as-of", "2099-06-01"); + + var reportIds = report.Out + .Split('\n') + .Where(l => l.StartsWith(" metrics/", StringComparison.Ordinal)) + .Select(l => l.Trim().Split(" ")[0]) + .ToList(); + var queryIds = query.Out + .Split('\n', StringSplitOptions.RemoveEmptyEntries) + .Select(l => l.Split(" ")[0]) + .ToList(); + + Assert.Equal(queryIds, reportIds); + } + + [Fact] + public void Audit_empty_selection_prints_nothing() + { + var r = Run("audit", V02BundlePath, "--status", "deprecated"); + + Assert.Equal(0, r.Code); + Assert.Equal("", r.Out); + } + + [Fact] + public void Audit_three_tier_idiom_returns_every_concept() + { + var r = Run("audit", V02BundlePath, "--trust", "unverified,machine-confirmed,human-reviewed"); + + Assert.Equal(0, r.Code); + Assert.Equal(2, r.Out.Split('\n', StringSplitOptions.RemoveEmptyEntries).Length); + } + + [Fact] + public void Audit_rejects_an_invalid_as_of_date() + { + var r = Run("audit", V02BundlePath, "--as-of", "2026-13-01"); + + Assert.Equal(1, r.Code); + Assert.Equal("error: --as-of is not a valid YYYY-MM-DD date: \"2026-13-01\"\n", r.Err); + } + + [Fact] + public void Audit_rejects_an_unknown_trust_tier() + { + var r = Run("audit", V02BundlePath, "--trust", "foo"); + + Assert.Equal(1, r.Code); + Assert.Equal( + "error: unknown trust tier \"foo\"; expected unverified, machine-confirmed or human-reviewed\n", + r.Err); + } + + [Fact] + public void Audit_rejects_an_unknown_status() + { + var r = Run("audit", V02BundlePath, "--status", "retired"); + + Assert.Equal(1, r.Code); + Assert.Equal("error: unknown status \"retired\"; expected draft, stable or deprecated\n", r.Err); + } + + [Fact] + public void Audit_rejects_an_empty_trust_entry_but_absorbs_duplicates() + { + var empty = Run("audit", V02BundlePath, "--trust", "unverified,,human-reviewed"); + Assert.Equal(1, empty.Code); + Assert.Contains("unknown trust tier", empty.Err); + + var duplicated = Run("audit", V02BundlePath, "--trust", "unverified,unverified"); + var single = Run("audit", V02BundlePath, "--trust", "unverified"); + Assert.Equal(0, duplicated.Code); + Assert.Equal(single.Out, duplicated.Out); + } + + /// + /// Regression guard: valued flags must be declared to Positional, or + /// their value is mistaken for the bundle path when they precede it. + /// + [Theory] + [InlineData("--as-of", "2099-06-01")] + [InlineData("--trust", "unverified")] + [InlineData("--status", "stable")] + [InlineData("--type", "Metric")] + public void Audit_valued_flags_before_the_positional_resolve_the_bundle(string flag, string value) + { + var r = Run("audit", flag, value, V02BundlePath); + + Assert.Equal(0, r.Code); + Assert.Equal("", r.Err); + } + + /// + /// A valued flag left without a value must name itself, even when it is the + /// only argument -- otherwise the user is told the bundle is missing and the + /// real mistake is hidden. This is why CmdAudit validates flag values before + /// resolving the positional. + /// + [Theory] + [InlineData("--as-of")] + [InlineData("--trust")] + [InlineData("--status")] + [InlineData("--type")] + public void Audit_reports_a_valued_flag_left_without_a_value(string flag) + { + var r = Run("audit", flag); + + Assert.Equal(1, r.Code); + Assert.Equal($"error: {flag} requires a value\n", r.Err); + } + + [Fact] + public void Audit_as_of_alone_stays_in_report_mode() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + + Assert.Contains("needs attention", r.Out); + } + + [Fact] + public void Help_lists_audit_right_after_validate() + { + var r = Run("--help"); + + Assert.Equal(0, r.Code); + var lines = r.Out.Split('\n').Select(l => l.TrimStart()).ToList(); + var validateIndex = lines.FindIndex(l => l.StartsWith("validate ", StringComparison.Ordinal)); + var auditIndex = lines.FindIndex(l => l.StartsWith("audit ", StringComparison.Ordinal)); + + Assert.True(validateIndex >= 0 && auditIndex == validateIndex + 1); + } +``` + +- [ ] **Step 2: Lancer les tests pour vérifier qu'ils échouent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~CliTests"` +Expected: échecs — `unknown subcommand: audit` sur chaque test d'audit. + +- [ ] **Step 3: Implémenter le verbe** + +Dans `src/OKF4net.Cli/OkfCli.cs`, ajouter `using System.Globalization;` en tête, puis : + +a) Dans la constante `Usage`, insérer la ligne **juste après** celle de `validate` : + +```csharp + " audit Report trust, freshness and lifecycle across the bundle\n" + +``` + +et remplacer la ligne d'option `--json` par : + +```csharp + " --json Machine-readable output for validate/info/audit\n" + +``` + +Mettre aussi à jour le commentaire XML de la classe : sur cette base il annonce +« Seven subcommands » — il passe à huit, en citant `audit`. + +b) Dans le `switch` de `Run`, ajouter après `"validate"` : + +```csharp + "audit" => CmdAudit(rest, stdout), +``` + +c) Ajouter les membres privés : + +```csharp + /// The flags that make audit a filtered query rather than a report. + private static readonly string[] AuditFilterFlags = ["--stale", "--trust", "--status", "--type"]; + + /// Every audit flag that consumes the following token as its value. + private static readonly string[] AuditValuedFlags = ["--trust", "--status", "--type", "--as-of"]; + + /// An pinned to one date, backing --as-of. + private sealed class PinnedClock(DateOnly today) : IOkfClock + { + public DateOnly Today { get; } = today; + } + + /// Implements the audit subcommand. + private static int CmdAudit(string[] args, TextWriter stdout) + { + // Flag values are validated BEFORE the positional is resolved. An + // unvalued flag is the more specific diagnosis, and `okf audit --as-of` + // -- the flag as the only argument -- would otherwise report + // "missing " and hide the actual mistake, because Positional + // skips a valued flag's slot without checking that it has a value. + var clock = ParseAsOf(args); + + // Report mode selects exactly what --stale selects; only the + // presentation differs. --as-of and --json never switch modes. + var filtered = AuditFilterFlags.Any(f => HasFlag(args, f)); + var query = filtered ? ParseAuditQuery(args) : new AuditQuery(StaleOnly: true); + + var path = Positional(args, "", AuditValuedFlags); + var bundle = Load(path); + var report = ConceptAudit.Run(bundle, query, clock); + + if (HasFlag(args, "--json")) + { + JsonOutput.WriteAudit(stdout, path, query, report); + return 0; + } + + if (filtered) + { + foreach (var finding in report.Findings) + { + stdout.Write(FormatAuditFinding(finding)); + stdout.Write("\n"); + } + + return 0; + } + + WriteAuditReport(stdout, path, report); + return 0; + } + + /// Parses --as-of; null when absent (the audit then uses the system clock). + private static IOkfClock? ParseAsOf(string[] args) + { + var raw = FlagValue(args, "--as-of"); + if (raw is null) + { + return null; + } + + // DateOnly has no (s, format, provider, out) overload -- the five-argument + // form is the only one that takes a culture, and it is the same contract + // Lifecycle.From uses for stale_after. + if (!DateOnly.TryParseExact(raw, "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out var asOf)) + { + throw new CliOperationException($"--as-of is not a valid YYYY-MM-DD date: \"{raw}\""); + } + + return new PinnedClock(asOf); + } + + /// Builds the query from the filter flags. Throws on an unknown vocabulary value. + private static AuditQuery ParseAuditQuery(string[] args) + { + HashSet? tiers = null; + var trustRaw = FlagValue(args, "--trust"); + if (trustRaw is not null) + { + tiers = []; + foreach (var entry in trustRaw.Split(',')) + { + if (!AuditVocabulary.TryParseTrustTier(entry.Trim(), out var tier)) + { + throw new CliOperationException( + $"unknown trust tier \"{entry.Trim()}\"; expected unverified, machine-confirmed or human-reviewed"); + } + + tiers.Add(tier); + } + } + + ConceptStatus? status = null; + var statusRaw = FlagValue(args, "--status"); + if (statusRaw is not null) + { + if (!AuditVocabulary.TryParseStatus(statusRaw.Trim(), out var parsed)) + { + throw new CliOperationException( + $"unknown status \"{statusRaw.Trim()}\"; expected draft, stable or deprecated"); + } + + status = parsed; + } + + return new AuditQuery( + HasFlag(args, "--stale"), + tiers, + status, + FlagValue(args, "--type")); + } + + /// Renders one concept line: id, freshness, trust tier, status -- two spaces between fields. + private static string FormatAuditFinding(AuditFinding finding) + { + var freshness = finding.Lifecycle.StaleAfter is { } date + ? (finding.IsStale ? "stale " : "fresh ") + date.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) + : "no-stale-after"; + + return $"{finding.Id} {freshness} {AuditVocabulary.Name(finding.Trust)} {AuditVocabulary.Name(finding.Lifecycle.Status)}"; + } + + /// Renders the report form: summary counters over the whole bundle, then the worklist. + private static void WriteAuditReport(TextWriter stdout, string bundlePath, AuditReport report) + { + stdout.Write($"bundle: {bundlePath}\n"); + stdout.Write($"as of: {report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)}\n"); + stdout.Write($"concepts: {report.ConceptCount}\n"); + + // Labels always come from AuditVocabulary -- never as literals here. + // Duplicating them in each renderer is exactly the drift the shared + // vocabulary exists to prevent. Only the ORDER is decided locally: the + // report shows the strongest tier first, so it walks the canonical + // (weakest-first) list in reverse. + stdout.Write("\ntrust:\n"); + foreach (var tier in AuditVocabulary.TrustTiersInOrder.Reverse()) + { + stdout.Write($" {report.TrustCounts[tier],4} {AuditVocabulary.Name(tier)}\n"); + } + + stdout.Write("\nstatus:\n"); + foreach (var status in AuditVocabulary.StatusesInOrder) + { + stdout.Write($" {report.StatusCounts[status],4} {AuditVocabulary.Name(status)}\n"); + } + + stdout.Write($"\nstale: {report.StaleCount} of {report.ConceptCount} past stale_after\n"); + + if (report.Findings.Count == 0) + { + stdout.Write("\nneeds attention: none\n"); + return; + } + + stdout.Write($"\nneeds attention ({report.Findings.Count}):\n"); + foreach (var finding in report.Findings) + { + stdout.Write(" "); + stdout.Write(FormatAuditFinding(finding)); + stdout.Write("\n"); + } + } +``` + +**Note pour la Task 3 :** `JsonOutput.WriteAudit` n'existe pas encore. Pour que la Task 2 compile seule, remplacer temporairement la branche `--json` par `throw new CliOperationException("--json is not implemented yet");` **et** ne pas écrire de test `--json` ici — la Task 3 rétablit l'appel réel. Si les deux tasks sont exécutées d'affilée, écrire directement l'appel final et implémenter la Task 3 avant de lancer les tests. + +- [ ] **Step 4: Lancer les tests pour vérifier qu'ils passent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~CliTests"` +Expected: PASS. Puis `dotnet test OKF4net.sln` en entier : aucun golden existant ne doit bouger. + +- [ ] **Step 5: Commit** + +```bash +git add src/OKF4net.Cli/OkfCli.cs tests/OKF4net.Tests/CliTests.cs +git commit -m "feat(cli): add the okf audit verb with its report and query modes" +``` + +--- + +### Task 3: La sortie `okf audit --json` + +**Files:** +- Modify: `src/OKF4net.Cli/JsonOutput.cs` +- Modify: `src/OKF4net.Cli/OkfCli.cs` (rétablir l'appel réel si la Task 2 a posé le stub) +- Test: `tests/OKF4net.Tests/CliTests.cs` + +**Interfaces:** +- Consumes: Task 1 (`AuditReport`, `AuditQuery`, `AuditVocabulary`), Task 2 (`CmdAudit`). +- Produces: `JsonOutput.WriteAudit(TextWriter stdout, string bundlePath, AuditQuery query, AuditReport report)` et le schéma JSON que la Task 4 fige en golden. + +- [ ] **Step 1: Écrire le test qui échoue** + +Ajouter à `tests/OKF4net.Tests/CliTests.cs` (ajouter `using System.Text.Json;` en tête du fichier si absent) : + +```csharp + [Fact] + public void Audit_json_carries_counts_query_and_findings() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01", "--json"); + + Assert.Equal(0, r.Code); + Assert.EndsWith("\n", r.Out); + + using var doc = JsonDocument.Parse(r.Out); + var root = doc.RootElement; + + Assert.Equal("2099-06-01", root.GetProperty("asOf").GetString()); + Assert.Equal(2, root.GetProperty("conceptCount").GetInt32()); + Assert.Equal(1, root.GetProperty("staleCount").GetInt32()); + + // Report mode selects what --stale selects, so the replayed query says so. + Assert.True(root.GetProperty("query").GetProperty("stale").GetBoolean()); + Assert.Equal(JsonValueKind.Null, root.GetProperty("query").GetProperty("trust").ValueKind); + + Assert.Equal(1, root.GetProperty("trust").GetProperty("humanReviewed").GetInt32()); + Assert.Equal(1, root.GetProperty("trust").GetProperty("unverified").GetInt32()); + Assert.Equal(2, root.GetProperty("status").GetProperty("stable").GetInt32()); + + var finding = root.GetProperty("findings").EnumerateArray().Single(); + Assert.Equal("metrics/dau", finding.GetProperty("conceptId").GetString()); + Assert.Equal("Metric", finding.GetProperty("type").GetString()); + Assert.Equal("Daily Active Users", finding.GetProperty("title").GetString()); + Assert.Equal("human-reviewed", finding.GetProperty("trust").GetString()); + Assert.Equal("2099-01-01", finding.GetProperty("staleAfter").GetString()); + Assert.True(finding.GetProperty("stale").GetBoolean()); + } + + [Fact] + public void Audit_json_serializes_trust_query_in_ladder_order() + { + var r = Run("audit", V02BundlePath, "--trust", "human-reviewed,unverified", "--json"); + + using var doc = JsonDocument.Parse(r.Out); + var trust = doc.RootElement.GetProperty("query").GetProperty("trust") + .EnumerateArray().Select(e => e.GetString()).ToList(); + + Assert.Equal(["unverified", "human-reviewed"], trust); + } + + [Fact] + public void Audit_json_keeps_a_malformed_stale_after_raw_and_not_stale() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstale_after: not-a-date\n---\n"); + + var r = Run("audit", tmp.Path, "--trust", "unverified", "--json"); + + using var doc = JsonDocument.Parse(r.Out); + var finding = doc.RootElement.GetProperty("findings").EnumerateArray().Single(); + + Assert.Equal("not-a-date", finding.GetProperty("staleAfter").GetString()); + Assert.False(finding.GetProperty("stale").GetBoolean()); + } +``` + +- [ ] **Step 2: Lancer les tests pour vérifier qu'ils échouent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~CliTests.Audit_json"` +Expected: échec (stub `--json is not implemented yet`, ou erreur de compilation sur `WriteAudit`). + +- [ ] **Step 3: Implémenter la sérialisation** + +Dans `src/OKF4net.Cli/JsonOutput.cs`, ajouter `using System.Globalization;` en tête, puis les records après `InfoJsonResult` : + +```csharp +/// The query okf audit applied, replayed for --json consumers. +internal sealed record AuditQueryJson(bool Stale, IReadOnlyList? Trust, string? Status, string? Type); + +/// Concept counts per trust tier (§5.3), over the whole bundle. +internal sealed record TrustCountsJson(int HumanReviewed, int MachineConfirmed, int Unverified); + +/// Concept counts per lifecycle status (§5.4), over the whole bundle. +internal sealed record StatusCountsJson(int Draft, int Stable, int Deprecated); + +/// One selected concept, projected for --json output. +internal sealed record AuditFindingJson( + string ConceptId, + string Path, + string? Type, + string? Title, + string Trust, + string Status, + string? StaleAfter, + bool Stale); + +/// The full result of okf audit --json. +internal sealed record AuditJsonResult( + string Bundle, + string AsOf, + int ConceptCount, + AuditQueryJson Query, + TrustCountsJson Trust, + StatusCountsJson Status, + int StaleCount, + IReadOnlyList Findings); +``` + +Ajouter l'attribut sur `CliJsonContext`, à côté des deux existants : + +```csharp +[JsonSerializable(typeof(AuditJsonResult))] +``` + +Puis la méthode, dans `JsonOutput` : + +```csharp + /// Writes okf audit --json's result to as a single line-terminated JSON document. + internal static void WriteAudit(TextWriter stdout, string bundlePath, AuditQuery query, AuditReport report) + { + // Serialized in ladder order, never in the order the user typed them: + // IReadOnlySet has no guaranteed order, and the document must be + // reproducible. Written as a statement rather than a ternary so the + // nullable analysis narrows `Trust` through the pattern -- it does not + // narrow a property across the arms of a conditional. + List? trustQuery = null; + if (query.Trust is { } selectedTiers) + { + trustQuery = AuditVocabulary.TrustTiersInOrder + .Where(selectedTiers.Contains) + .Select(AuditVocabulary.Name) + .ToList(); + } + + var findings = report.Findings + .Select(f => new AuditFindingJson( + f.Id.ToString(), + f.Path, + f.Type, + f.Title, + AuditVocabulary.Name(f.Trust), + AuditVocabulary.Name(f.Lifecycle.Status), + f.Lifecycle.StaleAfterRaw, + f.IsStale)) + .ToList(); + + var result = new AuditJsonResult( + bundlePath, + report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture), + report.ConceptCount, + new AuditQueryJson( + query.StaleOnly, + trustQuery, + query.Status is { } status ? AuditVocabulary.Name(status) : null, + query.Type), + new TrustCountsJson( + report.TrustCounts[TrustTier.HumanReviewed], + report.TrustCounts[TrustTier.MachineConfirmed], + report.TrustCounts[TrustTier.Unverified]), + new StatusCountsJson( + report.StatusCounts[ConceptStatus.Draft], + report.StatusCounts[ConceptStatus.Stable], + report.StatusCounts[ConceptStatus.Deprecated]), + report.StaleCount, + findings); + + stdout.Write(JsonSerializer.Serialize(result, CliJsonContext.Default.AuditJsonResult)); + stdout.Write("\n"); + } +``` + +Enfin, dans `OkfCli.CmdAudit`, remplacer le stub éventuel par l'appel réel : + +```csharp + if (HasFlag(args, "--json")) + { + JsonOutput.WriteAudit(stdout, path, query, report); + return 0; + } +``` + +- [ ] **Step 4: Lancer les tests pour vérifier qu'ils passent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~CliTests"` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add src/OKF4net.Cli/JsonOutput.cs src/OKF4net.Cli/OkfCli.cs tests/OKF4net.Tests/CliTests.cs +git commit -m "feat(cli): add machine-readable output to okf audit" +``` + +--- + +### Task 4: Les goldens byte-exact + +**Files:** +- Create: `tests/fixtures/golden/audit-v02.out` +- Create: `tests/fixtures/golden/audit-v02.json` +- Modify: `tests/OKF4net.Tests/GoldenParityTests.cs` +- Modify: `tests/fixtures/README.md` + +**Interfaces:** +- Consumes: Tasks 2 et 3 (les deux formats de sortie). +- Produces: rien pour les tasks suivantes — c'est un verrou de non-régression. + +**Rappel de règle :** aucun bundle de fixtures n'est créé ni modifié. `tests/fixtures/okf_v02` est réutilisé tel quel ; `--as-of 2099-06-01` suffit à rendre `metrics/dau` (`stale_after: 2099-01-01`) périmé. Ces deux goldens sont **écrits à la main** et vérifiés contre le texte du spec, **pas** capturés depuis un binaire de référence : `audit` n'existe pas en amont. + +- [ ] **Step 1: Écrire les deux goldens à la main** + +`tests/fixtures/golden/audit-v02.out` — **fins de ligne LF, pas de CRLF** (le `.gitattributes` marque `tests/fixtures/` en `-text`) : + +``` +bundle: tests/fixtures/okf_v02 +as of: 2099-06-01 +concepts: 2 + +trust: + 1 human-reviewed + 0 machine-confirmed + 1 unverified + +status: + 0 draft + 2 stable + 0 deprecated + +stale: 1 of 2 past stale_after + +needs attention (1): + metrics/dau stale 2099-01-01 human-reviewed stable +``` + +`tests/fixtures/golden/audit-v02.json` — une seule ligne, terminée par `\n` : + +``` +{"bundle":"tests/fixtures/okf_v02","asOf":"2099-06-01","conceptCount":2,"query":{"stale":true,"trust":null,"status":null,"type":null},"trust":{"humanReviewed":1,"machineConfirmed":0,"unverified":1},"status":{"draft":0,"stable":2,"deprecated":0},"staleCount":1,"findings":[{"conceptId":"metrics/dau","path":"tests/fixtures/okf_v02/metrics/dau.md","type":"Metric","title":"Daily Active Users","trust":"human-reviewed","status":"stable","staleAfter":"2099-01-01","stale":true}]} +``` + +- [ ] **Step 2: Écrire les tests de parité** + +Ajouter à `tests/OKF4net.Tests/GoldenParityTests.cs` : + +```csharp + /// + /// audit's goldens are hand-authored and verified against the spec + /// text (§5.3 tiers, §5.4 statuses, §5.5 staleness), not captured from the + /// reference CLI -- the verb has no upstream counterpart. The date is + /// pinned with --as-of so the output cannot drift with the calendar. + /// There is no audit-v02.exitcode: the verb always exits 0, so the + /// code is asserted inline (as does). + /// + [Fact] + public void Audit_report_matches_golden() + { + var r = WithRepoRootAsCwd(() => Run("audit", "tests/fixtures/okf_v02", "--as-of", "2099-06-01")); + + Assert.Equal(0, r.Code); + + // Every path in this output is a concept id, always '/'-normalized by + // ConceptId.FromPath, so the comparison is strict byte-for-byte. + Assert.Equal(Golden("audit-v02.out"), r.Out); + } + + [Fact] + public void Audit_json_matches_golden() + { + var r = WithRepoRootAsCwd(() => Run("audit", "tests/fixtures/okf_v02", "--as-of", "2099-06-01", "--json")); + + Assert.Equal(0, r.Code); + + // Only the findings' `path` field carries a native separator (it is a + // real file path, not a concept id), so it alone is normalized in the + // C# OUTPUT -- never in the golden -- exactly like validate.out. + Assert.Equal(Golden("audit-v02.json"), r.Out.Replace('\\', '/')); + } +``` + +- [ ] **Step 3: Lancer les tests de parité** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~GoldenParityTests"` +Expected: PASS. **Si un écart apparaît, corriger le code du CLI, jamais le golden** — sauf si l'écart révèle une erreur dans ce plan, auquel cas corriger le golden ET le noter dans le message de commit. + +- [ ] **Step 4: Documenter la provenance** + +Dans `tests/fixtures/README.md`, ajouter à la liste des goldens : + +```markdown +- `golden/audit-v02.out`, `golden/audit-v02.json` — output of + `okf audit tests/fixtures/okf_v02 --as-of 2099-06-01` (and its `--json` + form). **Hand-authored**, verified against the spec text (§5.3 trust tiers, + §5.4 statuses, §5.5 staleness) rather than captured from the reference CLI: + `audit` is an OKF4net verb with no upstream counterpart. The `--as-of` date + is pinned so the output cannot drift with the calendar. +``` + +- [ ] **Step 5: Commit** + +```bash +git add tests/fixtures/golden/audit-v02.out tests/fixtures/golden/audit-v02.json \ + tests/OKF4net.Tests/GoldenParityTests.cs tests/fixtures/README.md +git commit -m "test(audit): pin okf audit's text and JSON output with goldens" +``` + +--- + +### Task 5: Le tool agent `okf_audit` + +**Files:** +- Modify: `src/OKF4net.Agents/OkfBundleTools.cs` +- Test: `tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs` (créé) + +**Interfaces:** +- Consumes: Task 1 (`ConceptAudit.Run`, `AuditQuery`, `AuditReport`, `AuditVocabulary`), et les membres existants `GetBundle()`, `Today`, `UtcNow`, `GetTools()`, `WriteToolNames`. +- Produces: le tool `okf_audit` (lecture seule). + +- [ ] **Step 1: Écrire les tests qui échouent** + +Créer `tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs` : + +```csharp +// SPDX-License-Identifier: LGPL-3.0-or-later +using Microsoft.Extensions.AI; +using OKF4net.Agents; + +namespace OKF4net.Tests.Agents; + +/// +/// Tests for the okf_audit tool. Every test pins UtcNow: the tool +/// deliberately exposes no asOf parameter, so the shared clock seam is +/// the only way its output can be made deterministic. +/// +public class OkfAuditToolTests +{ + private static OkfBundleTools ToolsOver(TempDir tmp, DateOnly today) + => new(tmp.Path) + { + UtcNow = () => new DateTime(today.Year, today.Month, today.Day, 0, 0, 0, DateTimeKind.Utc), + }; + + [Fact] + public void Audit_is_registered_and_read_only() + { + var tools = new OkfBundleTools(Path.Combine(TestPaths.RepoRoot(), "tests", "fixtures", "okf_v02")); + + Assert.Contains("okf_audit", tools.GetTools().OfType().Select(t => t.Name)); + Assert.DoesNotContain("okf_audit", OkfBundleTools.WriteToolNames); + } + + /// + /// §5.5's boundary is today >= stale_after, so a concept whose + /// stale_after is exactly today is stale. Without the pinned seam this + /// assertion would silently depend on the day the suite runs. + /// + [Fact] + public void Audit_treats_today_equals_stale_after_as_stale() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstale_after: 2026-08-21\n---\n"); + + var onTheDay = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + var theDayBefore = ToolsOver(tmp, new DateOnly(2026, 8, 20)).Audit(); + + Assert.Contains("a stale 2026-08-21", onTheDay); + Assert.Contains("needs attention: none", theDayBefore); + } + + [Fact] + public void Audit_reports_counts_and_omits_the_bundle_line() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + + Assert.DoesNotContain("bundle:", text); + Assert.Contains("as of: 2026-08-21", text); + Assert.Contains(" 1 human-reviewed", text); + Assert.Contains(" 1 unverified", text); + } + + [Fact] + public void Audit_caps_the_listing_at_twenty_findings() + { + using var tmp = new TempDir(); + for (var i = 0; i < 25; i++) + { + tmp.Write($"c{i:D2}.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + } + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + + Assert.Contains("… and 5 more (narrow with stale/trust/status/type)", text); + + // Finding lines are the two-space-indented ones; matching on "c" alone + // would also catch the "concepts:" header. + Assert.Equal(20, text.Split('\n').Count(l => l.StartsWith(" c", StringComparison.Ordinal))); + } + + [Fact] + public void Audit_renders_a_usage_message_for_invalid_vocabulary() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + + var tools = ToolsOver(tmp, new DateOnly(2026, 8, 21)); + + Assert.Contains("Usage: okf_audit", tools.Audit(trust: "machine")); + Assert.Contains("Usage: okf_audit", tools.Audit(status: "retired")); + } + + /// + /// A function tool returns errors, it does not throw them: a bundle that + /// disappears after the tool was constructed must surface as an "Error: ..." + /// string, which is what the shared RunTool guard provides. + /// + [Fact] + public void Audit_returns_an_error_string_when_the_bundle_cannot_be_loaded() + { + var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + var tools = ToolsOver(tmp, new DateOnly(2026, 8, 21)); + tmp.Dispose(); // the directory is gone before the first load + + var text = tools.Audit(); + + Assert.StartsWith("Error: ", text); + } + + [Fact] + public void Audit_with_stale_false_and_no_filter_returns_the_whole_corpus() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false); + + Assert.Contains("needs attention (2):", text); + } +} +``` + +- [ ] **Step 2: Lancer les tests pour vérifier qu'ils échouent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~OkfAuditToolTests"` +Expected: échec de compilation — `Audit` n'existe pas sur `OkfBundleTools`. + +- [ ] **Step 3: Implémenter le tool** + +Dans `src/OKF4net.Agents/OkfBundleTools.cs` : + +a) Ajouter la constante d'usage à côté de `SearchUsageMessage` : + +```csharp + private const string AuditUsageMessage = + "Usage: okf_audit takes optional filters — stale (bool), trust (comma-separated: " + + "unverified, machine-confirmed, human-reviewed), status (draft, stable or deprecated) " + + "and type (exact frontmatter type). Example: okf_audit(stale: true, trust: \"unverified\")."; +``` + +b) Ajouter l'adaptateur d'horloge, en membre privé de la classe : + +```csharp + /// + /// Pins to — the same + /// UtcNow seam and use — so + /// the tool's output never depends on the day it runs. + /// + private sealed class PinnedClock(DateOnly today) : IOkfClock + { + public DateOnly Today { get; } = today; + } +``` + +c) Ajouter la méthode du tool : + +```csharp + /// + /// Audits the bundle's trust, freshness and lifecycle signals (§5.3–§5.5): + /// counts over the whole bundle, then the concepts the filters select, + /// bounded to 20 entries. + /// + /// Keep only concepts past their stale_after date. + /// Comma-separated trust tiers to keep. + /// Keep only concepts with this lifecycle status. + /// Keep only concepts with this frontmatter type (exact match). + [Description("Audit the bundle's trust, freshness and lifecycle signals: counts by trust tier and status, plus the concepts needing attention. Filter with stale/trust/status/type.")] + public string Audit( + [Description("Only concepts past their stale_after date. Defaults to true.")] bool stale = true, + [Description("Comma-separated trust tiers to include: unverified, machine-confirmed, human-reviewed.")] string? trust = null, + [Description("Only concepts with this lifecycle status: draft, stable or deprecated.")] string? status = null, + [Description("Only concepts with this frontmatter type (exact match).")] string? type = null) + { + HashSet? tiers = null; + if (trust is not null) + { + tiers = []; + foreach (var entry in trust.Split(',')) + { + if (!AuditVocabulary.TryParseTrustTier(entry.Trim(), out var tier)) + { + return AuditUsageMessage; + } + + tiers.Add(tier); + } + } + + ConceptStatus? parsedStatus = null; + if (status is not null) + { + if (!AuditVocabulary.TryParseStatus(status.Trim(), out var value)) + { + return AuditUsageMessage; + } + + parsedStatus = value; + } + + // Everything that can touch the filesystem goes through RunTool, the + // guard every bundle-loading tool uses: it turns OkfException (hence + // BundleLoadException), ArgumentException, IOException, + // UnauthorizedAccessException and DecoderFallbackException into an + // "Error: ..." string. A function tool must return an error, not throw + // one -- a directory deleted after construction would otherwise escape + // as an exception into the agent runtime. + return RunTool(() => + { + var report = ConceptAudit.Run( + GetBundle(), + new AuditQuery(stale, tiers, parsedStatus, type), + new PinnedClock(Today)); + + return RenderAudit(report); + }); + } + + /// + /// Renders an audit for an agent: the same shape as the CLI's report form, + /// minus the bundle line (the tool is bound to one bundle) and bounded to + /// 20 findings. Deliberately not shared with the CLI renderer, whose bytes + /// are golden-locked and must not move when this string is tuned. + /// + private static string RenderAudit(AuditReport report) + { + const int MaxResults = 20; + var sb = new StringBuilder(); + + sb.Append("as of: ").Append(report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)).Append('\n'); + sb.Append("concepts: ").Append(report.ConceptCount).Append('\n'); + + // Same rule as the CLI renderer: labels from AuditVocabulary, never + // literals. The two renderers are separate on purpose (the CLI's bytes + // are golden-locked), but they must not spell the vocabulary twice. + sb.Append("\ntrust:\n"); + foreach (var tier in AuditVocabulary.TrustTiersInOrder.Reverse()) + { + sb.Append($" {report.TrustCounts[tier],4} {AuditVocabulary.Name(tier)}\n"); + } + + sb.Append("\nstatus:\n"); + foreach (var status in AuditVocabulary.StatusesInOrder) + { + sb.Append($" {report.StatusCounts[status],4} {AuditVocabulary.Name(status)}\n"); + } + + sb.Append($"\nstale: {report.StaleCount} of {report.ConceptCount} past stale_after\n"); + + if (report.Findings.Count == 0) + { + sb.Append("\nneeds attention: none\n"); + return sb.ToString(); + } + + sb.Append($"\nneeds attention ({report.Findings.Count}):\n"); + foreach (var finding in report.Findings.Take(MaxResults)) + { + var freshness = finding.Lifecycle.StaleAfter is { } date + ? (finding.IsStale ? "stale " : "fresh ") + date.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) + : "no-stale-after"; + + sb.Append(" ") + .Append(finding.Id) + .Append(" ").Append(freshness) + .Append(" ").Append(AuditVocabulary.Name(finding.Trust)) + .Append(" ").Append(AuditVocabulary.Name(finding.Lifecycle.Status)) + .Append('\n'); + } + + if (report.Findings.Count > MaxResults) + { + sb.Append($"… and {report.Findings.Count - MaxResults} more (narrow with stale/trust/status/type)\n"); + } + + return sb.ToString(); + } +``` + +d) Enregistrer le tool dans `GetTools()`, après `okf_search` : + +```csharp + AIFunctionFactory.Create(Audit, "okf_audit"), +``` + +e) Aucun `using` à ajouter : `System.ComponentModel`, `System.Globalization` et `System.Text` sont déjà en tête de `OkfBundleTools.cs`, et `ImplicitUsings` couvre `System.Linq`. + +- [ ] **Step 4: Lancer les tests pour vérifier qu'ils passent** + +Run: `dotnet test OKF4net.sln --filter "FullyQualifiedName~OKF4net.Tests.Agents"` +Expected: PASS, y compris `WriteToolNames_matches_the_three_mutating_tools_and_filters_them_out` (le nouveau tool étant en lecture seule, il doit apparaître dans le sous-ensemble read-only). Puis `dotnet test OKF4net.sln` en entier. + +- [ ] **Step 5: Commit** + +```bash +git add src/OKF4net.Agents/OkfBundleTools.cs tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs +git commit -m "feat(agents): expose okf_audit as a read-only tool" +``` + +--- + +### Task 6: Documentation + +**Files:** +- Modify: `README.md` +- Modify: `CHANGELOG.md` + +**Interfaces:** +- Consumes: tout ce qui précède. +- Produces: rien. + +- [ ] **Step 1: Mettre à jour le README** + +Dans la liste des verbes CLI, ajouter `audit` juste après `validate`, avec un exemple qui montre la question de départ : + +````markdown +`okf audit ` reports trust, freshness and lifecycle across a whole +bundle: counts per trust tier (§5.3) and status (§5.4), plus the worklist of +stale concepts (§5.5). Filter it to ask corpus-level questions — + +```sh +# Which concepts are past stale_after and were never verified by a human? +okf audit bundles/acme_retail --stale --trust unverified,machine-confirmed +``` + +Without filter flags it selects exactly what `--stale` selects and prints the +summary form; with any filter flag it prints one line per matching concept, so +the output pipes. `--json` always emits the full document. Note the counts +always cover the whole bundle while `findings` covers the selection: `audit` is +a worklist, not an inventory (use `okf info --json` for that). +```` + +Dans la table « spec section → type », ajouter la ligne : + +```markdown +| §5.3–§5.5 | `ConceptAudit`, `AuditQuery`, `AuditReport` | Corpus-level trust/freshness query behind `okf audit` and `okf_audit` | +``` + +- [ ] **Step 2: Mettre à jour le CHANGELOG** + +Sous `## [Unreleased]`, section `### Added` (la créer si absente) : + +```markdown +- `okf audit` — a corpus-level query over a bundle's trust (§5.3), lifecycle + (§5.4) and staleness (§5.5) signals: counts plus a filterable worklist, with + `--stale`, `--trust`, `--status`, `--type`, `--as-of` and `--json`. Backed by + the new `ConceptAudit` in the core library and exposed to agents as the + read-only `okf_audit` tool. +``` + +- [ ] **Step 3: Formater et lancer la suite complète** + +```bash +dotnet format OKF4net.sln +dotnet build OKF4net.sln +dotnet test OKF4net.sln +``` + +Expected: build sans warning (ils sont des erreurs), 987 tests de base + les nouveaux, 0 échec, et `dotnet format --verify-no-changes` propre. + +- [ ] **Step 4: Commit** + +```bash +git add README.md CHANGELOG.md +git commit -m "docs(audit): document the okf audit verb and okf_audit tool" +``` + +--- + +## Reste à faire après le merge de la branche viewer + +`ROADMAP.md` et `CLAUDE.md` sont modifiés en parallèle par une autre session ; les toucher ici garantit un conflit. Une fois cette branche mergée : + +- `ROADMAP.md` : consigner l'échelle de l'article (dossier → index → recherche → graphe → bibliothèque fédérée) et marquer `okf audit` comme premier barreau livré. +- `CLAUDE.md` : une ligne sur `ConceptAudit` comme surface de requête unique partagée CLI/Agents, au même titre que la note « ne pas forker `ConceptSearch` ». diff --git a/docs/superpowers/specs/2026-08-21-okf-audit-design.md b/docs/superpowers/specs/2026-08-21-okf-audit-design.md new file mode 100644 index 0000000..e0d7981 --- /dev/null +++ b/docs/superpowers/specs/2026-08-21-okf-audit-design.md @@ -0,0 +1,642 @@ +# Design — `okf audit` : requête corpus-level sur les signaux de confiance et de fraîcheur + +Date : 2026-08-21 +Statut : validé, prêt pour le plan d'implémentation + +## 1. Objectif + +Répondre, sur un bundle entier et en une commande, à la question que l'article +[« OKF v0.2 Quietly Admits the Folder Has a Ceiling »](https://medium.com/@davidroliver/okf-v0-2-quietly-admits-the-folder-has-a-ceiling-the-way-up-is-a-library-25fa54e872f9) +(David R Oliver, 2026-08-01) met au centre : + +> *which of my concepts are past their `stale_after` date and have never been verified by a human?* + +L'argument : v0.2 a livré le **schéma** d'une base de connaissances (provenance, +trust, lifecycle) en déclarant l'infrastructure de requête hors périmètre. Les +champs `generated`/`verified`/`status`/`stale_after` ne rapportent rien tant +qu'on ne peut pas les interroger **à l'échelle du corpus** ; les ouvrir un par un +marche à 40 concepts, coûte cher à 4 000, et devient une requête de base de +données déguisée en YAML à 40 000. + +OKF4net a déjà tout le socle sémantique — `Trust.DeriveTier` (§5.3), +`Lifecycle`/`IsStale` (§5.4/§5.5), `Frontmatter.TrustTier` — mais **aucune +surface de requête** : ni les 7 verbes CLI ni les 11 tools agents ne posent de +question corpus-level. `okf audit` est cette surface. + +### 1.1 Pourquoi pas `okf validate` + +`BundleValidator` émet déjà un `ConceptStale` (warning) par concept périmé +([Validate.cs:404](../../../src/OKF4net/Validate.cs#L404)). La séparation est +délibérée et doit le rester : + +| | `validate` | `audit` | +|---|---|---| +| Question | ce bundle est-il conforme §11 ? | quels concepts demandent une action ? | +| Sortie | diagnostics par fichier, sévérités | compteurs corpus + worklist filtrable | +| Filtrable | non | oui (`--stale`, `--trust`, `--status`, `--type`) | +| Code retour | 1 si non conforme | **toujours 0** (un audit rapporte, il ne juge pas) | +| Sorties figées | goldens byte-exact v0.1 | nouveaux goldens (§7.3) | + +Corollaire dur : **aucune sortie existante de `validate`/`info` ne change**. Les +goldens v0.1 restent intacts, ce qui exclut d'emblée l'alternative « ajouter des +diagnostics `Info` au validateur » (§10, alternative B). + +## 2. Périmètre + +**Dans le périmètre** — trois unités, une par couche : + +1. `ConceptAudit` : le calcul, dans le cœur, partagé. +2. Le verbe CLI `okf audit`, texte + `--json`. +3. Le tool agent/MCP `okf_audit`, lecture seule. + +**Hors périmètre** (au backlog, mémorisé, non traité ici) : audit fédéré +multi-bundles via `OKF4net.Catalog` ; liens typés `links:` et export +property-graph ; `--fail-on` pour gater la CI ; recherche par passages/BM25 ; +tout champ frontmatter hors v0.2. + +## 3. Unité 1 — `ConceptAudit` (cœur) + +Nouveau fichier `src/OKF4net/Audit.cs`, zéro dépendance, calqué sur le doublet +existant `BundleValidator.Validate(bundle, clock) → ValidationReport`. + +### 3.1 API publique + +```csharp +/// Les prédicats de sélection d'un audit (§5.3–§5.5). Combinés en ET ; `default` ne filtre rien. +/// +/// L'égalité générée compare Trust par référence (c'est le comportement de +/// EqualityComparer<IReadOnlySet<T>>.Default) : deux requêtes logiquement +/// identiques peuvent être inégales. Ne pas s'appuyer dessus, ni utiliser une +/// AuditQuery comme clé de dictionnaire. Le type reste un record struct pour +/// `with` et `ToString`, pas pour son égalité. +/// +public readonly record struct AuditQuery( + bool StaleOnly = false, + IReadOnlySet? Trust = null, + ConceptStatus? Status = null, + string? Type = null) +{ + /// La requête qui retient tous les concepts. + public static AuditQuery All => default; + + /// Vrai dès qu'au moins un prédicat est posé. + public bool IsFiltered => StaleOnly || Trust is not null || Status is not null || Type is not null; +} + +/// Un concept retenu par un audit, avec ses signaux déjà dérivés. +public readonly record struct AuditFinding( + ConceptId Id, + string Path, + string? Type, + string? Title, + TrustTier Trust, + Lifecycle Lifecycle, + bool IsStale); + +/// Le résultat d'un audit : compteurs sur tout le bundle + les concepts sélectionnés. +public sealed class AuditReport +{ + public DateOnly AsOf { get; } + public int ConceptCount { get; } + public IReadOnlyDictionary TrustCounts { get; } // les 3 clés toujours présentes + public IReadOnlyDictionary StatusCounts { get; } // les 3 clés toujours présentes + public int StaleCount { get; } + public IReadOnlyList Findings { get; } // trié par Id, ordinal +} + +/// Interroge un bundle sur ses signaux §5.3–§5.5. Ne lit rien sur disque, n'écrit rien, ne lève rien. +public static class ConceptAudit +{ + public static AuditReport Run(Bundle bundle, AuditQuery query = default, IOkfClock? clock = null); +} +``` + +### 3.2 Sémantique, point par point + +- **Horloge.** `clock ?? new SystemClock()`, `AsOf = clock.Today`, exactement + comme `BundleValidator.Validate`. Aucun `DateTime.UtcNow` enfoui : c'est ce qui + rend les tests et les goldens déterministes (`FixedClock` existe déjà côté + tests). +- **Périmètre des compteurs.** `TrustCounts`, `StatusCounts`, `StaleCount` et + `ConceptCount` portent **toujours sur le bundle entier**, jamais sur la + sélection. Le dénominateur reste stable quand on filtre ; `Findings` seul + bouge. Les trois clés de chaque dictionnaire sont toujours présentes (valeur 0 + incluse) pour que la sortie ait une forme fixe. +- **Périmètre des concepts.** `bundle.Concepts` uniquement. Les `index.md` et + `log.md` (§8/§9) sont exposés séparément par `Bundle` et ne sont pas des + concepts. Les documents **non parsables** — frontmatter invalide + (`DocumentParseException`) ou clés requises manquantes (`ConceptIdException`) — + sont collectés dans `bundle.ParseErrors` par le chargement permissif (§11) et + n'entrent dans aucun compteur. À ne pas confondre avec un fichier réellement + **illisible** (I/O, droits, contenu non-UTF-8) : `Bundle.Load` lève alors + `BundleLoadException` et abandonne le chargement entier ; le CLI rend cela en + `error:` + code 1 (§4.3), et l'audit ne voit jamais ce cas. +- **Tier de confiance.** `concept.Document.Frontmatter.TrustTier`, donc + `Trust.DeriveTier` (§5.3) : un `human:` ⇒ `HumanReviewed`, sinon tout + vérificateur ⇒ `MachineConfirmed`, liste vide ⇒ `Unverified`. +- **Staleness.** `Lifecycle.IsStale(AsOf)`, donc §5.5 : `AsOf >= stale_after`. + Une borne exacte (`AsOf == stale_after`) est **périmée**. +- **`stale_after` malformé** ⇒ `IsStale` faux, jamais dans la worklist. C'est + `validate` qui possède le diagnostic `StaleAfterInvalid` ; l'audit ne le + redouble pas. +- **Statut inconnu** (`status: retired` dans les fixtures) ⇒ compté comme + `Stable`, conformément à §5.4 (« absent ou inconnu ⇒ stable »). `Lifecycle` + transporte `StatusIsKnown`, mais l'audit ne l'expose pas : signaler la valeur + inconnue est le travail de `validate`. +- **`Type`** : comparaison **ordinale exacte** sur `Frontmatter.Type`. Pas de + repli de casse — un concept sans `type` n'est jamais retenu par `--type`. La + question « `BigQuery Table` et `bigquery-table` sont-ils le même label ? » + appartient au futur lint de vocabulaire (backlog), pas ici. +- **Tri.** `Findings` trié par `ConceptId` ordinal croissant. Déterministe, et + indépendant de l'ordre de parcours du système de fichiers. +- **Robustesse.** Aucune exception : errors-as-data comme le reste du cœur. Un + bundle vide donne des compteurs à zéro et `Findings` vide. +- **Pourquoi pas `StalePolicy`.** Le cœur a déjà `StalePolicy` + (`Use`/`Tolerate(grace)`/`Strict`), que les Agents et le Catalog appliquent à + la *restitution*. Elle répond à « dois-je exposer ce concept à un + consommateur ? ». `AuditQuery.StaleOnly` répond à « ce concept est-il sur ma + liste de travail ? » — la question inverse, où un concept périmé est + précisément ce qu'on veut voir, pas ce qu'on veut filtrer. Les deux mécanismes + coexistent donc volontairement ; ne pas les fusionner. + +## 4. Unité 2 — verbe CLI `okf audit` + +Dans `src/OKF4net.Cli/OkfCli.cs` : `CmdAudit(string[] args, TextWriter stdout)`, +branché dans le `switch` de `Run`, plus `JsonOutput.WriteAudit`. + +### 4.1 Grammaire + +``` +okf audit [--stale] [--trust ] [--status ] [--type ] + [--as-of ] [--json] +``` + +| Flag | Valeur | Effet | +|---|---|---| +| `--stale` | — | ne retient que les concepts périmés à la date d'observation | +| `--trust` | liste séparée par `,` parmi `unverified`, `machine-confirmed`, `human-reviewed` | ne retient que ces tiers | +| `--status` | `draft` \| `stable` \| `deprecated` | ne retient que ce statut | +| `--type` | chaîne | ne retient que ce `type` (exact, ordinal) | +| `--as-of` | `YYYY-MM-DD` | fixe la date d'observation (défaut : aujourd'hui, UTC) | +| `--json` | — | document JSON unique, ligne terminée | + +Les prédicats se combinent en **ET**. La question de l'article s'écrit : + +```sh +okf audit bundles/acme_retail --stale --trust unverified,machine-confirmed +``` + +**Un seul vocabulaire** partout (entrée CLI, texte, JSON, tool agent) : +`unverified` / `machine-confirmed` / `human-reviewed` et +`draft` / `stable` / `deprecated`. Pas d'alias, pas de raccourci `--unverified` — +un booléen mentirait sur le cas `machine-confirmed`, qui est aussi « jamais +relu par un humain ». + +Règles de parsing des valeurs, pour lever toute ambiguïté : + +- `--trust` est le **seul flag à valeurs multiples** : liste séparée par `,`, + chaque entrée trimée puis validée contre le vocabulaire, doublons absorbés + (`IReadOnlySet`), entrée vide (`--trust "a,,b"` ou `--trust ""`) rejetée avec + le message « unknown trust tier » et la valeur fautive citée. +- `--status` prend **une seule valeur**, trimée puis validée contre le + vocabulaire §5.4. Pas de liste : `AuditQuery.Status`, le champ JSON `status` et + le paramètre du tool agent sont tous des scalaires (`ConceptStatus?` / + `string?`). Un besoin multi-statuts se traiterait par un changement de type + cohérent sur les trois surfaces, pas par une tolérance du parseur. +- `--type` est une **valeur libre** : prise verbatim, sans trim ni repli de + casse, puisque le spec ne contraint pas le vocabulaire de `type`. +- Flag répété : la **première occurrence gagne**, comportement hérité de + `FlagValue` (`Array.IndexOf`) et commun à tous les verbes existants. +- **Le parsing de l'entrée est strict, celui du frontmatter reste permissif.** + `Lifecycle.From` résout un `status` inconnu en `stable` (§5.4 : le chargement + ne rejette rien, §11), alors que `--status retired` doit échouer. Ce n'est pas + une incohérence : un producteur ne contrôle pas ce qu'il lit, un utilisateur + contrôle ce qu'il tape, et une faute de frappe silencieusement absorbée en + `stable` rendrait une worklist fausse. Les deux parsers restent donc distincts + — ne pas « harmoniser » le strict vers le permissif. +- **Ordre de validation.** Les valeurs des flags sont validées **avant** la + résolution du positionnel. Sinon `okf audit --as-of` (le flag comme unique + argument) rendrait `missing ` : `Positional` saute le créneau d'un flag + valué sans vérifier qu'il a une valeur, et le vrai défaut serait masqué par un + diagnostic moins précis. + +**Piège de parsing à ne pas rater.** `--trust`, `--status`, `--type` et `--as-of` +consomment le token suivant : ils doivent être déclarés dans les `valuedFlags` de +`Positional(args, "", ...)`, sinon `okf audit --as-of 2099-06-01 mon/bundle` +prendrait `2099-06-01` pour le chemin du bundle. + +C'est exactement ce que `--out` a résolu pour `render`. Note de base : ces deux +helpers (`Positional(…, valuedFlags)` et `FlagValue`) sont arrivés avec le +travail viewer ; ils sont présents depuis le merge d'`origin/dev` dans la branche +d'implémentation. Sur une base antérieure, il faudrait les ajouter d'abord. + +### 4.2 Deux modes de présentation, une seule sélection + +**Sans aucun flag de filtre**, `okf audit ` sélectionne le **même +ensemble** que `--stale` ; seule la présentation change. Cette équivalence est +une règle testable, pas un effet de bord. + +Concrètement, le CLI passe `new AuditQuery(StaleOnly: true)` dans les deux cas : +`AuditQuery.All` (aucun prédicat) n'est jamais utilisé par le verbe — il ne sert +qu'aux appelants qui veulent la totalité du corpus, dont le tool agent invoqué +avec `stale: false` et aucun autre filtre. + +**Les flags de filtre sont exactement `--stale`, `--trust`, `--status` et +`--type`.** `--as-of` et `--json` n'en font pas partie et ne changent jamais de +mode : `okf audit --as-of 2099-06-01` reste en mode rapport — c'est +précisément l'invocation du golden (§7.3). + +**Conséquence à assumer : `audit` ne sait pas sélectionner tout le corpus en une +option.** Sans filtre il rend la worklist des périmés, et `--json` porte alors +des `findings` limités à ceux-ci — les compteurs, eux, restent corpus-larges. Qui +veut l'inventaire concept par concept énumère les trois tiers +(`--trust unverified,machine-confirmed,human-reviewed`). Aucun `--all` n'est +ajouté : `audit` est une worklist, l'inventaire est déjà le métier de +`okf info --json` et de `okf_browse`. Ce point doit apparaître tel quel dans la +documentation du verbe, faute de quoi un consommateur du JSON prendra `findings` +pour le corpus. + +**Mode rapport** (aucun flag de filtre) — synthèse + worklist : + +``` +bundle: tests/fixtures/okf_v02 +as of: 2099-06-01 +concepts: 2 + +trust: + 1 human-reviewed + 0 machine-confirmed + 1 unverified + +status: + 0 draft + 2 stable + 0 deprecated + +stale: 1 of 2 past stale_after + +needs attention (1): + metrics/dau stale 2099-01-01 human-reviewed stable +``` + +Conventions d'alignement reprises telles quelles de `info` : libellé + padding +jusqu'à la colonne 13 (`bundle:` suivi de 5 espaces), compteurs en ` {n,4} {label}`. +Ordre des tiers : du plus fort au plus faible. Ordre des statuts : l'ordre du +cycle de vie §5.4 (`draft`, `stable`, `deprecated`), pas l'ordre alphabétique. +Worklist vide ⇒ la dernière section devient la ligne `needs attention: none`. +Aucun plafond : c'est un CLI, la sortie se pipe. + +**Mode requête** (au moins un flag de filtre) — une ligne par concept, rien +d'autre, pour rester pipe-friendly (`| wc -l`, `| xargs`) : + +``` +metrics/dau stale 2099-01-01 human-reviewed stable +``` + +Sélection vide ⇒ **aucune sortie**, code 0. + +**Format d'une ligne de concept** (identique dans les deux modes, à l'indentation +de deux espaces près en mode rapport) — quatre champs séparés par deux espaces : + +1. l'id du concept (`ConceptId`, donc toujours normalisé avec `/`) ; +2. la fraîcheur : `stale `, `fresh `, ou + `no-stale-after` si le champ est absent **ou malformé** ; +3. le tier de confiance ; +4. le statut résolu. + +Asymétrie assumée entre texte et JSON sur le cas malformé : le texte affiche +`no-stale-after` (un `stale_after` illisible ne dit rien de la fraîcheur, et +c'est `validate` qui signale la valeur fautive), tandis que le JSON conserve la +valeur brute dans `staleAfter` avec `stale: false` — un outil qui consomme le +JSON doit pouvoir distinguer « champ absent » de « champ present mais illisible » +sans relire les fichiers. + +Imprimer l'id plutôt que le chemin n'est pas cosmétique : `ConceptId.FromPath` +normalise toujours en `/`, donc les goldens de `audit` sont comparables +byte-for-byte sur les trois OS **sans** la normalisation `Replace('\\','/')` que +`validate.out` doit subir ([GoldenParityTests.cs:74-85](../../../tests/OKF4net.Tests/GoldenParityTests.cs#L74-L85)). + +### 4.3 Codes de retour et erreurs + +- **0** : succès, y compris avec des findings. `audit` rapporte, il ne juge pas + la conformité. Pas de `--fail-on` tant qu'un utilisateur ne le demande pas. +- **1** : erreur d'invocation ou bundle illisible, via `CliOperationException`, + rendue par `Run` en `error: {message}` sur stderr. + +Messages exacts (nouveaux) : + +| Cas | stderr | +|---|---| +| `--as-of` invalide | `error: --as-of is not a valid YYYY-MM-DD date: "2026-13-01"` | +| `--trust` inconnu | `error: unknown trust tier "foo"; expected unverified, machine-confirmed or human-reviewed` | +| `--status` inconnu | `error: unknown status "foo"; expected draft, stable or deprecated` | + +Réutilisés tels quels : `error: missing ` (`Positional`) et +`error: --as-of requires a value` (`FlagValue`). + +`--as-of` est parsé exactement comme `Lifecycle.From` le fait pour `stale_after` +— même contrat, et compatible `InvariantGlobalization` (AOT) : + +```csharp +DateOnly.TryParseExact(raw, "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out var asOf) +``` + +Les cinq arguments sont obligatoires : `DateOnly` n'offre **pas** de surcharge +`(s, format, provider, out)` — seulement `(s, format, out)` et la forme complète +ci-dessus. Omettre `DateTimeStyles.None` ne compile pas. + +### 4.4 Texte d'aide + +Ajouter à `Usage` la ligne du verbe **juste après `validate`** (l'ordre de la +liste est vérifié par le test 24 : conformité d'abord, santé du corpus ensuite), +alignée sur les autres — le verbe occupe 8 colonnes, d'où quatre espaces après +`audit` : + +``` + audit Report trust, freshness and lifecycle across the bundle +``` + +et étendre la ligne d'option existante en +`--json Machine-readable output for validate/info/audit`. Le commentaire +de classe de `OkfCli` énumère les sous-commandes et annonce leur nombre : +l'incrémenter en comptant les verbes réellement présents sur la base (sept +depuis le merge du viewer, donc huit avec `audit`) et citer `audit`. + +## 5. Unité 3 — tool agent `okf_audit` + +Dans `OkfBundleTools`, sur le modèle de `Search` : + +```csharp +[Description("Audit the bundle's trust, freshness and lifecycle signals: counts by trust tier and status, plus the concepts needing attention. Filter with stale/trust/status/type.")] +public string Audit( + [Description("Only concepts past their stale_after date. Defaults to true.")] bool stale = true, + [Description("Comma-separated trust tiers to include: unverified, machine-confirmed, human-reviewed.")] string? trust = null, + [Description("Only concepts with this lifecycle status: draft, stable or deprecated.")] string? status = null, + [Description("Only concepts with this frontmatter type (exact match).")] string? type = null) +``` + +Enregistré dans `GetTools()` via `AIFunctionFactory.Create(Audit, "okf_audit")`. +**Lecture seule**, donc absent de `WriteToolNames` : il reste disponible quand +`okf-mcp` tourne en mode read-only. + +**La date d'observation vient de la couture existante, pas d'une horloge neuve.** +Le tool n'expose délibérément pas de paramètre `asOf` (un agent n'a pas à choisir +sa notion d'aujourd'hui), mais il ne doit pas non plus laisser `ConceptAudit` +retomber sur `SystemClock` : `OkfBundleTools` possède déjà le seam interne +`Func UtcNow` et la propriété `Today` qui en dérive — « the shared seam +behind `ReadConcept`'s and `Search`'s staleness checks » +([OkfBundleTools.cs:136](../../../src/OKF4net.Agents/OkfBundleTools.cs#L136)). +`Audit` passe donc `Today` à `ConceptAudit`, via un adaptateur privé de quatre +lignes : + +```csharp +private sealed class PinnedClock(DateOnly today) : IOkfClock +{ + public DateOnly Today { get; } = today; +} +``` + +Sans cela, la sortie du tool dépendrait du jour d'exécution et le cas limite +`today == stale_after` serait intestable. L'alternative — une surcharge publique +`ConceptAudit.Run(bundle, query, DateOnly asOf)` — est écartée pour garder une +seule forme canonique dans le cœur ; l'adaptateur reste local aux Agents et +n'ajoute aucune surface publique. + +Différences assumées avec le CLI : + +- rend **toujours** la forme rapport (synthèse + liste), même filtré : la + synthèse est du contexte utile pour un agent ; +- **plafonne à 20 findings**, suivis de `… and N more (narrow with stale/trust/status/type)` + — même plafond que `okf_search`, et c'est précisément l'économie de contexte + que l'article défend ; +- omet la ligne `bundle:` (le tool est lié à un seul bundle) ; +- valeurs invalides de `trust`/`status` ⇒ message d'usage rendu comme chaîne (pas + d'exception), sur le modèle de `SearchUsageMessage` ; +- **tout ce qui peut toucher le disque passe par `RunTool`**, la garde partagée + par tous les tools qui chargent le bundle : elle convertit `OkfException` + (donc `BundleLoadException`), `ArgumentException`, `IOException`, + `UnauthorizedAccessException` et `DecoderFallbackException` en une chaîne + `Error: …`. Un tool de fonction **rend** une erreur, il n'en lève pas : un + répertoire supprimé après la construction du tool remonterait sinon en + exception jusqu'au runtime de l'agent. Le rendu du vocabulaire suit la même + règle que le CLI — les libellés viennent d'`AuditVocabulary`, jamais de + littéraux recopiés. + +**Le rendu texte n'est pas partagé entre CLI et Agents** : seul le calcul +(`ConceptAudit`) l'est. Raison : les octets du CLI sont verrouillés par des +goldens et ne doivent jamais bouger parce qu'une chaîne destinée à un agent a +changé ; les deux rendus font une quinzaine de lignes chacun. C'est exactement le +partage retenu pour `ConceptSearch` (scorer commun, présentations distinctes). + +## 6. Sortie `--json` + +`System.Text.Json` **source-generated** — obligatoire, le CLI est publié Native +AOT. Nouveaux records internes dans `JsonOutput.cs`, plus +`[JsonSerializable(typeof(AuditJsonResult))]` sur `CliJsonContext` (le générateur +couvre le graphe atteignable ; seul le type racine s'annote). Nommage camelCase +via la policy déjà en place. + +```csharp +internal sealed record AuditQueryJson(bool Stale, IReadOnlyList? Trust, string? Status, string? Type); +internal sealed record TrustCountsJson(int HumanReviewed, int MachineConfirmed, int Unverified); +internal sealed record StatusCountsJson(int Draft, int Stable, int Deprecated); +internal sealed record AuditFindingJson(string ConceptId, string Path, string? Type, string? Title, string Trust, string Status, string? StaleAfter, bool Stale); +internal sealed record AuditJsonResult( + string Bundle, string AsOf, int ConceptCount, AuditQueryJson Query, + TrustCountsJson Trust, StatusCountsJson Status, int StaleCount, + IReadOnlyList Findings); +``` + +Exemple (`okf audit tests/fixtures/okf_v02 --as-of 2099-06-01 --json`, ici +ré-indenté ; la sortie réelle est une seule ligne suivie de `\n`) : + +```json +{ + "bundle": "tests/fixtures/okf_v02", + "asOf": "2099-06-01", + "conceptCount": 2, + "query": { "stale": true, "trust": null, "status": null, "type": null }, + "trust": { "humanReviewed": 1, "machineConfirmed": 0, "unverified": 1 }, + "status": { "draft": 0, "stable": 2, "deprecated": 0 }, + "staleCount": 1, + "findings": [ + { + "conceptId": "metrics/dau", + "path": "tests/fixtures/okf_v02/metrics/dau.md", + "type": "Metric", + "title": "Daily Active Users", + "trust": "human-reviewed", + "status": "stable", + "staleAfter": "2099-01-01", + "stale": true + } + ] +} +``` + +Décisions de schéma : + +- `--json` rend **toujours** le document complet, dans les deux modes : les + machines voient une seule forme, quelle que soit la présentation texte. +- `query` **rejoue la requête appliquée**, ce qui rend le document + auto-descriptif et lève l'ambiguïté du mode par défaut (`stale: true` sans + qu'aucun flag n'ait été passé). +- `query.trust` est sérialisé dans **l'ordre du ladder** (`unverified`, + `machine-confirmed`, `human-reviewed`), jamais dans l'ordre de saisie : + `IReadOnlySet` n'a pas d'ordre garanti, et sans cette règle le JSON ne serait + pas reproductible dès qu'on passe plusieurs tiers. `null` quand `--trust` est + absent. +- `staleAfter` porte la valeur **brute** du frontmatter ; elle vaut `null` si le + champ est absent, et la valeur brute non parsable s'il est malformé (auquel cas + `stale` est `false`). +- Pas de champ `statusKnown` : le statut inconnu est le domaine de `validate` + (§3.2). Schéma minimal, donc stable. +- `path` est le chemin réel du concept, donc porteur du séparateur natif de l'OS + — c'est le seul champ à normaliser côté sortie C# dans le test golden (§7.3). + +## 7. Tests + +### 7.1 Unitaires — `tests/OKF4net.Tests/AuditTests.cs` (nouveau) + +Bundles synthétiques + `FixedClock` (déjà présent). + +1. Les trois tiers sont comptés distinctement (`human:` ⇒ human-reviewed ; + vérificateur non-humain seul ⇒ machine-confirmed ; absence ⇒ unverified). +2. Statut inconnu compté comme `stable`. +3. Borne de staleness §5.5 : `AsOf == stale_after` ⇒ périmé ; `AsOf == stale_after - 1j` ⇒ non. +4. `stale_after` malformé ⇒ non périmé, absent de la worklist. +5. `stale_after` absent ⇒ jamais périmé. +6. Les prédicats se combinent en ET (`--stale` + `--trust` ne retient que + l'intersection). +7. `Findings` trié par id ordinal, indépendamment de l'ordre de chargement. +8. Les compteurs portent sur tout le bundle même quand la requête filtre. +9. Bundle vide ⇒ compteurs à zéro, `Findings` vide, aucune exception. +10. Documents non parsables exclus des compteurs, sans exception : un fichier au + frontmatter invalide atterrit dans `ParseErrors`, et `ConceptCount` ne le + compte pas. (Pas de test « fichier illisible » : ce cas lève + `BundleLoadException` au chargement et n'atteint jamais `ConceptAudit`.) +11. `clock: null` ⇒ `AsOf` = date UTC du jour. +12. `--type` : match ordinal exact ; casse différente ⇒ pas de match ; concept + sans `type` ⇒ jamais retenu. + +### 7.2 CLI — `tests/OKF4net.Tests/CliTests.cs` (existant) + +13. Mode rapport : sections et alignements attendus — **y compris avec `--as-of` + seul**, qui ne doit pas basculer en mode requête (§4.2). +14. Mode requête : uniquement des lignes de concepts, pas de synthèse ; et + l'idiome des trois tiers (`--trust unverified,machine-confirmed,human-reviewed`) + retourne bien la totalité des concepts du bundle. +15. **Équivalence** : `audit ` et `audit --stale` sélectionnent le même + ensemble (comparaison sur les ids). +16. `--json` : document parsable, champs et valeurs attendus, `query` rejoué. +17. `--as-of` invalide ⇒ code 1 + stderr exact. +18. `--trust` inconnu ⇒ code 1 + stderr exact. +19. `--status` inconnu ⇒ code 1 + stderr exact. +20. `--trust` avec une entrée vide (`unverified,,human-reviewed`) ⇒ code 1 ; + avec un doublon (`unverified,unverified`) ⇒ code 0 et même résultat qu'une + seule occurrence. +21. **Régression de parsing** : `okf audit --as-of ` (flags avant + le positionnel) résout le bon bundle ; idem pour `--trust`, `--status`, `--type`. +22. Code 0 malgré des findings. +23. Sélection vide ⇒ sortie vide, code 0. +24. `--help` liste `audit`. + +### 7.3 Goldens — `tests/OKF4net.Tests/GoldenParityTests.cs` + +Bundle réutilisé : `tests/fixtures/okf_v02` (2 concepts, déjà porteur des champs +v0.2), avec `--as-of 2099-06-01` **figé** — cette date rend `metrics/dau` +(`stale_after: 2099-01-01`) périmé sans toucher au bundle. Formulation exacte de +l'engagement : **aucun bundle de fixtures n'est créé ni modifié**, et **aucun +golden existant n'est modifié** ; les seuls ajouts sont des goldens neufs, +vérifiés à la main (voir plus bas). + +Nouveaux fichiers dans `tests/fixtures/golden/` : `audit-v02.out` et +`audit-v02.json`. **Pas de `audit-v02.exitcode`** : le code de retour d'`audit` +est constamment 0 (§4.3), un golden pour une constante n'apporte rien et grossit +la surface de fixtures. Le test l'assère en ligne (`Assert.Equal(0, r.Code)`), +exactement comme `Info_output_matches_golden`, qui n'a pas non plus de golden de +code retour ; les `*.exitcode` de `validate` n'existent que parce que son code +varie. + +- `audit-v02.out` : comparé **byte-for-byte sans normalisation** (la sortie ne + contient que des ids de concepts, toujours en `/`) — sauf la ligne `bundle:`, + qui reprend l'argument tel que passé, d'où le recours à `WithRepoRootAsCwd` + comme pour `validate`/`info`. +- `audit-v02.json` : comparé après `Replace('\\','/')` **sur la sortie C#, + jamais sur le golden**, à cause du champ `path` (§6) — même traitement, et même + justification, que `validate.out`. + +Provenance à consigner explicitement, la règle du repo l'exige : ces goldens sont +**écrits à la main** et vérifiés contre le texte du spec (§5.3 tiers, §5.4 +statuts, §5.5 staleness), **pas** capturés depuis le CLI de référence — `audit` +n'existe pas en amont. À documenter dans `tests/fixtures/README.md` et dans le +commentaire de classe de `GoldenParityTests`, au même titre que +`validate-v02.out`. + +### 7.4 Non-régression + +Aucun golden existant ne change. `dotnet test OKF4net.sln` doit rester vert sans +qu'un seul fichier de `tests/fixtures/` préexistant soit touché. + +### 7.5 Agents — `tests/OKF4net.Tests/Agents/` + +25. `okf_audit` est enregistré dans `GetTools()`. +26. Il est **absent** de `WriteToolNames` (donc exposé en mode read-only). +27. Plafond à 20 findings + ligne `… and N more`. +28. `trust`/`status` invalides ⇒ message d'usage rendu, pas d'exception. +29. **Date pinnée par le seam `UtcNow`** : avec `UtcNow` figé, la sortie est + déterministe, et la borne `today == stale_after` classe bien le concept comme + périmé. Sans ce test, le comportement clé dépendrait du jour d'exécution. + +## 8. Documentation à mettre à jour + +- `README.md` : liste des verbes CLI, et la table §-du-spec → type (ajouter + `ConceptAudit` en regard de §5.3–§5.5). +- `CHANGELOG.md` : entrée sous `Unreleased`. +- `ROADMAP.md` : inscrire l'échelle de l'article (index → recherche → graphe → + bibliothèque fédérée) et marquer ce premier barreau. +- `CLAUDE.md` : une ligne sur `ConceptAudit` comme surface de requête unique + partagée CLI/Agents — même statut que la note « ne pas forker `ConceptSearch` ». + +**Coordination.** Au moment d'écrire cette spec, `CLAUDE.md` et `ROADMAP.md` +étaient déjà modifiés, non commités, dans le worktree principal par une autre +session (travail viewer/CI). Les toucher ici provoquera un conflit au rebase : +ces deux fichiers sont à traiter en fin de branche, une fois l'autre session +mergée, et non au fil de l'implémentation. + +## 9. Contraintes respectées + +- **Zéro dépendance tierce** : `Audit.cs` n'utilise que la BCL ; aucun + `PackageReference` ajouté nulle part. +- **Native AOT** : JSON source-generated obligatoire, parsing de date en + `InvariantCulture` (compatible `InvariantGlobalization`). +- **Fidélité au spec** : aucun champ frontmatter nouveau, aucune extension. Tout + se dérive de champs v0.2 existants (§5.3/§5.4/§5.5). `audit` est un + **consommateur** du spec, pas une extension de celui-ci. +- **Fixtures** : aucune fixture existante modifiée ; les nouvelles relèvent de + l'exception documentée « comportement non couvert par une capture existante », + vérifiées à la main contre le texte du spec. +- **Conventions de fichier** : en-tête `// SPDX-License-Identifier: LGPL-3.0-or-later`, + namespace file-scoped, XML doc sur toute l'API publique, nullable activé, + warnings = erreurs. + +## 10. Alternatives écartées + +**A. Calculer dans le CLI, dupliquer côté Agents plus tard.** Écartée : elle +forke la logique dans deux couches, ce que le repo interdit explicitement pour le +scorer de recherche. Une seule implémentation, deux présentations. + +**B. Étendre `BundleValidator` avec des diagnostics `Info`, `audit` devenant une +vue filtrée de `validate`.** Écartée pour deux raisons cumulatives : elle mélange +la conformité §11 avec de l'hygiène éditoriale, et elle modifierait la sortie de +`validate`, donc les goldens byte-exact — interdit. + +**C. Un flag `--unverified` plutôt que `--trust `.** Écartée : « jamais +relu par un humain » recouvre deux tiers (`unverified` **et** +`machine-confirmed`) ; un booléen serait faux sur le second, et il ferait un +second vocabulaire en plus des noms de tiers. + +**D. Code de retour non nul quand il y a des findings.** Reportée : un concept +périmé n'est pas une erreur de conformité, et l'usage décrit par l'article est +une worklist hebdomadaire routée par équipe, pas un gate de CI. Si le besoin de +gate apparaît, il se traitera par un `--fail-on` explicite. diff --git a/samples/acme-retail-agent/README.md b/samples/acme-retail-agent/README.md index ec9202e..6fbaa55 100644 --- a/samples/acme-retail-agent/README.md +++ b/samples/acme-retail-agent/README.md @@ -51,6 +51,32 @@ OKF_CHAT_BASE_URL=http://localhost:11434/v1 OKF_CHAT_MODEL=llama3 \ (or pipe a prompt via stdin instead of `--prompt`). Type `exit` or `quit` to leave the interactive REPL. +## Questions worth asking + +Grounded answers about one concept — these go through +`okf_search`/`okf_read_concept`: + +- *What is Acme's FY2026 revenue recognition policy?* +- *How is gross margin defined, and what does it depend on?* + +Questions about the bundle **as a whole** — these go through `okf_audit`, in +one call rather than by opening concepts one at a time: + +- *Which concepts have never been verified by a human?* — the interesting + one today: eight of the nine concepts carry a `human:` verifier, so this + isolates `skills/run-on-bq`. +- *How healthy is this knowledge base — how much of it is human-reviewed, + and how much is stale?* +- *Is anything deprecated?* + +Note on the freshness question specifically: no concept in this bundle is +past its `stale_after` date *yet*, so "what is stale?" correctly answers +"nothing" today. Most concepts carry `stale_after: 2026-12-31`, so this +sample starts reporting seven stale concepts on 2027-01-01 — which is the +point the bundle is making, not a bug in it. To see the stale path before +then, the CLI can pin the date: `okf audit bundles/acme_retail --as-of +2027-06-01`. + ## What it does Wires `OkfBundleTools` (constructed without an `AttestationOrchestrator`, so @@ -62,6 +88,13 @@ interactive mode keeps one `AgentSession` across turns; one-shot mode runs a single turn and exits. Each response prints a `[tools: ...]` line naming any `okf_*` tools the agent called, for visibility into what it did. +Two kinds of question are demonstrated, and they use the tools differently. +Retrieval questions (`okf_search`, `okf_read_concept`, `okf_graph`) pull one +concept, or a few. Corpus-level questions about trust, freshness and +lifecycle go to `okf_audit`, which answers them in one call by reading every +concept's §5.3–§5.5 frontmatter — the `[tools: ...]` line is what shows you +which path the model actually took. + "Read-only" is enforced by construction, not just by convention: `Program.cs` filters `okf_write_concept`, `okf_append_log`, and `okf_regenerate_indexes` out of the tool list before it ever reaches the diff --git a/samples/acme-retail-agent/src/AcmeRetailAgent/AcmeRetailAgent.csproj b/samples/acme-retail-agent/src/AcmeRetailAgent/AcmeRetailAgent.csproj index 00d08b4..6dcd460 100644 --- a/samples/acme-retail-agent/src/AcmeRetailAgent/AcmeRetailAgent.csproj +++ b/samples/acme-retail-agent/src/AcmeRetailAgent/AcmeRetailAgent.csproj @@ -8,7 +8,7 @@ - + diff --git a/samples/acme-retail-agent/src/AcmeRetailAgent/Program.cs b/samples/acme-retail-agent/src/AcmeRetailAgent/Program.cs index 4fb2949..027f232 100644 --- a/samples/acme-retail-agent/src/AcmeRetailAgent/Program.cs +++ b/samples/acme-retail-agent/src/AcmeRetailAgent/Program.cs @@ -32,7 +32,10 @@ + "the okf_* tools to answer questions -- do not guess at bundle " + "content. Attested Computations can be inspected with " + "okf_get_computation (their contract and sanctioned SQL) but this " - + "sample cannot run them."; + + "sample cannot run them. Questions about the bundle as a whole -- " + + "which concepts are stale, unverified, deprecated, or of a given type -- " + + "are answered by a single okf_audit call, never by reading concepts " + + "one at a time."; var tools = new OkfBundleTools(bundleRoot); var contextProvider = new OkfContextProvider(tools); diff --git a/src/OKF4net.Agents/OkfBundleTools.cs b/src/OKF4net.Agents/OkfBundleTools.cs index 0a3b556..9371fd7 100644 --- a/src/OKF4net.Agents/OkfBundleTools.cs +++ b/src/OKF4net.Agents/OkfBundleTools.cs @@ -25,6 +25,11 @@ public sealed class OkfBundleTools "Usage: okf_changes_since requires a valid ISO date (yyyy-MM-dd), inclusive. " + "Example: okf_changes_since(\"2026-01-01\")."; + private const string AuditUsageMessage = + "Usage: okf_audit takes optional filters — stale (bool), trust (comma-separated: " + + "unverified, machine-confirmed, human-reviewed), status (draft, stable or deprecated) " + + "and type (exact frontmatter type). Example: okf_audit(stale: true, trust: \"unverified\")."; + /// /// The core write primitive this tool set delegates every write to: /// producer-validated create/update () and @@ -138,6 +143,16 @@ public OkfBundleTools(string bundleRoot, AttestationOrchestrator? orchestrator) /// Today's date, derived from — the shared seam behind 's and 's staleness checks. private DateOnly Today => DateOnly.FromDateTime(UtcNow().Date); + /// + /// Pins to — the same + /// UtcNow seam and use — so + /// the tool's output never depends on the day it runs. + /// + private sealed class PinnedClock(DateOnly today) : IOkfClock + { + public DateOnly Today { get; } = today; + } + /// /// Returns the loaded bundle, loading it from on /// first access and caching it thereafter until @@ -194,7 +209,7 @@ internal void InvalidateBundle() /// from each method's own /// — the single source of truth, so /// the two can never drift apart. The order is stable: read → browse → - /// graph → search → write → append → regenerate → validate → + /// graph → search → audit → write → append → regenerate → validate → /// changes-since → get-computation → (conditionally) run-computation. /// /// okf_get_computation is always included — it is read-only and @@ -213,6 +228,7 @@ public IList GetTools() AIFunctionFactory.Create(Browse, "okf_browse"), AIFunctionFactory.Create(Graph, "okf_graph"), AIFunctionFactory.Create(Search, "okf_search"), + AIFunctionFactory.Create(Audit, "okf_audit"), AIFunctionFactory.Create(WriteConcept, "okf_write_concept"), AIFunctionFactory.Create(AppendLog, "okf_append_log"), AIFunctionFactory.Create(RegenerateIndexes, "okf_regenerate_indexes"), @@ -446,6 +462,66 @@ public string Search( .Select(s => (s.Concept, s.Score)) .ToList(); + /// + /// Audits the bundle's trust, freshness and lifecycle signals (§5.3–§5.5): + /// counts over the whole bundle, then the concepts the filters select, + /// bounded to 20 entries. The worklist heading tracks : + /// with it true (the default) the selection is the stale worklist, so the + /// heading reads "needs attention"; with it false the selection can include + /// perfectly fresh concepts, so the heading reads the neutral "selected" + /// instead. + /// + /// Keep only concepts past their stale_after date. + /// Comma-separated trust tiers to keep. + /// Keep only concepts with this lifecycle status. + /// Keep only concepts with this frontmatter type (exact match). + [Description("Audit the bundle's trust, freshness and lifecycle signals: counts by trust tier and status, plus the concepts needing attention. Filter with stale/trust/status/type.")] + public string Audit( + [Description("Only concepts past their stale_after date. Defaults to true.")] bool stale = true, + [Description("Comma-separated trust tiers to include: unverified, machine-confirmed, human-reviewed.")] string? trust = null, + [Description("Only concepts with this lifecycle status: draft, stable or deprecated.")] string? status = null, + [Description("Only concepts with this frontmatter type (exact match).")] string? type = null) + { + HashSet? tiers = null; + if (trust is not null) + { + if (!AuditVocabulary.TryParseTrustTiers(trust, out var parsed, out _)) + { + return AuditUsageMessage; + } + + tiers = parsed; + } + + ConceptStatus? parsedStatus = null; + if (status is not null) + { + if (!AuditVocabulary.TryParseStatus(status.Trim(), out var value)) + { + return AuditUsageMessage; + } + + parsedStatus = value; + } + + // Everything that can touch the filesystem goes through RunTool, the + // guard every bundle-loading tool uses: it turns OkfException (hence + // BundleLoadException), ArgumentException, IOException, + // UnauthorizedAccessException and DecoderFallbackException into an + // "Error: ..." string. A function tool must return an error, not throw + // one -- a directory deleted after construction would otherwise escape + // as an exception into the agent runtime. + return RunTool(() => + { + var report = ConceptAudit.Run( + GetBundle(), + new AuditQuery(stale, tiers, parsedStatus, type), + new PinnedClock(Today)); + + return RenderAudit(report, staleOnly: stale); + }); + } + /// /// Creates or updates one concept document. Producer-grade validation /// (: non-empty type, @@ -1083,6 +1159,76 @@ private static string FormatSearchResults(string query, string? tag, IReadOnlyLi return sb.ToString(); } + /// + /// Renders an audit for an agent: the same shape as the CLI's report form, + /// minus the bundle line (the tool is bound to one bundle) and bounded to + /// 20 findings. Deliberately not shared with the CLI renderer, whose bytes + /// are golden-locked and must not move when this string is tuned. + /// + /// The audit report to render. + /// + /// Whether the selection IS the stale worklist (the tool's stale + /// parameter). When true, the worklist heading reads needs attention + /// -- otherwise, the selection was narrowed or widened by other filters, so + /// the neutral selected heading is used instead: calling every + /// selected concept a concept that "needs attention" would misstate a + /// selection like stale: false, which can include perfectly fresh + /// concepts. + /// + private static string RenderAudit(AuditReport report, bool staleOnly) + { + const int MaxResults = 20; + var sb = new StringBuilder(); + + sb.Append("as of: ").Append(report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)).Append('\n'); + sb.Append("concepts: ").Append(report.ConceptCount).Append('\n'); + + // Same rule as the CLI renderer: labels from AuditVocabulary, never + // literals. The two renderers are separate on purpose (the CLI's bytes + // are golden-locked), but they must not spell the vocabulary twice. + sb.Append("\ntrust:\n"); + foreach (var tier in AuditVocabulary.TrustTiersInOrder.Reverse()) + { + sb.Append($" {report.TrustCounts[tier],4} {AuditVocabulary.Name(tier)}\n"); + } + + sb.Append("\nstatus:\n"); + foreach (var status in AuditVocabulary.StatusesInOrder) + { + sb.Append($" {report.StatusCounts[status],4} {AuditVocabulary.Name(status)}\n"); + } + + sb.Append($"\nstale: {report.StaleCount} of {report.ConceptCount} past stale_after\n"); + + var heading = staleOnly ? "needs attention" : "selected"; + + if (report.Findings.Count == 0) + { + sb.Append($"\n{heading}: none\n"); + return sb.ToString(); + } + + sb.Append($"\n{heading} ({report.Findings.Count}):\n"); + foreach (var finding in report.Findings.Take(MaxResults)) + { + var freshness = AuditVocabulary.Freshness(finding.Lifecycle, finding.IsStale); + + sb.Append(" ") + .Append(finding.Id) + .Append(" ").Append(freshness) + .Append(" ").Append(AuditVocabulary.Name(finding.Trust)) + .Append(" ").Append(AuditVocabulary.Name(finding.Lifecycle.Status)) + .Append('\n'); + } + + if (report.Findings.Count > MaxResults) + { + sb.Append($"… and {report.Findings.Count - MaxResults} more (narrow with stale/trust/status/type)\n"); + } + + return sb.ToString(); + } + /// /// Builds the fallback listing for a bundle level with no index.md: /// the subdirectories and concepts found directly under diff --git a/src/OKF4net.Agents/README.md b/src/OKF4net.Agents/README.md index 70fe308..f65e2c3 100644 --- a/src/OKF4net.Agents/README.md +++ b/src/OKF4net.Agents/README.md @@ -26,11 +26,13 @@ AIAgent agent = chatClient.AsAIAgent( var response = await agent.RunAsync("Summarize the concepts in this bundle."); ``` -## The nine tools +## The eleven tools -`okf_read_concept`, `okf_browse`, `okf_graph`, `okf_search`, +`okf_read_concept`, `okf_browse`, `okf_graph`, `okf_search`, `okf_audit`, `okf_write_concept`, `okf_append_log`, `okf_regenerate_indexes`, -`okf_validate_bundle`, `okf_changes_since`. +`okf_validate_bundle`, `okf_changes_since`, `okf_get_computation` — plus a +twelfth, `okf_run_computation`, only when the tool set is constructed with an +`OKF4net.Attestation` orchestrator wired in. All tools return agent-friendly markdown/plain text and never throw for expected errors (unknown ids, invalid paths, malformed input) — the agent diff --git a/src/OKF4net.Cli/JsonOutput.cs b/src/OKF4net.Cli/JsonOutput.cs index 31b5668..9b8ad62 100644 --- a/src/OKF4net.Cli/JsonOutput.cs +++ b/src/OKF4net.Cli/JsonOutput.cs @@ -1,4 +1,5 @@ // SPDX-License-Identifier: LGPL-3.0-or-later +using System.Globalization; using System.Text.Json; using System.Text.Json.Serialization; @@ -38,6 +39,37 @@ internal sealed record InfoJsonResult( int BrokenLinkCount, IReadOnlyList ParseErrors); +/// The query okf audit applied, replayed for --json consumers. +internal sealed record AuditQueryJson(bool Stale, IReadOnlyList? Trust, string? Status, string? Type); + +/// Concept counts per trust tier (§5.3), over the whole bundle. +internal sealed record TrustCountsJson(int HumanReviewed, int MachineConfirmed, int Unverified); + +/// Concept counts per lifecycle status (§5.4), over the whole bundle. +internal sealed record StatusCountsJson(int Draft, int Stable, int Deprecated); + +/// One selected concept, projected for --json output. +internal sealed record AuditFindingJson( + string ConceptId, + string Path, + string? Type, + string? Title, + string Trust, + string Status, + string? StaleAfter, + bool Stale); + +/// The full result of okf audit --json. +internal sealed record AuditJsonResult( + string Bundle, + string AsOf, + int ConceptCount, + AuditQueryJson Query, + TrustCountsJson Trust, + StatusCountsJson Status, + int StaleCount, + IReadOnlyList Findings); + /// /// Source-generated for every /// --json output type. Required, not optional, because the CLI is @@ -49,11 +81,12 @@ internal sealed record InfoJsonResult( [JsonSourceGenerationOptions(PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)] [JsonSerializable(typeof(ValidateJsonResult))] [JsonSerializable(typeof(InfoJsonResult))] +[JsonSerializable(typeof(AuditJsonResult))] internal partial class CliJsonContext : JsonSerializerContext { } -/// Builds and writes the --json output for validate and info. +/// Builds and writes the --json output for validate, info and audit. internal static class JsonOutput { /// Writes okf validate --json's result to as a single line-terminated JSON document. @@ -112,6 +145,59 @@ internal static void WriteInfo(TextWriter stdout, string bundlePath, Bundle bund stdout.Write("\n"); } + /// Writes okf audit --json's result to as a single line-terminated JSON document. + internal static void WriteAudit(TextWriter stdout, string bundlePath, AuditQuery query, AuditReport report) + { + // Serialized in ladder order, never in the order the user typed them: + // IReadOnlySet has no guaranteed order, and the document must be + // reproducible. Written as a statement rather than a ternary so the + // nullable analysis narrows `Trust` through the pattern -- it does not + // narrow a property across the arms of a conditional. + List? trustQuery = null; + if (query.Trust is { } selectedTiers) + { + trustQuery = AuditVocabulary.TrustTiersInOrder + .Where(selectedTiers.Contains) + .Select(AuditVocabulary.Name) + .ToList(); + } + + var findings = report.Findings + .Select(f => new AuditFindingJson( + f.Id.ToString(), + f.Path, + f.Type, + f.Title, + AuditVocabulary.Name(f.Trust), + AuditVocabulary.Name(f.Lifecycle.Status), + f.Lifecycle.StaleAfterRaw, + f.IsStale)) + .ToList(); + + var result = new AuditJsonResult( + bundlePath, + report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture), + report.ConceptCount, + new AuditQueryJson( + query.StaleOnly, + trustQuery, + query.Status is { } status ? AuditVocabulary.Name(status) : null, + query.Type), + new TrustCountsJson( + report.TrustCounts[TrustTier.HumanReviewed], + report.TrustCounts[TrustTier.MachineConfirmed], + report.TrustCounts[TrustTier.Unverified]), + new StatusCountsJson( + report.StatusCounts[ConceptStatus.Draft], + report.StatusCounts[ConceptStatus.Stable], + report.StatusCounts[ConceptStatus.Deprecated]), + report.StaleCount, + findings); + + stdout.Write(JsonSerializer.Serialize(result, CliJsonContext.Default.AuditJsonResult)); + stdout.Write("\n"); + } + /// Counts concepts by frontmatter type (a missing type counts as "(none)"), sorted ordinally by type name. internal static SortedDictionary BuildTypeHistogram(Bundle bundle) { diff --git a/src/OKF4net.Cli/OkfCli.cs b/src/OKF4net.Cli/OkfCli.cs index 301ef28..5f9dbf9 100644 --- a/src/OKF4net.Cli/OkfCli.cs +++ b/src/OKF4net.Cli/OkfCli.cs @@ -1,4 +1,5 @@ // SPDX-License-Identifier: LGPL-3.0-or-later +using System.Globalization; using System.Text; using OKF4net.Internal; using OKF4net.Viewer; @@ -6,10 +7,10 @@ namespace OKF4net.Cli; /// -/// The okf command-line tool. Seven subcommands (validate, -/// info, index, graph, parse, fmt, -/// render) over hand-rolled argument parsing -- no third-party -/// dependencies. +/// The okf command-line tool. Eight subcommands (validate, +/// audit, info, index, graph, parse, +/// fmt, render) over hand-rolled argument parsing -- no +/// third-party dependencies. /// /// is the sole public entry point so tests can drive the /// CLI in-process (capturing stdout/stderr) without spawning a subprocess; @@ -29,6 +30,7 @@ public static class OkfCli "\n" + "COMMANDS:\n" + " validate Check a bundle against OKF v0.2 conformance (§11)\n" + + " audit Report trust, freshness and lifecycle across the bundle\n" + " info Summarize a bundle (concepts, types, links, version)\n" + " index (Re)generate every index.md in the bundle\n" + " graph Print the cross-link graph (--dot for Graphviz DOT)\n" + @@ -39,7 +41,7 @@ public static class OkfCli "OPTIONS:\n" + " -h, --help Show this help\n" + " -V, --version Show version\n" + - " --json Machine-readable output for validate/info\n" + + " --json Machine-readable output for validate/info/audit\n" + " --out Output directory for `render`"; /// @@ -86,6 +88,7 @@ public static int Run(string[] args, TextWriter stdout, TextWriter stderr) return cmd switch { "validate" => CmdValidate(rest, stdout), + "audit" => CmdAudit(rest, stdout), "info" => CmdInfo(rest, stdout), "index" => CmdIndex(rest, stdout), "graph" => CmdGraph(rest, stdout), @@ -313,6 +316,163 @@ private static int CmdValidate(string[] args, TextWriter stdout) return 1; } + /// The flags that make audit a filtered query rather than a report. + private static readonly string[] AuditFilterFlags = ["--stale", "--trust", "--status", "--type"]; + + /// Every audit flag that consumes the following token as its value. + private static readonly string[] AuditValuedFlags = ["--trust", "--status", "--type", "--as-of"]; + + /// An pinned to one date, backing --as-of. + private sealed class PinnedClock(DateOnly today) : IOkfClock + { + public DateOnly Today { get; } = today; + } + + /// Implements the audit subcommand. + private static int CmdAudit(string[] args, TextWriter stdout) + { + // Flag values are validated BEFORE the positional is resolved. An + // unvalued flag is the more specific diagnosis, and `okf audit --as-of` + // -- the flag as the only argument -- would otherwise report + // "missing " and hide the actual mistake, because Positional + // skips a valued flag's slot without checking that it has a value. + var clock = ParseAsOf(args); + + // Report mode selects exactly what --stale selects; only the + // presentation differs. --as-of and --json never switch modes. + var filtered = AuditFilterFlags.Any(f => HasFlag(args, f)); + var query = filtered ? ParseAuditQuery(args) : new AuditQuery(StaleOnly: true); + + var path = Positional(args, "", AuditValuedFlags); + var bundle = Load(path); + var report = ConceptAudit.Run(bundle, query, clock); + + if (HasFlag(args, "--json")) + { + JsonOutput.WriteAudit(stdout, path, query, report); + return 0; + } + + if (filtered) + { + foreach (var finding in report.Findings) + { + stdout.Write(FormatAuditFinding(finding)); + stdout.Write("\n"); + } + + return 0; + } + + WriteAuditReport(stdout, path, report); + return 0; + } + + /// Parses --as-of; null when absent (the audit then uses the system clock). + private static IOkfClock? ParseAsOf(string[] args) + { + var raw = FlagValue(args, "--as-of"); + if (raw is null) + { + return null; + } + + // DateOnly has no (s, format, provider, out) overload -- the five-argument + // form is the only one that takes a culture, and it is the same contract + // Lifecycle.From uses for stale_after. + if (!DateOnly.TryParseExact(raw, "yyyy-MM-dd", CultureInfo.InvariantCulture, DateTimeStyles.None, out var asOf)) + { + throw new CliOperationException($"--as-of is not a valid YYYY-MM-DD date: \"{raw}\""); + } + + return new PinnedClock(asOf); + } + + /// Builds the query from the filter flags. Throws on an unknown vocabulary value. + private static AuditQuery ParseAuditQuery(string[] args) + { + HashSet? tiers = null; + var trustRaw = FlagValue(args, "--trust"); + if (trustRaw is not null) + { + if (!AuditVocabulary.TryParseTrustTiers(trustRaw, out var parsed, out var badEntry)) + { + throw new CliOperationException( + $"unknown trust tier \"{badEntry}\"; expected unverified, machine-confirmed or human-reviewed"); + } + + tiers = parsed; + } + + ConceptStatus? status = null; + var statusRaw = FlagValue(args, "--status"); + if (statusRaw is not null) + { + if (!AuditVocabulary.TryParseStatus(statusRaw.Trim(), out var parsed)) + { + throw new CliOperationException( + $"unknown status \"{statusRaw.Trim()}\"; expected draft, stable or deprecated"); + } + + status = parsed; + } + + return new AuditQuery( + HasFlag(args, "--stale"), + tiers, + status, + FlagValue(args, "--type")); + } + + /// Renders one concept line: id, freshness, trust tier, status -- two spaces between fields. + private static string FormatAuditFinding(AuditFinding finding) + { + var freshness = AuditVocabulary.Freshness(finding.Lifecycle, finding.IsStale); + + return $"{finding.Id} {freshness} {AuditVocabulary.Name(finding.Trust)} {AuditVocabulary.Name(finding.Lifecycle.Status)}"; + } + + /// Renders the report form: summary counters over the whole bundle, then the worklist. + private static void WriteAuditReport(TextWriter stdout, string bundlePath, AuditReport report) + { + stdout.Write($"bundle: {bundlePath}\n"); + stdout.Write($"as of: {report.AsOf.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture)}\n"); + stdout.Write($"concepts: {report.ConceptCount}\n"); + + // Labels always come from AuditVocabulary -- never as literals here. + // Duplicating them in each renderer is exactly the drift the shared + // vocabulary exists to prevent. Only the ORDER is decided locally: the + // report shows the strongest tier first, so it walks the canonical + // (weakest-first) list in reverse. + stdout.Write("\ntrust:\n"); + foreach (var tier in AuditVocabulary.TrustTiersInOrder.Reverse()) + { + stdout.Write($" {report.TrustCounts[tier],4} {AuditVocabulary.Name(tier)}\n"); + } + + stdout.Write("\nstatus:\n"); + foreach (var status in AuditVocabulary.StatusesInOrder) + { + stdout.Write($" {report.StatusCounts[status],4} {AuditVocabulary.Name(status)}\n"); + } + + stdout.Write($"\nstale: {report.StaleCount} of {report.ConceptCount} past stale_after\n"); + + if (report.Findings.Count == 0) + { + stdout.Write("\nneeds attention: none\n"); + return; + } + + stdout.Write($"\nneeds attention ({report.Findings.Count}):\n"); + foreach (var finding in report.Findings) + { + stdout.Write(" "); + stdout.Write(FormatAuditFinding(finding)); + stdout.Write("\n"); + } + } + /// Implements the info subcommand. private static int CmdInfo(string[] args, TextWriter stdout) { diff --git a/src/OKF4net.Mcp/README.md b/src/OKF4net.Mcp/README.md index f42b528..bbc7e6a 100644 --- a/src/OKF4net.Mcp/README.md +++ b/src/OKF4net.Mcp/README.md @@ -68,20 +68,21 @@ or `OKF_BUNDLE_ROOT` in `claude_desktop_config.json`. ## Tools -`okf_read_concept`, `okf_browse`, `okf_graph`, `okf_search`, `okf_write_concept`, -`okf_append_log`, `okf_regenerate_indexes`, `okf_validate_bundle`, +`okf_read_concept`, `okf_browse`, `okf_graph`, `okf_search`, `okf_audit`, +`okf_write_concept`, `okf_append_log`, `okf_regenerate_indexes`, `okf_validate_bundle`, `okf_changes_since`, `okf_get_computation`. Each is the corresponding `OkfBundleTools` operation, so all OKF v0.2 behaviour, producer-grade validation, path-safety, and locking apply unchanged. -That's ten tools full (seven read-only tools above plus the three write -tools), or seven when `OKF_MCP_READONLY=1` drops the three write tools. +That's eleven tools full (eight read-only tools above plus the three write +tools), or eight when `OKF_MCP_READONLY=1` drops the three write tools. `okf_get_computation` reads a §10 attested-computation concept's contract and -sanctioned computation source — read-only, no attestation runtime needed. The -eleventh `OkfBundleTools` tool, `okf_run_computation`, is **not** exposed by -this server: it only appears in `GetTools()` when the tool set is constructed -with an `OKF4net.Attestation` `AttestationOrchestrator` wired in, and this -server starts `OkfBundleTools` with no orchestrator (it wires no -host-specific binder/executor/attester runtime). Embed `OKF4net.Agents` -directly if you need `okf_run_computation`. +sanctioned computation source — read-only, no attestation runtime needed. +`okf_audit` reads the bundle's trust/freshness/lifecycle signals — also +read-only. The twelfth `OkfBundleTools` tool, `okf_run_computation`, is +**not** exposed by this server: it only appears in `GetTools()` when the tool +set is constructed with an `OKF4net.Attestation` `AttestationOrchestrator` +wired in, and this server starts `OkfBundleTools` with no orchestrator (it +wires no host-specific binder/executor/attester runtime). Embed +`OKF4net.Agents` directly if you need `okf_run_computation`. diff --git a/src/OKF4net/Audit.cs b/src/OKF4net/Audit.cs new file mode 100644 index 0000000..8fb4582 --- /dev/null +++ b/src/OKF4net/Audit.cs @@ -0,0 +1,287 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +using System.Globalization; + +namespace OKF4net; + +/// +/// The selection predicates of an audit (§5.3–§5.5). Predicates combine with +/// AND; default selects every concept. +/// +/// +/// The generated equality compares by reference (the +/// behaviour of EqualityComparer<IReadOnlySet<T>>.Default), so two +/// logically identical queries may compare unequal. Do not rely on it, and do +/// not use an as a dictionary key: the record struct is +/// for with and ToString, not for its equality. +/// +/// Keep only concepts past their stale_after date. +/// Keep only concepts whose derived tier is in this set; null keeps every tier. +/// Keep only concepts with this lifecycle status; null keeps every status. +/// Keep only concepts whose frontmatter type matches exactly (ordinal); null keeps every type. +public readonly record struct AuditQuery( + bool StaleOnly = false, + IReadOnlySet? Trust = null, + ConceptStatus? Status = null, + string? Type = null) +{ + /// The query that keeps every concept. + public static AuditQuery All => default; + + /// + /// Whether this query constrains the selection below "every concept" -- + /// true whenever any predicate is non-default. This is not "did the + /// caller type a filter flag": the CLI's report mode builds + /// new AuditQuery(StaleOnly: true) itself, with no flag typed, and + /// this is still for it. + /// + public bool IsFiltered => StaleOnly || Trust is not null || Status is not null || Type is not null; +} + +/// One concept selected by an audit, with its signals already derived. +/// The concept id (§2). +/// The concept's file path, as built from the bundle root. +/// The frontmatter type, or null when absent. +/// The frontmatter title, or null when absent. +/// The derived trust tier (§5.3). +/// The lifecycle fields (§5.4/§5.5). +/// Whether the concept is stale as of the report's . +public readonly record struct AuditFinding( + ConceptId Id, + string Path, + string? Type, + string? Title, + TrustTier Trust, + Lifecycle Lifecycle, + bool IsStale); + +/// +/// The result of an audit: counts over the whole bundle, plus the concepts the +/// query selected. The counts never narrow with the query -- the denominator +/// stays stable while moves. +/// +public sealed class AuditReport +{ + internal AuditReport( + DateOnly asOf, + int conceptCount, + IReadOnlyDictionary trustCounts, + IReadOnlyDictionary statusCounts, + int staleCount, + IReadOnlyList findings) + { + AsOf = asOf; + ConceptCount = conceptCount; + TrustCounts = trustCounts; + StatusCounts = statusCounts; + StaleCount = staleCount; + Findings = findings; + } + + /// The observation date staleness was evaluated against. + public DateOnly AsOf { get; } + + /// The number of concepts in the bundle (not in the selection). + public int ConceptCount { get; } + + /// Concept counts per trust tier over the whole bundle; all three keys are always present. + public IReadOnlyDictionary TrustCounts { get; } + + /// Concept counts per lifecycle status over the whole bundle; all three keys are always present. + public IReadOnlyDictionary StatusCounts { get; } + + /// The number of stale concepts in the whole bundle. + public int StaleCount { get; } + + /// The selected concepts, sorted by concept id (ordinal). + public IReadOnlyList Findings { get; } +} + +/// +/// The single spelling of the audit vocabularies, shared by every surface (CLI +/// input, CLI text, JSON, agent tool) so no two layers can drift apart. +/// +public static class AuditVocabulary +{ + /// + /// The trust tiers, weakest to strongest -- the canonical order. JSON + /// serializes in this order; the text report walks it in reverse, showing + /// the strongest tier first. + /// + public static IReadOnlyList TrustTiersInOrder { get; } = + [TrustTier.Unverified, TrustTier.MachineConfirmed, TrustTier.HumanReviewed]; + + /// The lifecycle statuses in §5.4 order -- the order every surface displays them in. + public static IReadOnlyList StatusesInOrder { get; } = + [ConceptStatus.Draft, ConceptStatus.Stable, ConceptStatus.Deprecated]; + + /// The wire/display name of a trust tier. + public static string Name(TrustTier tier) => tier switch + { + TrustTier.HumanReviewed => "human-reviewed", + TrustTier.MachineConfirmed => "machine-confirmed", + _ => "unverified", + }; + + /// The wire/display name of a lifecycle status. + public static string Name(ConceptStatus status) => status switch + { + ConceptStatus.Draft => "draft", + ConceptStatus.Deprecated => "deprecated", + _ => "stable", + }; + + /// Parses a trust tier name (exact, ordinal). Unlike frontmatter parsing, an unknown name fails rather than defaulting. + public static bool TryParseTrustTier(string text, out TrustTier tier) + { + switch (text) + { + case "unverified": tier = TrustTier.Unverified; return true; + case "machine-confirmed": tier = TrustTier.MachineConfirmed; return true; + case "human-reviewed": tier = TrustTier.HumanReviewed; return true; + default: tier = TrustTier.Unverified; return false; + } + } + + /// Parses a lifecycle status name (exact, ordinal). Unlike , an unknown name fails rather than resolving to stable. + public static bool TryParseStatus(string text, out ConceptStatus status) + { + switch (text) + { + case "draft": status = ConceptStatus.Draft; return true; + case "stable": status = ConceptStatus.Stable; return true; + case "deprecated": status = ConceptStatus.Deprecated; return true; + default: status = ConceptStatus.Stable; return false; + } + } + + /// + /// Parses a comma-separated list of trust tier names (e.g. + /// "unverified,human-reviewed"). Each entry is trimmed and parsed + /// with ; a duplicate entry is absorbed + /// silently since is a set. The single grammar + /// shared by the CLI's --trust flag and the okf_audit tool's + /// trust parameter -- callers differ only in what they do on + /// failure (the CLI raises a CliOperationException naming + /// ; the tool returns a usage message). + /// + /// The comma-separated list, as typed by the caller. + /// On success, the parsed tiers; empty on failure. + /// On failure, the first entry (trimmed) that failed to parse; on success. + /// when every entry parsed. + public static bool TryParseTrustTiers(string raw, out HashSet tiers, out string? badEntry) + { + tiers = []; + foreach (var entry in raw.Split(',')) + { + var trimmed = entry.Trim(); + if (!TryParseTrustTier(trimmed, out var tier)) + { + tiers = []; + badEntry = trimmed; + return false; + } + + tiers.Add(tier); + } + + badEntry = null; + return true; + } + + /// + /// The freshness token for one concept line (§5.5): "stale <date>" + /// or "fresh <date>" when stale_after parsed, or + /// "no-stale-after" when it is absent or malformed. The single + /// spelling shared by both renderers -- do not spell these tokens as + /// literals outside this method. + /// + /// The concept's lifecycle fields. + /// Whether the concept is stale as of the report's AsOf date. + public static string Freshness(Lifecycle lifecycle, bool isStale) => + lifecycle.StaleAfter is { } date + ? (isStale ? "stale " : "fresh ") + date.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture) + : "no-stale-after"; +} + +/// +/// Queries a bundle's §5.3–§5.5 signals. Reads nothing from disk, writes +/// nothing, and never throws on data: an unparseable document is already absent +/// from , and a malformed stale_after simply +/// reads as "not stale" (the validator owns that diagnostic). +/// +public static class ConceptAudit +{ + /// Runs over . + /// The loaded bundle. + /// The selection predicates; default selects everything. + /// Supplies "today" for staleness (§5.5); defaults to . + public static AuditReport Run(Bundle bundle, AuditQuery query = default, IOkfClock? clock = null) + { + var asOf = (clock ?? new SystemClock()).Today; + + var trustCounts = new Dictionary + { + [TrustTier.Unverified] = 0, + [TrustTier.MachineConfirmed] = 0, + [TrustTier.HumanReviewed] = 0, + }; + + var statusCounts = new Dictionary + { + [ConceptStatus.Draft] = 0, + [ConceptStatus.Stable] = 0, + [ConceptStatus.Deprecated] = 0, + }; + + var staleCount = 0; + var findings = new List(); + + foreach (var concept in bundle.Concepts) + { + var frontmatter = concept.Document.Frontmatter; + var tier = frontmatter.TrustTier; + var lifecycle = frontmatter.Lifecycle; + var isStale = lifecycle.IsStale(asOf); + + trustCounts[tier]++; + statusCounts[lifecycle.Status]++; + if (isStale) + { + staleCount++; + } + + if (query.StaleOnly && !isStale) + { + continue; + } + + if (query.Trust is { } tiers && !tiers.Contains(tier)) + { + continue; + } + + if (query.Status is { } status && lifecycle.Status != status) + { + continue; + } + + if (query.Type is { } type && !string.Equals(frontmatter.Type, type, StringComparison.Ordinal)) + { + continue; + } + + findings.Add(new AuditFinding( + concept.Id, + concept.Path, + frontmatter.Type, + frontmatter.Title, + tier, + lifecycle, + isStale)); + } + + findings.Sort(static (a, b) => a.Id.CompareTo(b.Id)); + + return new AuditReport(asOf, bundle.Count, trustCounts, statusCounts, staleCount, findings); + } +} diff --git a/tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs b/tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs index de95130..4b81f0b 100644 --- a/tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs +++ b/tests/OKF4net.Tests/Agents/AIFunctionExposureTests.cs @@ -9,13 +9,14 @@ namespace OKF4net.Tests.Agents; /// -/// Tests : the ten tool methods exposed -/// as Agent Framework s (via ) -/// when no attestation orchestrator is wired (so okf_run_computation -/// is omitted; see for the wired -/// case), with no LLM involved — everything is verified at the -/// level, including a real end-to-end invocation -/// that proves argument binding from a plain dictionary works. +/// Tests : the eleven tool methods +/// exposed as Agent Framework s (via +/// ) when no attestation orchestrator is wired (so +/// okf_run_computation is omitted; see +/// for the wired case), with no LLM +/// involved — everything is verified at the level, +/// including a real end-to-end invocation that proves argument binding from +/// a plain dictionary works. /// public class AIFunctionExposureTests { @@ -27,6 +28,7 @@ public class AIFunctionExposureTests "okf_browse", "okf_graph", "okf_search", + "okf_audit", "okf_write_concept", "okf_append_log", "okf_regenerate_indexes", @@ -36,14 +38,14 @@ public class AIFunctionExposureTests ]; [Fact] - public void GetTools_returns_exactly_ten_tools() + public void GetTools_returns_exactly_eleven_tools() { var tools = new OkfBundleTools(BundlePath); - Assert.Equal(10, tools.GetTools().Count); + Assert.Equal(11, tools.GetTools().Count); } [Fact] - public void GetTools_names_are_the_ten_snake_case_names_in_stable_order() + public void GetTools_names_are_the_eleven_snake_case_names_in_stable_order() { var tools = new OkfBundleTools(BundlePath); var names = tools.GetTools().Cast().Select(f => f.Name).ToList(); diff --git a/tests/OKF4net.Tests/Agents/AgentIntegrationTests.cs b/tests/OKF4net.Tests/Agents/AgentIntegrationTests.cs index 02fde3e..ea7a613 100644 --- a/tests/OKF4net.Tests/Agents/AgentIntegrationTests.cs +++ b/tests/OKF4net.Tests/Agents/AgentIntegrationTests.cs @@ -126,4 +126,76 @@ public async Task Agent_writes_concept_regenerates_indexes_and_validates_end_to_ Assert.Contains("✓ conformant", scriptedClient.ObservedFunctionResults[2]); Assert.DoesNotContain("✗", scriptedClient.ObservedFunctionResults[2]); } + + /// + /// The read-only counterpart of the write scenario, and the question the + /// audit surface exists for: "which concepts are past their stale_after + /// date and were never verified by a human?". It is the only test that + /// drives okf_audit through the framework's real function-invoking + /// pipeline rather than calling the C# method directly, so it is what + /// proves the JSON argument binding works: the scripted "model" passes a + /// boolean and a comma-separated tier list as JSON, and the framework + /// must bind them to the method's bool and string? + /// parameters for the filter to take effect. + /// + /// The bundle is built inline so the observation date can be pinned via + /// the UtcNow seam: metrics/orphan is stale AND unverified + /// (the answer), while metrics/dau is equally stale but + /// human-reviewed (the concept the trust filter must exclude). A binding + /// failure that silently dropped the trust argument would return + /// both and fail here. + /// + [Fact] + public async Task Agent_audits_the_bundle_for_stale_never_human_verified_concepts() + { + using var tmp = new TempDir(); + tmp.Write( + "metrics/dau.md", + "---\ntype: Metric\ntitle: Daily Active Users\nstale_after: 2026-01-01\n" + + "verified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write( + "metrics/orphan.md", + "---\ntype: Metric\ntitle: Orphaned Metric\nstale_after: 2026-01-01\n---\n"); + + var tools = new OkfBundleTools(tmp.Path) + { + UtcNow = () => new DateTime(2026, 8, 21, 0, 0, 0, DateTimeKind.Utc), + }; + + const string auditAnswer = "One concept is stale and was never verified by a human: metrics/orphan."; + + var scriptedClient = new ScriptedChatClient( + [ + ScriptStep.Call("okf_audit", new Dictionary + { + ["stale"] = true, + ["trust"] = "unverified,machine-confirmed", + }), + ScriptStep.Answer(auditAnswer), + ]); + + AIAgent agent = scriptedClient.AsAIAgent(tools: tools.GetTools()); + + var response = await agent.RunAsync( + "Quels concepts ont dépassé leur stale_after sans avoir jamais été vérifiés par un humain ?"); + + Assert.Equal(auditAnswer, response.Text); + + // Two round-trips: the tool-call turn and the final-answer turn. + Assert.Equal(2, scriptedClient.TurnsTaken); + + var auditResult = Assert.Single(scriptedClient.ObservedFunctionResults); + + // The filters bound and took effect: the stale, unverified concept is + // listed under the worklist heading; the stale but human-reviewed one + // is not. + Assert.Contains("needs attention (1):", auditResult); + Assert.Contains("metrics/orphan stale 2026-01-01 unverified", auditResult); + Assert.DoesNotContain("metrics/dau", auditResult); + + // The counters still describe the whole bundle, not the selection -- + // the distinction the audit surface is built on. + Assert.Contains("concepts: 2", auditResult); + Assert.Contains("stale: 2 of 2 past stale_after", auditResult); + } } diff --git a/tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs b/tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs new file mode 100644 index 0000000..ab8a9e7 --- /dev/null +++ b/tests/OKF4net.Tests/Agents/OkfAuditToolTests.cs @@ -0,0 +1,171 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +using Microsoft.Extensions.AI; +using OKF4net.Agents; + +namespace OKF4net.Tests.Agents; + +/// +/// Tests for the okf_audit tool. Every test pins UtcNow: the tool +/// deliberately exposes no asOf parameter, so the shared clock seam is +/// the only way its output can be made deterministic. +/// +public class OkfAuditToolTests +{ + private static OkfBundleTools ToolsOver(TempDir tmp, DateOnly today) + => new(tmp.Path) + { + UtcNow = () => new DateTime(today.Year, today.Month, today.Day, 0, 0, 0, DateTimeKind.Utc), + }; + + [Fact] + public void Audit_is_registered_and_read_only() + { + var tools = new OkfBundleTools(Path.Combine(TestPaths.RepoRoot(), "tests", "fixtures", "okf_v02")); + + Assert.Contains("okf_audit", tools.GetTools().OfType().Select(t => t.Name)); + Assert.DoesNotContain("okf_audit", OkfBundleTools.WriteToolNames); + } + + /// + /// §5.5's boundary is today >= stale_after, so a concept whose + /// stale_after is exactly today is stale. Without the pinned seam this + /// assertion would silently depend on the day the suite runs. + /// + [Fact] + public void Audit_treats_today_equals_stale_after_as_stale() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstale_after: 2026-08-21\n---\n"); + + var onTheDay = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + var theDayBefore = ToolsOver(tmp, new DateOnly(2026, 8, 20)).Audit(); + + Assert.Contains("a stale 2026-08-21", onTheDay); + Assert.Contains("needs attention: none", theDayBefore); + } + + [Fact] + public void Audit_reports_counts_and_omits_the_bundle_line() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + + Assert.DoesNotContain("bundle:", text); + Assert.Contains("as of: 2026-08-21", text); + Assert.Contains(" 1 human-reviewed", text); + Assert.Contains(" 1 unverified", text); + } + + [Fact] + public void Audit_caps_the_listing_at_twenty_findings() + { + using var tmp = new TempDir(); + for (var i = 0; i < 25; i++) + { + tmp.Write($"c{i:D2}.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + } + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(); + + Assert.Contains("… and 5 more (narrow with stale/trust/status/type)", text); + + // Finding lines are the two-space-indented ones; matching on "c" alone + // would also catch the "concepts:" header. + Assert.Equal(20, text.Split('\n').Count(l => l.StartsWith(" c", StringComparison.Ordinal))); + } + + [Fact] + public void Audit_renders_a_usage_message_for_invalid_vocabulary() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + + var tools = ToolsOver(tmp, new DateOnly(2026, 8, 21)); + + Assert.Contains("Usage: okf_audit", tools.Audit(trust: "machine")); + Assert.Contains("Usage: okf_audit", tools.Audit(status: "retired")); + } + + /// + /// A function tool returns errors, it does not throw them: a bundle that + /// disappears after the tool was constructed must surface as an "Error: ..." + /// string, which is what the shared RunTool guard provides. + /// + [Fact] + public void Audit_returns_an_error_string_when_the_bundle_cannot_be_loaded() + { + var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + var tools = ToolsOver(tmp, new DateOnly(2026, 8, 21)); + tmp.Dispose(); // the directory is gone before the first load + + var text = tools.Audit(); + + Assert.StartsWith("Error: ", text); + } + + [Fact] + public void Audit_with_stale_false_and_no_filter_returns_the_whole_corpus() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false); + + Assert.Contains("selected (2):", text); + } + + /// + /// Regression guard for the bug fixed here: with stale: false, the + /// selection can include perfectly fresh concepts, so calling every one of + /// them "needs attention" would be a factual misstatement the agent could + /// relay verbatim. + /// + [Fact] + public void Audit_with_stale_false_never_says_needs_attention() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false); + + Assert.DoesNotContain("needs attention", text); + } + + /// + /// The stale-only path (the tool's default) is the one place "needs + /// attention" is an accurate label -- every selected concept really is + /// past its stale_after date. + /// + [Fact] + public void Audit_stale_only_path_still_says_needs_attention() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: true); + + Assert.Contains("needs attention (1):", text); + } + + /// + /// The freshness token for a concept with no stale_after at all + /// must come from , same as the CLI + /// -- this line is the tool-side coverage the shared vocabulary lacked. + /// + [Fact] + public void Audit_finding_line_reports_no_stale_after_when_the_field_is_absent() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + + var text = ToolsOver(tmp, new DateOnly(2026, 8, 21)).Audit(stale: false); + + Assert.Contains("a no-stale-after unverified stable", text); + } +} diff --git a/tests/OKF4net.Tests/Agents/OkfBundleToolsTests.cs b/tests/OKF4net.Tests/Agents/OkfBundleToolsTests.cs index 154f03f..6fd5fb5 100644 --- a/tests/OKF4net.Tests/Agents/OkfBundleToolsTests.cs +++ b/tests/OKF4net.Tests/Agents/OkfBundleToolsTests.cs @@ -53,12 +53,13 @@ public void WriteToolNames_matches_the_three_mutating_tools_and_filters_them_out .Where(name => !OkfBundleTools.WriteToolNames.Contains(name)) .ToHashSet(); - Assert.Equal(7, readOnlyNames.Count); + Assert.Equal(8, readOnlyNames.Count); Assert.DoesNotContain("okf_write_concept", readOnlyNames); Assert.DoesNotContain("okf_append_log", readOnlyNames); Assert.DoesNotContain("okf_regenerate_indexes", readOnlyNames); Assert.Contains("okf_read_concept", readOnlyNames); Assert.Contains("okf_get_computation", readOnlyNames); + Assert.Contains("okf_audit", readOnlyNames); } /// diff --git a/tests/OKF4net.Tests/AuditTests.cs b/tests/OKF4net.Tests/AuditTests.cs new file mode 100644 index 0000000..163c302 --- /dev/null +++ b/tests/OKF4net.Tests/AuditTests.cs @@ -0,0 +1,243 @@ +// SPDX-License-Identifier: LGPL-3.0-or-later +namespace OKF4net.Tests; + +/// +/// Unit tests for : tier and status counting, +/// the §5.5 staleness boundary, predicate composition and ordering. +/// Every test pins the date with so nothing here +/// depends on the day the suite runs. +/// +public class AuditTests +{ + private static readonly DateOnly Today = new(2026, 8, 21); + + private static Bundle Load(TempDir tmp) => Bundle.Load(tmp.Path); + + private static AuditReport Audit(TempDir tmp, AuditQuery query = default) + => ConceptAudit.Run(Load(tmp), query, new FixedClock(Today)); + + [Fact] + public void Counts_the_three_trust_tiers() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\nverified:\n - { by: process:nightly, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("c.md", "---\ntype: Metric\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(3, report.ConceptCount); + Assert.Equal(1, report.TrustCounts[TrustTier.HumanReviewed]); + Assert.Equal(1, report.TrustCounts[TrustTier.MachineConfirmed]); + Assert.Equal(1, report.TrustCounts[TrustTier.Unverified]); + } + + [Fact] + public void Unknown_status_counts_as_stable() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstatus: retired\n---\n"); + tmp.Write("b.md", "---\ntype: Metric\nstatus: draft\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(1, report.StatusCounts[ConceptStatus.Stable]); + Assert.Equal(1, report.StatusCounts[ConceptStatus.Draft]); + Assert.Equal(0, report.StatusCounts[ConceptStatus.Deprecated]); + } + + [Theory] + [InlineData("2026-08-21", true)] // §5.5: today >= stale_after -- the exact boundary IS stale. + [InlineData("2026-08-22", false)] + public void Staleness_boundary_follows_section_5_5(string staleAfter, bool expectedStale) + { + using var tmp = new TempDir(); + tmp.Write("a.md", $"---\ntype: Metric\nstale_after: {staleAfter}\n---\n"); + + // The default query filters nothing, so the single concept is always returned. + var report = Audit(tmp); + + Assert.Equal(expectedStale, report.Findings.Single().IsStale); + Assert.Equal(expectedStale ? 1 : 0, report.StaleCount); + } + + [Fact] + public void Malformed_or_absent_stale_after_is_never_stale() + { + using var tmp = new TempDir(); + tmp.Write("bad.md", "---\ntype: Metric\nstale_after: not-a-date\n---\n"); + tmp.Write("none.md", "---\ntype: Metric\n---\n"); + + var report = Audit(tmp); + + Assert.Equal(0, report.StaleCount); + Assert.Empty(ConceptAudit.Run(Load(tmp), new AuditQuery(StaleOnly: true), new FixedClock(Today)).Findings); + Assert.True(report.Findings.Single(f => f.Id.ToString() == "bad").Lifecycle.StaleAfterMalformed); + } + + [Fact] + public void Predicates_compose_with_and() + { + using var tmp = new TempDir(); + tmp.Write("stale-unverified.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + tmp.Write("stale-human.md", "---\ntype: Metric\nstale_after: 2026-01-01\nverified:\n - { by: human:ada, at: 2026-01-01T00:00:00Z }\n---\n"); + tmp.Write("fresh-unverified.md", "---\ntype: Metric\nstale_after: 2099-01-01\n---\n"); + + var query = new AuditQuery( + StaleOnly: true, + Trust: new HashSet { TrustTier.Unverified }); + + var findings = ConceptAudit.Run(Load(tmp), query, new FixedClock(Today)).Findings; + + Assert.Equal(["stale-unverified"], findings.Select(f => f.Id.ToString())); + } + + [Fact] + public void Findings_are_sorted_by_concept_id_component_wise() + { + using var tmp = new TempDir(); + tmp.Write("zeta.md", "---\ntype: Metric\n---\n"); + tmp.Write("alpha.md", "---\ntype: Metric\n---\n"); + tmp.Write("mid/beta.md", "---\ntype: Metric\n---\n"); + // Discriminating pair: component-wise ordering differs from flat ordinal. + // ConceptId.CompareTo compares segments; "orders/extra" has segments ["orders", "extra"], + // and "orders-extra" is a single segment. The first segment "orders" < "orders-extra", + // so "orders/extra" comes before "orders-extra" under component-wise ordering. + tmp.Write("orders-extra.md", "---\ntype: Metric\n---\n"); + tmp.Write("orders/extra.md", "---\ntype: Metric\n---\n"); + + var findings = Audit(tmp).Findings.Select(f => f.Id.ToString()).ToList(); + + Assert.Equal(["alpha", "mid/beta", "orders/extra", "orders-extra", "zeta"], findings); + } + + [Fact] + public void Counts_cover_the_whole_bundle_even_when_the_query_filters() + { + using var tmp = new TempDir(); + tmp.Write("stale.md", "---\ntype: Metric\nstale_after: 2026-01-01\n---\n"); + tmp.Write("fresh.md", "---\ntype: Metric\n---\n"); + + var report = ConceptAudit.Run(Load(tmp), new AuditQuery(StaleOnly: true), new FixedClock(Today)); + + Assert.Single(report.Findings); + Assert.Equal(2, report.ConceptCount); + Assert.Equal(2, report.TrustCounts[TrustTier.Unverified]); + Assert.Equal(1, report.StaleCount); + } + + [Fact] + public void Empty_bundle_yields_zeroed_counts_and_no_findings() + { + using var tmp = new TempDir(); + + var report = Audit(tmp); + + Assert.Equal(0, report.ConceptCount); + Assert.Empty(report.Findings); + Assert.Equal(0, report.TrustCounts[TrustTier.HumanReviewed]); + Assert.Equal(0, report.StatusCounts[ConceptStatus.Deprecated]); + } + + /// + /// A document whose frontmatter cannot be parsed lands in + /// Bundle.ParseErrors (permissive loading) and is not a concept, so + /// it must not reach any counter. Note this is a *parse* failure -- a truly + /// unreadable file (I/O, permissions, non-UTF-8) throws + /// BundleLoadException and never reaches . + /// + [Fact] + public void Unparseable_documents_are_excluded_from_every_count() + { + using var tmp = new TempDir(); + tmp.Write("ok.md", "---\ntype: Metric\n---\n"); + tmp.Write("broken.md", "---\ntype: Metric\n"); // unterminated frontmatter block + + var bundle = Load(tmp); + Assert.Single(bundle.ParseErrors); + + var report = ConceptAudit.Run(bundle, default, new FixedClock(Today)); + + Assert.Equal(1, report.ConceptCount); + Assert.Single(report.Findings); + } + + [Fact] + public void Null_clock_falls_back_to_today_in_utc() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\n---\n"); + + var report = ConceptAudit.Run(Load(tmp)); + + Assert.Equal(DateOnly.FromDateTime(DateTime.UtcNow.Date), report.AsOf); + } + + [Fact] + public void Type_filter_is_exact_and_ordinal() + { + using var tmp = new TempDir(); + tmp.Write("metric.md", "---\ntype: Metric\n---\n"); + tmp.Write("lower.md", "---\ntype: metric\n---\n"); + tmp.Write("untyped.md", "---\ntitle: No type here\n---\n"); + + var findings = ConceptAudit.Run(Load(tmp), new AuditQuery(Type: "Metric"), new FixedClock(Today)).Findings; + + Assert.Equal(["metric"], findings.Select(f => f.Id.ToString())); + } + + [Fact] + public void Vocabulary_names_round_trip() + { + Assert.Equal("human-reviewed", AuditVocabulary.Name(TrustTier.HumanReviewed)); + Assert.Equal("machine-confirmed", AuditVocabulary.Name(TrustTier.MachineConfirmed)); + Assert.Equal("unverified", AuditVocabulary.Name(TrustTier.Unverified)); + Assert.Equal("draft", AuditVocabulary.Name(ConceptStatus.Draft)); + + Assert.True(AuditVocabulary.TryParseTrustTier("machine-confirmed", out var tier)); + Assert.Equal(TrustTier.MachineConfirmed, tier); + Assert.False(AuditVocabulary.TryParseTrustTier("machine", out _)); + + Assert.True(AuditVocabulary.TryParseStatus("deprecated", out var status)); + Assert.Equal(ConceptStatus.Deprecated, status); + Assert.False(AuditVocabulary.TryParseStatus("retired", out _)); + } + + /// + /// The single grammar shared by the CLI's --trust flag and the + /// okf_audit tool's trust parameter: comma-split, trim each + /// entry, absorb duplicates (the result is a set), fail on the first + /// unparseable entry and report it (trimmed) via . + /// + [Fact] + public void TryParseTrustTiers_trims_absorbs_duplicates_and_reports_the_bad_entry() + { + Assert.True(AuditVocabulary.TryParseTrustTiers( + "unverified, unverified,human-reviewed", out var tiers, out var badEntry)); + Assert.Equal( + new HashSet { TrustTier.Unverified, TrustTier.HumanReviewed }, + tiers); + Assert.Null(badEntry); + + Assert.False(AuditVocabulary.TryParseTrustTiers( + "unverified,,human-reviewed", out var afterFailure, out var emptyEntry)); + Assert.Empty(afterFailure); + Assert.Equal("", emptyEntry); + + Assert.False(AuditVocabulary.TryParseTrustTiers("bogus", out _, out var unknownEntry)); + Assert.Equal("bogus", unknownEntry); + } + + /// + /// answers "does this query constrain + /// the selection?", not "did the caller type a flag?" -- the CLI's report + /// mode builds new AuditQuery(StaleOnly: true) itself, with no flag + /// typed, and must still see here. + /// + [Fact] + public void IsFiltered_reflects_whether_the_query_constrains_the_selection() + { + Assert.False(AuditQuery.All.IsFiltered); + Assert.True(new AuditQuery(StaleOnly: true).IsFiltered); + } +} diff --git a/tests/OKF4net.Tests/CliTests.cs b/tests/OKF4net.Tests/CliTests.cs index c5c3edd..be1e29a 100644 --- a/tests/OKF4net.Tests/CliTests.cs +++ b/tests/OKF4net.Tests/CliTests.cs @@ -1,5 +1,6 @@ // SPDX-License-Identifier: LGPL-3.0-or-later using System.Linq; +using System.Text.Json; using System.Text.RegularExpressions; using OKF4net.Cli; @@ -20,6 +21,9 @@ public class CliTests // rather than assumed relative to the process's current directory. private static readonly string BundlePath = Path.Combine(TestPaths.RepoRoot(), "tests", "fixtures", "appendix_a"); + private static readonly string V02BundlePath = + Path.Combine(TestPaths.RepoRoot(), "tests", "fixtures", "okf_v02"); + private static (int Code, string Out, string Err) Run(params string[] args) => TestPaths.Run(args); [Fact] @@ -434,4 +438,296 @@ public void Render_with_only_out_and_no_bundle_fails_rather_than_treating_the_ou Assert.Contains("error:", r.Err); Assert.False(Directory.Exists(outDir)); } + + // ---------------------------------------------------------------- + // audit + // ---------------------------------------------------------------- + + [Fact] + public void Audit_report_mode_prints_summary_and_worklist() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + + Assert.Equal(0, r.Code); + Assert.Contains("as of: 2099-06-01\n", r.Out); + Assert.Contains("concepts: 2\n", r.Out); + Assert.Contains(" 1 human-reviewed\n", r.Out); + Assert.Contains(" 1 unverified\n", r.Out); + Assert.Contains(" 2 stable\n", r.Out); + Assert.Contains("stale: 1 of 2 past stale_after\n", r.Out); + Assert.Contains("needs attention (1):\n", r.Out); + Assert.Contains(" metrics/dau stale 2099-01-01 human-reviewed stable\n", r.Out); + } + + [Fact] + public void Audit_query_mode_prints_bare_lines_only() + { + var r = Run("audit", V02BundlePath, "--stale", "--as-of", "2099-06-01"); + + Assert.Equal(0, r.Code); + Assert.Equal("metrics/dau stale 2099-01-01 human-reviewed stable\n", r.Out); + } + + [Fact] + public void Audit_without_flags_selects_the_same_set_as_stale() + { + var report = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + var query = Run("audit", V02BundlePath, "--stale", "--as-of", "2099-06-01"); + + var reportIds = report.Out + .Split('\n') + .Where(l => l.StartsWith(" metrics/", StringComparison.Ordinal)) + .Select(l => l.Trim().Split(" ")[0]) + .ToList(); + var queryIds = query.Out + .Split('\n', StringSplitOptions.RemoveEmptyEntries) + .Select(l => l.Split(" ")[0]) + .ToList(); + + Assert.Equal(queryIds, reportIds); + } + + [Fact] + public void Audit_empty_selection_prints_nothing() + { + var r = Run("audit", V02BundlePath, "--status", "deprecated"); + + Assert.Equal(0, r.Code); + Assert.Equal("", r.Out); + } + + /// + /// Positive-selection coverage for --type: both fixture concepts + /// are type: Metric, so the exact-case value must select both, and + /// the lowercase variant must select none -- pinning the documented + /// ordinal, case-sensitive rule (§ Audit.cs AuditQuery.Type) at the + /// CLI boundary. Without this, transposing Type/Status in + /// ParseAuditQuery would leave the suite green. + /// + [Fact] + public void Audit_type_filter_selects_matching_concepts_case_sensitively() + { + var match = Run("audit", V02BundlePath, "--type", "Metric"); + Assert.Equal(0, match.Code); + Assert.Equal(2, match.Out.Split('\n', StringSplitOptions.RemoveEmptyEntries).Length); + + var noMatch = Run("audit", V02BundlePath, "--type", "metric"); + Assert.Equal(0, noMatch.Code); + Assert.Equal("", noMatch.Out); + } + + /// + /// Positive-selection coverage for --status: both fixture concepts + /// resolve to status: stable (one explicitly, one via + /// Lifecycle.From's fallback for the unknown "retired" value), so + /// this must select both. + /// + [Fact] + public void Audit_status_filter_selects_matching_concepts() + { + var r = Run("audit", V02BundlePath, "--status", "stable"); + + Assert.Equal(0, r.Code); + Assert.Equal(2, r.Out.Split('\n', StringSplitOptions.RemoveEmptyEntries).Length); + } + + [Fact] + public void Audit_three_tier_idiom_returns_every_concept() + { + var r = Run("audit", V02BundlePath, "--trust", "unverified,machine-confirmed,human-reviewed"); + + Assert.Equal(0, r.Code); + Assert.Equal(2, r.Out.Split('\n', StringSplitOptions.RemoveEmptyEntries).Length); + } + + [Fact] + public void Audit_rejects_an_invalid_as_of_date() + { + var r = Run("audit", V02BundlePath, "--as-of", "2026-13-01"); + + Assert.Equal(1, r.Code); + Assert.Equal("error: --as-of is not a valid YYYY-MM-DD date: \"2026-13-01\"\n", r.Err); + } + + [Fact] + public void Audit_rejects_an_unknown_trust_tier() + { + var r = Run("audit", V02BundlePath, "--trust", "foo"); + + Assert.Equal(1, r.Code); + Assert.Equal( + "error: unknown trust tier \"foo\"; expected unverified, machine-confirmed or human-reviewed\n", + r.Err); + } + + [Fact] + public void Audit_rejects_an_unknown_status() + { + var r = Run("audit", V02BundlePath, "--status", "retired"); + + Assert.Equal(1, r.Code); + Assert.Equal("error: unknown status \"retired\"; expected draft, stable or deprecated\n", r.Err); + } + + [Fact] + public void Audit_rejects_an_empty_trust_entry_but_absorbs_duplicates() + { + var empty = Run("audit", V02BundlePath, "--trust", "unverified,,human-reviewed"); + Assert.Equal(1, empty.Code); + Assert.Contains("unknown trust tier", empty.Err); + + var duplicated = Run("audit", V02BundlePath, "--trust", "unverified,unverified"); + var single = Run("audit", V02BundlePath, "--trust", "unverified"); + Assert.Equal(0, duplicated.Code); + Assert.Equal(single.Out, duplicated.Out); + } + + /// + /// Regression guard: valued flags must be declared to Positional, or + /// their value is mistaken for the bundle path when they precede it. + /// + [Theory] + [InlineData("--as-of", "2099-06-01")] + [InlineData("--trust", "unverified")] + [InlineData("--status", "stable")] + [InlineData("--type", "Metric")] + public void Audit_valued_flags_before_the_positional_resolve_the_bundle(string flag, string value) + { + var r = Run("audit", flag, value, V02BundlePath); + + Assert.Equal(0, r.Code); + Assert.Equal("", r.Err); + } + + /// + /// A valued flag left without a value must name itself, even when it is the + /// only argument -- otherwise the user is told the bundle is missing and the + /// real mistake is hidden. This is why CmdAudit validates flag values before + /// resolving the positional. + /// + [Theory] + [InlineData("--as-of")] + [InlineData("--trust")] + [InlineData("--status")] + [InlineData("--type")] + public void Audit_reports_a_valued_flag_left_without_a_value(string flag) + { + var r = Run("audit", flag); + + Assert.Equal(1, r.Code); + Assert.Equal($"error: {flag} requires a value\n", r.Err); + } + + [Fact] + public void Audit_as_of_alone_stays_in_report_mode() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01"); + + Assert.Contains("needs attention", r.Out); + } + + /// + /// Report mode's empty-worklist branch: --as-of pinned before + /// metrics/dau's stale_after (2099-01-01) leaves nothing + /// stale, so WriteAuditReport takes its early-return branch and + /// prints the "none" line instead of a "needs attention (N):" worklist. + /// + [Fact] + public void Audit_report_mode_prints_none_when_nothing_is_stale() + { + var r = Run("audit", V02BundlePath, "--as-of", "2026-01-01"); + + Assert.Equal(0, r.Code); + Assert.Contains("stale: 0 of 2 past stale_after\n", r.Out); + Assert.Contains("needs attention: none\n", r.Out); + Assert.DoesNotContain("needs attention (", r.Out); + } + + /// + /// metrics/legacy has no stale_after at all, so + /// FormatAuditFinding takes its "no-stale-after" branch rather than + /// "stale "/"fresh " + a date. --trust unverified selects exactly + /// this concept (it has no verified entries), independent of + /// --as-of/the system clock. + /// + [Fact] + public void Audit_finding_line_reports_no_stale_after_when_the_field_is_absent() + { + var r = Run("audit", V02BundlePath, "--trust", "unverified"); + + Assert.Equal(0, r.Code); + Assert.Equal("metrics/legacy no-stale-after unverified stable\n", r.Out); + } + + [Fact] + public void Help_lists_audit_right_after_validate() + { + var r = Run("--help"); + + Assert.Equal(0, r.Code); + var lines = r.Out.Split('\n').Select(l => l.TrimStart()).ToList(); + var validateIndex = lines.FindIndex(l => l.StartsWith("validate ", StringComparison.Ordinal)); + var auditIndex = lines.FindIndex(l => l.StartsWith("audit ", StringComparison.Ordinal)); + + Assert.True(validateIndex >= 0 && auditIndex == validateIndex + 1); + } + + [Fact] + public void Audit_json_carries_counts_query_and_findings() + { + var r = Run("audit", V02BundlePath, "--as-of", "2099-06-01", "--json"); + + Assert.Equal(0, r.Code); + Assert.EndsWith("\n", r.Out); + + using var doc = JsonDocument.Parse(r.Out); + var root = doc.RootElement; + + Assert.Equal("2099-06-01", root.GetProperty("asOf").GetString()); + Assert.Equal(2, root.GetProperty("conceptCount").GetInt32()); + Assert.Equal(1, root.GetProperty("staleCount").GetInt32()); + + // Report mode selects what --stale selects, so the replayed query says so. + Assert.True(root.GetProperty("query").GetProperty("stale").GetBoolean()); + Assert.Equal(JsonValueKind.Null, root.GetProperty("query").GetProperty("trust").ValueKind); + + Assert.Equal(1, root.GetProperty("trust").GetProperty("humanReviewed").GetInt32()); + Assert.Equal(1, root.GetProperty("trust").GetProperty("unverified").GetInt32()); + Assert.Equal(2, root.GetProperty("status").GetProperty("stable").GetInt32()); + + var finding = root.GetProperty("findings").EnumerateArray().Single(); + Assert.Equal("metrics/dau", finding.GetProperty("conceptId").GetString()); + Assert.Equal("Metric", finding.GetProperty("type").GetString()); + Assert.Equal("Daily Active Users", finding.GetProperty("title").GetString()); + Assert.Equal("human-reviewed", finding.GetProperty("trust").GetString()); + Assert.Equal("2099-01-01", finding.GetProperty("staleAfter").GetString()); + Assert.True(finding.GetProperty("stale").GetBoolean()); + } + + [Fact] + public void Audit_json_serializes_trust_query_in_ladder_order() + { + var r = Run("audit", V02BundlePath, "--trust", "human-reviewed,unverified", "--json"); + + using var doc = JsonDocument.Parse(r.Out); + var trust = doc.RootElement.GetProperty("query").GetProperty("trust") + .EnumerateArray().Select(e => e.GetString()).ToList(); + + Assert.Equal(["unverified", "human-reviewed"], trust); + } + + [Fact] + public void Audit_json_keeps_a_malformed_stale_after_raw_and_not_stale() + { + using var tmp = new TempDir(); + tmp.Write("a.md", "---\ntype: Metric\nstale_after: not-a-date\n---\n"); + + var r = Run("audit", tmp.Path, "--trust", "unverified", "--json"); + + using var doc = JsonDocument.Parse(r.Out); + var finding = doc.RootElement.GetProperty("findings").EnumerateArray().Single(); + + Assert.Equal("not-a-date", finding.GetProperty("staleAfter").GetString()); + Assert.False(finding.GetProperty("stale").GetBoolean()); + } } diff --git a/tests/OKF4net.Tests/GoldenParityTests.cs b/tests/OKF4net.Tests/GoldenParityTests.cs index 3fce43c..f5dd381 100644 --- a/tests/OKF4net.Tests/GoldenParityTests.cs +++ b/tests/OKF4net.Tests/GoldenParityTests.cs @@ -117,6 +117,45 @@ public void Info_output_matches_golden() Assert.Equal(Golden("info.out"), r.Out); } + /// + /// audit's goldens are hand-authored and verified against the spec + /// text (§5.3 tiers, §5.4 statuses, §5.5 staleness), not captured from the + /// reference CLI -- the verb has no upstream counterpart. The date is + /// pinned with --as-of so the output cannot drift with the calendar. + /// There is no audit-v02.exitcode: the verb always exits 0, so the + /// code is asserted inline (as does). + /// + [Fact] + public void Audit_report_matches_golden() + { + var r = WithRepoRootAsCwd(() => Run("audit", "tests/fixtures/okf_v02", "--as-of", "2099-06-01")); + + Assert.Equal(0, r.Code); + + // Every path in this output is a concept id, always '/'-normalized by + // ConceptId.FromPath, so the comparison is strict byte-for-byte. + Assert.Equal(Golden("audit-v02.out"), r.Out); + } + + [Fact] + public void Audit_json_matches_golden() + { + var r = WithRepoRootAsCwd(() => Run("audit", "tests/fixtures/okf_v02", "--as-of", "2099-06-01", "--json")); + + Assert.Equal(0, r.Code); + + // Only the findings' `path` field carries a native separator (it is a + // real file path, not a concept id), so it alone is normalized in the + // C# OUTPUT -- never in the golden -- exactly like validate.out. Unlike + // validate.out (plain text, one real backslash char per separator), + // r.Out here is serialized JSON: JsonSerializer escapes each backslash + // as the two-character sequence `\\` in the JSON text itself. A naive + // single-char Replace('\\', '/') would turn that pair into "//" instead + // of collapsing it to one '/', so the search pattern below is the + // two-character escaped sequence, not the bare character. + Assert.Equal(Golden("audit-v02.json"), r.Out.Replace("\\\\", "/")); + } + [Fact] public void Graph_dot_matches_golden() { diff --git a/tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs b/tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs index 2ea625a..1c8de63 100644 --- a/tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs +++ b/tests/OKF4net.Tests/Mcp/OkfMcpServerTests.cs @@ -76,7 +76,7 @@ public async Task Write_then_read_round_trips_through_mcp() } [Fact] - public async Task Build_exposes_all_ten_tools() + public async Task Build_exposes_all_eleven_tools() { var bundle = NewBundleDir(); try @@ -92,7 +92,7 @@ public async Task Build_exposes_all_ten_tools() Assert.Equal( new[] { - "okf_append_log", "okf_browse", "okf_changes_since", "okf_get_computation", + "okf_append_log", "okf_audit", "okf_browse", "okf_changes_since", "okf_get_computation", "okf_graph", "okf_read_concept", "okf_regenerate_indexes", "okf_search", "okf_validate_bundle", "okf_write_concept", }, @@ -122,7 +122,7 @@ public async Task Build_readOnly_omits_the_three_write_tools() var names = (await client.ListToolsAsync()).Select(t => t.Name).ToHashSet(); - Assert.Equal(7, names.Count); + Assert.Equal(8, names.Count); Assert.DoesNotContain("okf_write_concept", names); Assert.DoesNotContain("okf_append_log", names); Assert.DoesNotContain("okf_regenerate_indexes", names); @@ -130,6 +130,8 @@ public async Task Build_readOnly_omits_the_three_write_tools() // okf_get_computation is read-only and needs no attestation runtime, // so it surfaces in read-only mode too -- this is deliberate. Assert.Contains("okf_get_computation", names); + // okf_audit is read-only too, so it surfaces in read-only mode. + Assert.Contains("okf_audit", names); } finally { @@ -138,7 +140,7 @@ public async Task Build_readOnly_omits_the_three_write_tools() } [Fact] - public void ConfigureServices_registers_all_ten_tools() + public void ConfigureServices_registers_all_eleven_tools() { var bundle = NewBundleDir(); try @@ -147,7 +149,7 @@ public void ConfigureServices_registers_all_ten_tools() OkfMcpHost.ConfigureServices(services, bundle, readOnly: false, version: "0.0.0"); using var provider = services.BuildServiceProvider(); var options = provider.GetRequiredService>().Value; - Assert.Equal(10, options.ToolCollection?.Count); + Assert.Equal(11, options.ToolCollection?.Count); } finally { @@ -156,7 +158,7 @@ public void ConfigureServices_registers_all_ten_tools() } [Fact] - public void ConfigureServices_readOnly_registers_seven_tools() + public void ConfigureServices_readOnly_registers_eight_tools() { var bundle = NewBundleDir(); try @@ -165,7 +167,7 @@ public void ConfigureServices_readOnly_registers_seven_tools() OkfMcpHost.ConfigureServices(services, bundle, readOnly: true, version: "0.0.0"); using var provider = services.BuildServiceProvider(); var options = provider.GetRequiredService>().Value; - Assert.Equal(7, options.ToolCollection?.Count); + Assert.Equal(8, options.ToolCollection?.Count); } finally { diff --git a/tests/fixtures/README.md b/tests/fixtures/README.md index 50b7517..2591ba7 100644 --- a/tests/fixtures/README.md +++ b/tests/fixtures/README.md @@ -192,3 +192,12 @@ them a re-capture from the (removed) Rust binary: (same diagnostic construction, different caught exception type), so the risk of it being wrong is low, but it remains unexercised by an automated test. + +## `okf audit` goldens (2026-08-21) + +- `golden/audit-v02.out`, `golden/audit-v02.json` — output of + `okf audit tests/fixtures/okf_v02 --as-of 2099-06-01` (and its `--json` + form). **Hand-authored**, verified against the spec text (§5.3 trust tiers, + §5.4 statuses, §5.5 staleness) rather than captured from the reference CLI: + `audit` is an OKF4net verb with no upstream counterpart. The `--as-of` date + is pinned so the output cannot drift with the calendar. diff --git a/tests/fixtures/golden/audit-v02.json b/tests/fixtures/golden/audit-v02.json new file mode 100644 index 0000000..23303ad --- /dev/null +++ b/tests/fixtures/golden/audit-v02.json @@ -0,0 +1 @@ +{"bundle":"tests/fixtures/okf_v02","asOf":"2099-06-01","conceptCount":2,"query":{"stale":true,"trust":null,"status":null,"type":null},"trust":{"humanReviewed":1,"machineConfirmed":0,"unverified":1},"status":{"draft":0,"stable":2,"deprecated":0},"staleCount":1,"findings":[{"conceptId":"metrics/dau","path":"tests/fixtures/okf_v02/metrics/dau.md","type":"Metric","title":"Daily Active Users","trust":"human-reviewed","status":"stable","staleAfter":"2099-01-01","stale":true}]} diff --git a/tests/fixtures/golden/audit-v02.out b/tests/fixtures/golden/audit-v02.out new file mode 100644 index 0000000..a0a7486 --- /dev/null +++ b/tests/fixtures/golden/audit-v02.out @@ -0,0 +1,18 @@ +bundle: tests/fixtures/okf_v02 +as of: 2099-06-01 +concepts: 2 + +trust: + 1 human-reviewed + 0 machine-confirmed + 1 unverified + +status: + 0 draft + 2 stable + 0 deprecated + +stale: 1 of 2 past stale_after + +needs attention (1): + metrics/dau stale 2099-01-01 human-reviewed stable