From 5b9651d53e2311602d26b5aded9c7a908c3a9ff2 Mon Sep 17 00:00:00 2001 From: BradleyDB <23158057+BradleyDB@users.noreply.github.com> Date: Tue, 15 Sep 2026 08:49:14 -0700 Subject: [PATCH 01/10] report truth (Session C1): byDepth / domainCounts / changeDetection / docPathsUnknown on report; mark refuses a pathless documented mark; reconcile-docs backfills doc_path (F-455, F-454, F-451, F-459; plugin 0.40.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit manifest.mjs report now carries the T-2 v3 GsReport contract: byDepth {full, metadata, listOnly, unrecorded} over documented entries in total and per domain under domains., domainCounts {indexed, withAssets, empty}, per-domain changeDetection (date | none | unrecorded) + datelessEntries, and docPathsUnknown; describeStateOf is the one helper the stub banner and the report share. mark --status documented refuses an entry with no recorded doc_path when none is passed; remove reports docPathsUnknown beside docPaths; the new reconcile-docs verb backfills doc_path through doc-lib's docNameMatcher (the claimer's read-side twin, sharing readDocStems). Setup Phase 4 relay, Phase 5 close, --deep relay, Phase 6 precondition, final report and the mechanics §4 shapes quote ONE quiescent report; refresh's report ends with a Not-checked-for-change block. Fixtures: manifest-ops c1 section (35 arms), doc-lib matcher arms (14), contract pin (T-2 v3), relationships-build marks pass --doc-path. Bus: F-455/F-454/F-451/F-459 FIXED with Class/Judge/Sibling sweep; plan As-shipped notes; C1-V prompt refreshed. --- dev/FEEDBACK.md | 24 +- .../gs-superadmin/.claude-plugin/plugin.json | 2 +- plugins/gs-superadmin/CHANGELOG.md | 36 +++ plugins/gs-superadmin/MAINTAINERS.md | 2 +- plugins/gs-superadmin/scripts/doc-lib.mjs | 94 ++++-- plugins/gs-superadmin/scripts/manifest.mjs | 305 +++++++++++++++++- plugins/gs-superadmin/skills/refresh/SKILL.md | 19 ++ plugins/gs-superadmin/skills/setup/SKILL.md | 69 +++- .../setup/references/document-mechanics.md | 17 +- .../test/contract-conformance.mjs | 32 ++ .../gs-superadmin/test/doc-lib-fixtures.mjs | 71 +++- plugins/gs-superadmin/test/manifest-ops.mjs | 162 +++++++++- .../test/relationships-build.mjs | 12 +- 13 files changed, 776 insertions(+), 69 deletions(-) diff --git a/dev/FEEDBACK.md b/dev/FEEDBACK.md index 982326c..08c6a08 100644 --- a/dev/FEEDBACK.md +++ b/dev/FEEDBACK.md @@ -325,7 +325,7 @@ What: three facts are stored at a scope that does not match what they describe. (c) Scope limits are absent from the manifest entirely: `domains_indexed["journey-email-templates"]` carries the same six fields as a fully-enumerable domain, so any consumer reading the manifest sees the count as complete. The limitation exists only as prose in `skills/setup/references/index-scope-notes.md`. Expected: one decision about where each kind of fact lives — per-tenant, per-workspace, or per-CLI-version — with readers deriving from that home rather than three independent placements. -## F-451 — OPEN +## F-451 — FIXED Reported: 2026-09-10 (tester — first live multi-tenant setup) Severity: normal — silently removes assets from change detection; the KB looks current and is not What: on the sandbox tenant, 557 of 1,180 `journey-email-templates` entries carry no `modified_date` and 560 carry no `name` — the entries registered by the list-invisible recovery path. /refresh detects change by comparing dates, so those entries have nothing to compare and are permanently invisible to staleness: documented once in July, never flagged again. The missing `name` means inventory-side name lookups return null for them. @@ -338,6 +338,10 @@ The reach is wider than the recovery path, and this is the part that needs answe The whole `report` domain is dateless on both tenants. The manifest supports an explicit-none `dateField`, so this may be a handled state with a documented fallback — or it may mean the entire report inventory sits outside change detection. Expected: either a documented fallback /refresh applies to dateless entries, or an honest statement in the refresh report naming what it could not check — never silence. +Fix: 2026-09-15 (builder, Session C1 / RP-3; ruled 2026-09-11, Bradley: honest statement now, fingerprint-based staleness is #12) — `report` emits per domain (`domains.`) `changeDetection: "date" | "none" | "unrecorded"` and `datelessEntries`: `date` = a modified-date field is recorded on the stamp; `none` = recorded-none (`dateField: null`) — the whole domain sits outside change detection; `unrecorded` = a legacy stamp with no `dateField` key, a bare-string stamp, or no stamp at all (T-2's tri-state, never read as "none"); `datelessEntries` counts entries with no stored `modified_date` whatever the state, which under `date` are exactly the rows the comparison can never flag (the recovery-path registrations). The three measured shapes each have a fixture domain (recorded-none; a recorded field under which every row is null; a healthy dated domain) plus the legacy no-key stamp. Prose: refresh step 1 keeps the report's `domains` block and step 4's report ends with a `Not checked for change this run:` block — one line per `none` domain (whole domain outside detection), per `unrecorded` domain (field recorded this run, detection starts next refresh), and per `date` domain with `datelessEntries > 0` (n of t rows carry no date) — or the single word `none`, never omitted. "Unchanged" now says what it could see. Class: the refresh report's "Unchanged: Z" asserted "no change" from a date comparison for rows that carry no date to compare. Measured locally over both real manifests (read-only), reproducing the bus counts: prod `report` domain 1754 dateless under `none`, five recorded-none domains totalling 2362 of 5076; the sandbox `journey-email-templates` 557 dateless under a recorded field (`date`), its `report` domain 1696 under `none`, and its `report-objects` legacy stamp reads `unrecorded` (588 dateless, all pending). Rides 0.40.0. +Class: outcome-from-proxy +Judge: the tester's `report` over both real manifests reproducing those counts (prod report 1754 under `none`; sandbox templates 557 under `date`), and the refresh walk's report carrying the `Not checked for change this run:` block naming them — a refresh report that ends at "Unchanged" with those domains unnamed REOPENS this entry. +Sibling sweep: outcome-from-proxy recipe, shared with F-455's line above — the one sibling site NAMED and not closed is `upsert-batch`'s `unchanged` count, which folds dateless rows in; the refresh report now states the same fact from `report`'s per-domain `datelessEntries`, so nothing is silent, and a qualified upsert field is banked for the next round rather than changed without a ruling. Cost: minutes. ## F-452 — OPEN Reported: 2026-09-10 (tester — first live multi-tenant setup) @@ -354,13 +358,17 @@ Expected: wherever the skills instruct building a list of commands to run, the l Amended 2026-09-10 (tester, same round) — the What above is WRONG on its central claim, and the correction changes the finding. The remedy does not "appear nowhere in the skills": `templates/operating-model.md:135` states it directly — "**Spell subcommands literally**: the mutation guard checks command *text*, so a subcommand hidden in a shell variable (`gs-admin --json $c` in a loop) can't be verified and triggers approval prompts even for reads" — anticipating this exact case, including that it fires on reads. The session that hit the refusal had read that file in full roughly forty minutes earlier, then wrote the variable-built loop anyway. So this is not a documentation gap, and a second copy of the rule would fix nothing. It is a rung-3 statement (AGENTS.md's shared-prose table: a session-global doc every session reads) that did not fire at the moment of composition. That table's own remedy is to move the rule down a rung: the guard already refuses at exactly the right instant and is the only thing present at that instant, but its refusal message does not name the remedy. Stating "spell the domain/group/command words literally; variables belong in flag values and paths" in the refusal turns a correct-but-opaque block into a self-resolving one, at rung 1. Severity unchanged; what changes is the target — the guard's message, not the docs. -## F-454 — OPEN +## F-454 — FIXED Reported: 2026-09-10 (tester — first live multi-tenant setup) Severity: polish — a reporting discrepancy, no data effect observed What: the Phase 4 relay reported "4,487 assets across 16 domains" while the manifest held 17 `domains_indexed` at that moment, and the inventory carried 17 domains with at least one asset (18 after a further adoption later in the same run). Unexplained — possibly an empty-adopted domain counted differently by `report` than by `domains_indexed`, which is exactly the distinction the empty stamp exists to preserve. Expected: the relayed domain count and the manifest's domain count agree, or the report states which one it is counting. +Fix: 2026-09-15 (builder, Session C1 / RP-2) — `report` emits `domainCounts: { indexed, withAssets, empty }` (stamps; domains holding entries; `emptyDomains.length` — the three reconcile unless an unstamped `--partial` registration holds entries, which the new `domains..stamped: false` names), and the Phase 4 relay quotes `domainCounts.indexed` and names every `emptyDomains` entry instead of counting `byDomain` keys — which is exactly the discrepancy: a key count drops every listed-but-empty domain, and the tester's 16 vs 17 was one empty domain. Class: the relayed "M domains" asserted the manifest's domain count from a fold over entries, a signal that describes a different unit (domains WITH entries). Measured locally over the sandbox manifest (read-only): 18 indexed / 17 withAssets / 1 empty, the empty one being its surveys domain. Fixture: manifest-ops "c1" — 5/4/1 with the empty domain named, then 5/5/1 with `stamped: false` after a partial registration. Rides 0.40.0. +Class: outcome-from-proxy +Judge: the tester's `report` over the sandbox manifest reading 18 / 17 / 1 with the surveys domain in `emptyDomains`, and the setup walk's Phase 4 relay quoting `domainCounts.indexed`. +Sibling sweep: batched — polish; swept with F-455's recipe above (the relay's `byDomain` key count was the only site deriving a domain count from entries; refresh's step 3 reads per-domain counts from `byDomain` rows for the domain it just upserted, a same-unit read). -## F-455 — OPEN +## F-455 — FIXED Reported: 2026-09-10 (tester — first live multi-tenant setup, during a shallow crawl with per-domain deep ingests) Severity: normal — no stored data is wrong and no stub misrepresents itself; what is wrong is the completion signal the user is asked to act on, and downstream decisions (rebuilding relationship maps, trusting a deps-report answer) rest on it What: `manifest.mjs report` is the deterministic account of KB state, and its schema carries no depth at all. A metadata stub and a full describe are both counted as `documented`, in `byStatus` and in every `byDomain` row, so the report cannot distinguish a documented domain from an ingested one: @@ -374,6 +382,10 @@ Consequence observed live: after one domain's deep ingest completed, the user wa Note the same conflation is deliberate and correct elsewhere: Phase 6's precondition counts stubs as documented on purpose, because a stub never blocks synthesis. The defect is that the one mechanical statement of state offers no depth anywhere, so a human-facing completion claim cannot be grounded in it. Second, smaller point found while measuring this: the manifest is rewritten continuously during a run, so a tally taken mid-ingest is a snapshot of a moving target and will disagree with one taken a minute later. A completeness check is only meaningful at a quiescent point. (The reporter made exactly this mistake while drafting the entry — a mid-run sample showed one asset short of a domain's full ingest, which the next read showed complete.) Expected: depth is part of the mechanical account, not something a reader reconstructs. `report` (or a sibling verb) carries a per-domain full/stub/list-only breakdown derived from the `depth` the manifest already stores, with list-only domains reported complete by definition rather than as stubs, so the three states stay distinguishable. And every place a skill states that documentation is complete, or that a domain is fully ingested, derives that statement from the script's output rather than from its own narrative — one invocation, quoted, at a point where nothing is still writing. +Fix: 2026-09-15 (builder, Session C1 / RP-1) — `manifest.mjs report` carries `byDepth` = { full, metadata, listOnly, unrecorded } over DOCUMENTED entries, in total and per domain under a new `domains.` row (ruling, Bradley 2026-09-15: four states, per-domain block — `unrecorded` is NOT folded into `metadata`, because a stub under a domain with no describe recording is the F-346 false-complete class, and nothing new goes inside a `byDomain` row because describe-batch's progress probe sums those). A metadata stub is bucketed by its DOMAIN's recorded describe state through ONE helper (`describeStateOf`) the stub banner now shares, so the report and the docs on disk say the same three things; a documented entry with no depth at all (a pre-depth-field doc, never a stub) counts as full — the stub verb's own never-downgrade rule. Class: the report field `documented` asserted completeness from a status that describes doc EXISTENCE, not depth — every skill sentence about completeness was derived from that proxy or from the run's own narrative. Prose: setup Phase 5 closes on ONE quoted `report` at a quiescent point (`byDepth` + `domains..byDepth`; "fully ingested" = `metadata: 0` and `unrecorded: 0` with no pending/stale — never a tally taken mid-batch), the `--deep` relay quotes the same row, Phase 6's precondition reads that report's `byStatus` (stubs still count as documented, deliberately), the final report gains a `Depth:` line and a `Not complete:` line, and document-mechanics §4's shapes derive from it (new "Completeness unknown" line for unrecorded stubs). Contract: report's output is now T-2's `GsReport` (manifest.mjs header, v3, additive) pinned by test/contract-conformance.mjs (exact key sets, byDomain numbers-only). Measured locally over both real manifests (read-only): prod reads 6 domains listOnly (722 entries) and 0 metadata / 0 unrecorded — every `none` domain listOnly, none as metadata; the sandbox reads 5 domains unrecorded (its five legacy stamps) and 0 listOnly. Fixtures: test/manifest-ops.mjs "c1" section (35 arms over a manifest carrying all four states + a pre-depth doc; 27 red against the pre-fix script, the rest rig setup and controls). Version 0.40.0. Pick up: /reload-plugins. +Class: outcome-from-proxy +Judge: (1) the T-2 v3 conformance pin (key sets re-derived from the typedef, never imported); (2) independent of the fixer: the tester runs `report` over BOTH real manifests in the consumer workspace and holds the output to the claims above — every list-only domain of prod reads listOnly, none reads metadata; the sandbox's five legacy stamps read unrecorded — then walks setup once and reads the Phase 5 close / final report quoting `byDepth` from one invocation. A skill relay that states completeness from anything but that quoted report REOPENS this entry. +Sibling sweep: outcome-from-proxy recipe — one read per outcome-writing site. manifest.mjs summary fields: `report.byStatus.documented` (this entry — qualified by `byDepth`), `remove.docPaths` (F-459 — qualified by `docPathsUnknown`), `upsert-batch.unchanged` (NAMED, not closed: it counts rows whose stored and incoming dates are both null as "unchanged" — the same fact F-451's `datelessEntries` now states one layer down in `report`, which the refresh report names; a qualified field on the upsert summary is the one remaining site, banked for C2 / #12 rather than changed without a ruling), `stub.stubbed` / `mark.status` / `next.count` / `crawl` / `reconcile-docs` (signal = the write this verb just made, unit = the same entry: no mismatch). Skills: `grep -rn -i "fully ingested\|deep crawl\|is complete\|up to date\|fully documented" skills templates` — every remaining hit is either the crawl-mode question text (a promise about the mechanism, not a state claim) or a sentence that now quotes the report. Cost: minutes. ## F-456 — VERIFIED Reported: 2026-09-10 (tester — first live multi-tenant setup, deep-ingesting a connectors domain) @@ -448,8 +460,12 @@ Side observation (not this finding): each failed spawn in step 6, and the auth d Verified: 2026-09-14 @ hb-20260914-01 (tester, Session B-V) — a token death past the half-life aborted with reason auth, failures empty and zero auth-failed marks; the resume after login documented the in-flight entry; #13's limit stopped a mismatched describe at exactly 5 spawns. Blind spots (tester, hb-20260914-01): the rig was a timed start, not an oversized batch from a fresh token — the deadline and outcome are the same, the run length (80 entries, not ~600) is not; one death observed, on one domain (report, raw doc-mode) — the designer lane's mid-drilldown auth death rests on the fixtures only; the three sibling auth literals (no stored token; no refresh token; refresh failed with a status) were not produced live; the resume was read mid-run at 105 of 200, and the rest of that run is not part of this verdict; step 6 ran on a substitute domain. -## F-459 — OPEN +## F-459 — FIXED Reported: 2026-09-11 (tester, Session A-V — surfaced while running the F-449 sibling arm). Section written 2026-09-14 by the builder from the round's own record: the A-V close-out comment claimed the number and dev/VALIDATION.md's F-449 sibling section describes the observation, but no section was ever appended, and Session C1's kickoff expects one. Severity: normal — an entry marked documented with no recorded doc_path is invisible to every doc_path consumer; a cleanup list that reads 0 for it under-reports silently What: `manifest.mjs remove` over a July-crawled sandbox domain (the F-449 sibling arm's step 2, 69 entries) reported `docPaths` 0 while the stub docs existed on disk — the entries carry status `documented` but no `doc_path` (docs written before path recording existed, then deleted by folder). Measured on the sandbox: 25 such entries across three July-crawled domains; zero on prod. `manifest.mjs mark --status documented` accepts a mark with no `doc_path` today (a stub or describe-batch always passes one; only a hand-run mark omits it), so the state is reachable by hand as well as by legacy. Expected: (1) `mark --status documented` refuses when the entry has no recorded `doc_path` and none is passed; (2) `report` and `remove` count documented entries carrying no `doc_path` and say so by number (`docPathsUnknown`), so `remove`'s `docPaths` can never read as "no docs" when the truth is "unknown"; (3) a backfill verb (`manifest.mjs reconcile-docs --domain `) that matches `//*.md` to entries through the doc-name claimer's naming, records `doc_path` for the matches and reports the unmatched on both sides. Scheduled as RP-5 (Session C1) in the round plan; the sandbox's three domains are the live judge. +Fix: 2026-09-15 (builder, Session C1 / RP-5; ruling, Bradley 2026-09-15: refuse in mark, count in report and remove, backfill as its OWN verb — report stays read-only by contract) — all three parts as Expected. (1) `mark --status documented` refuses, all-or-nothing and naming the remedy, when an entry has no recorded `doc_path` and none is passed; measured before shipping: every doc writer passes `--doc-path` on the mark after its write (stub and describe-batch by construction, the manual path by its reference), and describe-batch's `--if-changed` skip re-marks WITHOUT a path only on an entry that already carries one (a `typeof entry.doc_path === "string"` conjunct guards that branch), so the only mark refused is the hand-run one — the fixture suites' own hand-run marks were that route and now pass the path. (2) `report` carries `docPathsUnknown` per domain and in total; `remove` carries it beside `docPaths`. (3) `reconcile-docs --domain [--dir ] [--out ] [--dry-run]` matches `//*.md` to the domain's entries through doc-lib's new `docNameMatcher` — the claimer's READ-side twin: `docBaseName`, then the `-dup` chain a writer appends while a stem is claimed in-run or a different spelling of it sits on disk — after pre-claiming every RECORDED path (a recorded doc is never re-attributed); records `doc_path` in the spelling stub writes (folder as passed, forward slashes) and reports `recorded`, `alreadyRecorded`, `recordedMissing` (recorded path, file gone — never rewritten: a deleted doc is the regeneration signal), `unmatchedDocumented` (documented, no doc on disk) and `orphanFiles` (docs no entry claims), samples on stdout and full lists under `--out`; writes the manifest once, only when a path was recorded, and never touches a doc file. Review-gate note: the first cut re-spelled the `-dup` loop inside manifest.mjs and check-doc-drift check 9 refused it — the grammar has one home, so the matcher moved into doc-lib beside the claimer (shared disk snapshot), pinned by 14 arms in test/doc-lib-fixtures.mjs including the hop bound. Legacy count re-measured locally on the sandbox (read-only) before the fixture was written: 20 documented entries with no `doc_path` today across the same three July-crawled domains (3 + 14 + 3), down from the 25 logged — the B-V round's #13 arm re-marked five chain entries with a path; prod 0. Class: `remove.docPaths` asserted "no docs to clean" from a field only the newer writers record. Fixtures: manifest-ops "c1" (the refusal incl. the all-or-nothing keys-file arm and three controls; remove's count; reconcile over a scratch folder — matched, the Co/co collision pair via `co-dup`, recorded-missing, unmatched, two orphans, dry-run, `--out`, idempotent re-run, missing folder). Version 0.40.0. MAINTAINERS.md's manifest row names the verb. +Class: outcome-from-proxy +Judge: independent of the fixer, banked in dev/VALIDATION.md (F-459 section, Session C1): the tester reads `report`'s `docPathsUnknown` on the sandbox (expected 20 across three domains), runs `reconcile-docs --dry-run` then `reconcile-docs` over each of the three (the ONE manifest write this round asks of the tester), reads `docPathsUnknown` again (expected 0, every orphan named), and re-runs the F-449 sibling arm's step 2 SHAPE as a read — `report`'s row for one of those domains showing a complete `docPaths` picture (no `docPathsUnknown`), no `remove` needed. A `docPathsUnknown` that does not reach 0 without an explanation in `unmatchedDocumented` REOPENS this entry. +Sibling sweep: outcome-from-proxy recipe, shared with F-455's line above (`remove.docPaths` is this entry's site; `report.byStatus.documented` F-455's; `upsert-batch.unchanged` named there). consumer-parity check on the naming grammar: `git grep -n '"-dup"' -- plugins build` — doc-lib.mjs only (claimer + matcher); check-doc-drift check 9 sanctions no other file, and the first cut's second copy in manifest.mjs is what that check refused. Cost: seconds. diff --git a/plugins/gs-superadmin/.claude-plugin/plugin.json b/plugins/gs-superadmin/.claude-plugin/plugin.json index f9c8731..63150eb 100644 --- a/plugins/gs-superadmin/.claude-plugin/plugin.json +++ b/plugins/gs-superadmin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "gs-superadmin", - "version": "0.39.0", + "version": "0.40.0", "description": "Persistent Gainsight Admin CLI workspace — bootstrap, index, and operate on a Gainsight tenant via gs-admin.", "author": { "name": "BradleyDB" diff --git a/plugins/gs-superadmin/CHANGELOG.md b/plugins/gs-superadmin/CHANGELOG.md index bfbb55e..8d56083 100644 --- a/plugins/gs-superadmin/CHANGELOG.md +++ b/plugins/gs-superadmin/CHANGELOG.md @@ -5,6 +5,42 @@ marketplace doesn't pin versions — users get main — so entries describe what user who updates, not internal refactors. Entries before 0.8.0 were reconstructed from git history when this file was introduced. +## 0.40.0 — 2026-09-15 + +The report is the account of KB state, and it now says how deep that state goes. What a +user gets by updating: `manifest.mjs report` carries `byDepth` — `full`, `metadata` +(stubs awaiting `--deep`), `listOnly` (stubs of a domain recorded `none`: complete by +definition) and `unrecorded` (stubs of a domain with no describe recording: completeness +unknown, never assumed) — in total and per domain under a new `domains.` row, so +"documented" can no longer be relayed as "complete" (F-455: the user was told a deep crawl +had finished while eight domains were entirely stubs, and nothing in the report could +contradict it). The setup skill's Phase 5 close, Phase 6 precondition, `--deep` relay and +final report, and the budget-report shapes, all quote ONE report invocation taken at a +quiescent point instead of narrating. + +- `domainCounts: { indexed, withAssets, empty }` (F-454): the Phase 4 relay quotes + `indexed` and names every `emptyDomains` entry, instead of counting `byDomain` keys — + a count that silently dropped every listed-but-empty domain. +- Per domain, `changeDetection: "date" | "none" | "unrecorded"` and `datelessEntries` + (F-451): the refresh report ends with a `Not checked for change this run:` block naming + every domain with no date field and every count of dateless rows under a recorded one + (on one production workspace nearly half the inventory sits outside change detection; + "Unchanged" now says what it could see). Honest statement only — fingerprint-based + staleness for those rows is issue #12 territory. +- `docPathsUnknown` (F-459), in `report` (per domain and total) and beside `remove`'s + `docPaths`: documented entries with no recorded `doc_path` (docs written before path + recording existed) are counted, so a cleanup list can never read 0 as "no docs" when the + truth is "unknown". `mark --status documented` now refuses an entry that has no recorded + `doc_path` when none is passed (stub, describe-batch and the manual path always pass + one). New verb `manifest.mjs reconcile-docs --domain [--dir ] [--out ] + [--dry-run]` backfills `doc_path` from the docs on disk through the writers' own naming + (doc-lib's `docNameMatcher`, the claimer's read-side twin — same `-dup` collision chain), + and names recorded-missing, unmatched and orphan files; it writes only `doc_path`, + never a doc. This is the reconcile-on-resume step batched marks (#12) will reuse. +- Contract: report's output is now part of T-2 (`GsReport` in manifest.mjs's header, + pinned by `test/contract-conformance.mjs`); every addition is additive — `byDomain` + rows stay numbers-only because describe-batch's progress probe sums them. + ## 0.39.0 — 2026-09-14 The describe loop stops itself, and says why. What a user gets by updating: a dead token diff --git a/plugins/gs-superadmin/MAINTAINERS.md b/plugins/gs-superadmin/MAINTAINERS.md index 7245d44..42a220f 100644 --- a/plugins/gs-superadmin/MAINTAINERS.md +++ b/plugins/gs-superadmin/MAINTAINERS.md @@ -29,7 +29,7 @@ modules they import, `doc-lib.mjs` and `journal-lib.mjs`): | Script | Purpose | |--------|---------| -| `scripts/manifest.mjs` | Deterministic `_manifest.json` operations (init / upsert-batch / mark / next / stub / crawl / report / remove / exclude / block) — atomic writes, schema-checked; the skills never hand-edit manifest JSON. `upsert-batch` records each domain's id field, describe recipe, list command (`--list-command`, matched by the candidate gate), and modified-date field (`--date-field` / `--no-date-field`) in `domains_indexed` and reuses the recordings on re-runs, guarded against contradiction (`--allow-rekey` / `--allow-redate` override deliberate scheme changes); its summary warns on probable under-pagination (incoming list smaller than the domain's non-failed inventory) and on mass-stale flips — it never removes entries itself; `--partial` declares a deliberate-subset registration (gap-fill flows, F-313), which skips the under-pagination warning and leaves the domain's coverage stamp untouched. `exclude` / `block` persist the candidate gate's per-command decisions ("looked and said no" vs "could not look" — F-108/F-218), keyed by canonical catalog path; an exclusion records its evidence (F-449): exactly one of `--check ` (the candidate's own `check --out` file, whose `command` must match) or `--no-check ""`, with `--covered-by ` as the coverage claim the check's numbers must support (and must be made when they do), and `--recheck-after` on both record kinds | +| `scripts/manifest.mjs` | Deterministic `_manifest.json` operations (init / upsert-batch / mark / next / stub / crawl / report / remove / exclude / block / reconcile-docs) — atomic writes, schema-checked; the skills never hand-edit manifest JSON. `report` is the account of KB state and its output is the T-2 GsReport contract (Session C1): beside the status counts it carries `byDepth` (`full` / `metadata` = stubs awaiting `--deep` / `listOnly` = complete by definition / `unrecorded` = completeness unknown, over documented entries, per domain under `domains.` and in total — F-455), `domainCounts` (`indexed` / `withAssets` / `empty` — F-454), per-domain `changeDetection` (`date` / `none` / `unrecorded`) with `datelessEntries` (F-451), and `docPathsUnknown` (documented entries with no `doc_path`, per domain and in total — F-459); `byDomain` rows stay numbers-only because describe-batch sums them. `mark --status documented` refuses an entry with no recorded `doc_path` when none is passed (F-459), `remove` reports `docPathsUnknown` beside `docPaths`, and `reconcile-docs --domain ` backfills `doc_path` from the docs on disk through the writers' own naming (doc-lib `docBaseName` + the claimer's `-dup` chain), naming recorded-missing, unmatched and orphan files — the reconcile-on-resume step issue #12's batched marks will reuse. `upsert-batch` records each domain's id field, describe recipe, list command (`--list-command`, matched by the candidate gate), and modified-date field (`--date-field` / `--no-date-field`) in `domains_indexed` and reuses the recordings on re-runs, guarded against contradiction (`--allow-rekey` / `--allow-redate` override deliberate scheme changes); its summary warns on probable under-pagination (incoming list smaller than the domain's non-failed inventory) and on mass-stale flips — it never removes entries itself; `--partial` declares a deliberate-subset registration (gap-fill flows, F-313), which skips the under-pagination warning and leaves the domain's coverage stamp untouched. `exclude` / `block` persist the candidate gate's per-command decisions ("looked and said no" vs "could not look" — F-108/F-218), keyed by canonical catalog path; an exclusion records its evidence (F-449): exactly one of `--check ` (the candidate's own `check --out` file, whose `command` must match) or `--no-check ""`, with `--covered-by ` as the coverage claim the check's numbers must support (and must be made when they do), and `--recheck-after` on both record kinds | | `scripts/domain-candidates.mjs` | The setup Phase 4 candidate gate (F-108), read-only: `diff` derives every list-shaped tenant-wide command from the catalog and maps each to indexed / excluded / blocked / undecided against the manifest's recordings (`--require-decided` exits 1 while any candidate is undecided or a domain lacks a `listCommand` recording); `check` answers the GLOBAL "are these rows already indexed anywhere" overlap question over a captured list payload (fails loudly on unresolvable id fields, partial extractions, and — without `--allow-empty` — payloads with no items array), and with `--command "" --out ` writes its JSON as the evidence file `manifest.mjs exclude --check` reads (F-449); the diff's `excluded` rows carry `kind` (coverage / judgment / no-check / legacy), the evidence, and `recheckDue` | | `scripts/describe-batch.mjs` | Sanctioned describe→doc→mark loop for one domain: selects the batch via `manifest.mjs next`, runs the describes sequentially with `{id}`/`{name}` substituted as literal argv (no shell — pipe-bearing names are safe; resolves the CLI's JS entry, so no Windows `.cmd` traps), writes one structured doc per asset, and marks each as it lands, emitting a stderr progress line every few describes plus a `domainProgress` summary field (documented/total; on `--upgrade` runs `{ full, metadata, failed, total }`, since stubs already count as documented). `--command` defaults to the describe recipe the domain's index recorded (`manifest.mjs upsert-batch --describe-command`); an explicit `--command` wins. For the domain recorded from `jo email templates` it auto-selects its template doc-mode (`--doc-mode template`) and for the one recorded from `jo programs list` its program doc-mode (`--doc-mode program`) — the recording decides, the naming rule's `journey-email-templates` / `journey` only as the no-recording fallback (F-429; the lane table is doc-lib's `RECORDED_LANES`) — writing compact docs instead of raw-JSON docs — rendering shared with `template-doc.mjs`/`program-doc.mjs` via `scripts/doc-lib.mjs`. Records a content fingerprint of each documented payload (volatile modified/updated fields dropped), and `--if-changed` uses it as a re-document gate: an unchanged payload skips the doc write and is just re-marked documented (`skippedUnchanged` in the summary), so a platform event that bumps modified dates en masse doesn't rewrite every doc. Enforces read-only fail-closed against the catalog — mutating or unknown commands are refused, recorded or passed. For `data-designer` it auto-selects its designer doc-mode (`--doc-mode designer`, GP-B5 W9): three describe levels per template — the template, one `--task-id` drilldown per task, one `--field` detail per show-field label (labels derived from the drilldown's tables by doc-lib's `designerTaskFieldLabels`, aggregation suffix stripped per its enumerated vocabulary; both flag spellings verified against the catalog before any spawn) — composed into ONE doc by doc-lib's `renderDesignerDoc` (the template payload with the drilldowns under `_kb`, T-3 v5). Cost is bounded per invocation by `--spawn-budget` (default 30 calls) and the doc is rewritten after every call, so a run cut off mid-template leaves an INCOMPLETE doc that the next invocation resumes from (`designerDocProgress`); the entry is marked documented only when every item is ok, failed (doc kept, doc_path recorded) when any item failed after all were attempted, and `--if-changed` skips only a COMPLETE composite of unchanged content. Within-run abort (issue #13, F-458): five consecutive `failed` marks stop the domain with the marks standing, and a FAILED spawn carrying the CLI's re-login sentence (doc-lib `isAuthDeath`, version-stamped) stops it on the first sighting with nothing marked (a designer entry keeps its composite with the item pending); either is reported as one ADDITIVE summary field `aborted: { reason: "consecutive-failures" | "auth", after, lastError }` — absent on a run that did not stop early; `budgetExhausted`, `moreRemaining` and `failures` keep their shapes (the setup skill branches on them: contract) | | `scripts/capture.mjs` | The shipped capture helper (GP-B5 DS-17): runs one read-only `gs-admin` command (argv array, no shell — the CLI's JS entry resolved the same way as `describe-batch.mjs`) and BYTE-copies its stdout to `--out` as UTF-8 **without a BOM** via temp+rename (no decode — the child's bytes are preserved exactly) — the redirect-encoding rule the capturing skills used to restate as prose, carried by construction; `--normalize ` re-encodes an existing capture in place, tolerant of what PS 5.1 redirects write (UTF-16LE/BE with BOM, BOM'd UTF-8) and REFUSING, file untouched, anything whose decode would be lossy (BOM-less non-UTF-8 console-codepage captures, lone-surrogate UTF-16). Spawns the CLI itself, so the mutation guard never sees the embedded command: it gates fail-closed through doc-lib's shared `assertPlainGsAdminCommand` + `assertReadOnlyCommand` (non-gs-admin / shell-operator / unknown / catalog-mutating / ask-override / read-shape / write-endpoint all refused) with a capture-shaped policy built on doc-lib's shared `isDescribeRead` (describe-shaped by catalog `actionKey` or by the path's trailing word, plus the `READ_VERB_EXACT` per-item reads — F-456) and list shapes by verb, actionKey, or summary (both of `domain-candidates.mjs`'s prongs), and `dm deps check`. A failed child propagates its exit code and writes nothing — an earlier capture at `--out` is left intact and named as earlier. `--wait` (GP-B5 DS-26) bounded-polls the async `dm deps check` scan: re-run every `--wait-interval` s (default 15, floor 1 — each attempt is a real tenant request) up to `--wait-timeout` s (default 120, honored in full), writing only a payload whose `data.progressStatus.overallStatus` is COMPLETED — the envelope requirement is exactly the readers' rule, differential-locked against `parseLiveDepsAreas` in the suite; on timeout NOTHING is written and the non-zero exit names the elapsed time and last status ("not ready after N s", never a confident zero). `--paginate` (GP-B5 DS-30) owns the list-sweep pagination doctrine: `--page-flag ` (the command's paging flag, appended per round — never guessed; `none` = one reconciled fetch), a `{page}` placeholder in `--out` (one file per page), `--max-pages` safety stop; each page's envelope is scanned with doc-lib's `scanListEnvelope` (er-count's no-descend traversal + parse-don't-validate totals over the measured envelope variance) and the summary is the honesty report — pages fetched vs parsed, rows counted, every total with its path, and a verdict, with only `reconciled`/`unverified` exiting 0 (`mismatch`/`suspect`/`total-conflict`/`failed-page`/`safety-stop` exit non-zero; an unparseable page is a failed sweep page kept as evidence, never 0 entries). `build/check-doc-drift.mjs` check 14 enforces that skill captures route through this helper | diff --git a/plugins/gs-superadmin/scripts/doc-lib.mjs b/plugins/gs-superadmin/scripts/doc-lib.mjs index cfd8470..fc7d1eb 100644 --- a/plugins/gs-superadmin/scripts/doc-lib.mjs +++ b/plugins/gs-superadmin/scripts/doc-lib.mjs @@ -294,29 +294,37 @@ export function docBaseName(id) { // prefix collision between two clean ids. Narrow, but silent when it lands: the // exact-case disk match reads the second id's write as the first id's own doc // and reuses the name. +// The doc stems in a folder — lower-cased stem -> the exact spellings on disk. +// ONE snapshot reader for the claimer (write side) and the matcher (read side, +// F-459): both see the same files the same way, so a name the matcher +// attributes is a name the claimer would have chosen. +/** @param {string} dir @returns {Map>} */ +function readDocStems(dir) { + /** @type {Map>} */ + const disk = new Map(); + let names = []; + try { + names = readdirSync(dir); + } catch { + /* dir not created yet — nothing on disk to respect */ + } + for (const n of names) { + if (!/\.md$/i.test(n)) continue; + const stem = n.slice(0, -3); + const lower = stem.toLowerCase(); + if (!disk.has(lower)) disk.set(lower, new Set()); + // The line above guarantees the key; the cast states that to the + // strictNullChecks ratchet (Map.get is `| undefined` by signature). + /** @type {Set} */ (disk.get(lower)).add(stem); + } + return disk; +} export function docNameClaimer(dir) { const claimedLower = new Set(); /** @type {Map> | null} */ let disk = null; // lower-cased stem -> Set of exact on-disk stems return (id) => { - if (disk === null) { - disk = new Map(); - let names = []; - try { - names = readdirSync(dir); - } catch { - /* dir not created yet — nothing on disk to respect */ - } - for (const n of names) { - if (!/\.md$/i.test(n)) continue; - const stem = n.slice(0, -3); - const lower = stem.toLowerCase(); - if (!disk.has(lower)) disk.set(lower, new Set()); - // The line above guarantees the key; the cast states that to the - // strictNullChecks ratchet (Map.get is `| undefined` by signature). - /** @type {Set} */ (disk.get(lower)).add(stem); - } - } + if (disk === null) disk = readDocStems(dir); let base = docBaseName(id); for (;;) { const spellings = disk.get(base.toLowerCase()); @@ -329,6 +337,56 @@ export function docNameClaimer(dir) { }; } +// The claimer's READ-side twin (F-459, manifest.mjs reconcile-docs): which file +// on disk did a writer running the claimer's rule give this id? It replays the +// same grammar — docBaseName, then "-dup" while the stem is claimed in-run or a +// DIFFERENT spelling of it sits on disk — and answers with the exact stem when +// that file exists and no earlier id took it, or null when the writer's choice +// is a name no file carries (no doc for this id). Callers claim() the stems of +// RECORDED paths first, so a replay never re-attributes a doc whose owner is a +// fact; unclaimed() is what remains — the orphans. Same residual as the +// claimer (two ids whose docBaseName output is byte-identical resolve in call +// order), and the same 64-hop bound a pathological folder could otherwise +// defeat. The suffix loop is written once more here on purpose: it is the +// grammar's second, read-side home, in the file check-doc-drift check 9 +// sanctions for it — never copy it into a script. +/** + * @param {string} dir + * @returns {{ match: (id: string) => string|null, claim: (stem: string) => void, unclaimed: () => string[] }} + */ +export function docNameMatcher(dir) { + const disk = readDocStems(dir); + /** @type {Set} */ + const claimed = new Set(); + /** @type {Set} */ + const claimedLower = new Set(); + /** @param {string} stem */ + const claim = (stem) => { + claimed.add(stem); + claimedLower.add(stem.toLowerCase()); + }; + return { + claim, + match(id) { + let base = docBaseName(id); + for (let hops = 0; hops < 64; hops++) { + const lower = base.toLowerCase(); + const spellings = disk.get(lower); + if (!claimedLower.has(lower)) { + if (spellings !== undefined && spellings.has(base)) { + claim(base); + return base; + } + if (spellings === undefined) return null; // the writer's first free choice — and no file carries it + } + base += "-dup"; + } + return null; + }, + unclaimed: () => [...disk.values()].flatMap((s) => [...s]).filter((stem) => !claimed.has(stem)).sort(), + }; +} + // Entity decode (amp last, so &lt; stays a literal `<` only after one pass) export const decode = (s) => String(s) diff --git a/plugins/gs-superadmin/scripts/manifest.mjs b/plugins/gs-superadmin/scripts/manifest.mjs index a4ec370..49eba19 100644 --- a/plugins/gs-superadmin/scripts/manifest.mjs +++ b/plugins/gs-superadmin/scripts/manifest.mjs @@ -106,7 +106,12 @@ // remove --key | --keys-file // delete inventory entries (e.g. rekey orphans); reports the // removed entries' doc_path values so stale docs can be -// cleaned up — the script never deletes doc files itself +// cleaned up — the script never deletes doc files itself — +// and `docPathsUnknown`, the removed entries that were +// documented with NO doc_path recorded (F-459: legacy docs +// written before path recording existed), so an empty +// `docPaths` can never read as "no docs" when the truth is +// "unknown"; reconcile-docs (below) records those paths // | --domain [--allow-populated] // de-register a domains_indexed coverage stamp (F-333: the // sanctioned exit for a phantom stamp — a typo'd --domain @@ -132,6 +137,14 @@ // documented stamps last_verified; depth keeps the entry's // existing depth unless --depth is given (else defaults full); // --doc-path records where the doc landed (remove reports it); +// a documented mark is REFUSED when the entry has no recorded +// doc_path and none is passed (F-459: every doc writer — +// stub, describe-batch, the manual path — knows where its doc +// landed, and an entry documented with nowhere recorded is +// invisible to every doc_path consumer; with --keys-file the +// whole batch is refused, nothing written). An entry that +// already carries a doc_path may be re-marked without one +// (describe-batch's --if-changed skip does exactly that); // --fingerprint records the content fingerprint of the // documented describe payload (doc-lib.mjs // canonicalFingerprint — describe-batch.mjs's --if-changed @@ -220,6 +233,55 @@ // cannot be wrong about coverage). `touch-refresh` is gone — // an unknown verb; last_refresh stays in the manifest as a // legacy field nothing derives from. +// Depth and change-detection truth (F-455, F-454, F-451, +// F-459 — Session C1; the output shape is the GsReport +// contract below): `byDepth` = { full, metadata, listOnly, +// unrecorded } over DOCUMENTED entries, where a metadata +// stub counts by its domain's recorded describe state — +// `none` → listOnly (complete by definition), a template → +// metadata (awaiting --deep), no recording → unrecorded +// (completeness unknown, never assumed — the F-346 class) — +// and a documented entry with no depth at all (a doc written +// before the field existed; never a stub) counts as full, the +// same rule the stub verb's never-downgrade applies; pending / +// stale / failed entries appear under byStatus only, never +// under a depth. `domainCounts` = { indexed (stamps), +// withAssets (domains holding entries), empty } so a relay +// never re-derives the domain count from byDomain's keys. +// `domains.` = { stamped, describeState, byDepth, +// changeDetection ("date" = a field is recorded; "none" = +// recorded-none; "unrecorded" = legacy stamp, or no stamp), +// datelessEntries (entries with no modified_date — under a +// recorded field these are the rows change detection cannot +// see), docPathsUnknown } — one row per domain that holds +// entries or a stamp. `docPathsUnknown` at the top level is +// the tenant total. Read-only by contract: report never +// writes the manifest (the backfill is reconcile-docs). +// reconcile-docs --domain [--dir ] [--out ] +// [--dry-run] +// backfill doc_path on the domain's entries from the docs +// already on disk (F-459: legacy docs written before path +// recording existed; this is also the reconcile-on-resume +// step batched marks will need — issue #12). Scans +// (default /, the folder every writer +// uses) for *.md files and matches each entry through the +// doc writers' OWN naming — doc-lib's docBaseName plus the +// claimer's "-dup" collision chain, replayed against what is +// on disk — so the match is the name a writer would have +// chosen, never a guess. Records doc_path (the same +// /.md spelling stub writes) on every matched +// entry that has none; entries that already carry a doc_path +// are left as recorded (a recorded path whose file is gone is +// reported under recordedMissing, never rewritten — a deleted +// doc is the operator's regeneration signal). Reports, by +// count with samples and in full under --out: recorded, +// alreadyRecorded, recordedMissing, unmatchedDocumented +// (documented entries with no doc on disk — the docs are +// gone; re-document or remove, a decision, never automatic) +// and orphanFiles (docs no entry claims — a rekey or a +// removed entry left them; cleanup is the caller's). Never +// writes or deletes a doc file. --dry-run computes everything +// and writes nothing. // // Output: one JSON object on stdout. Non-zero exit + stderr message on error. // Zero dependencies — Node built-ins only. @@ -302,6 +364,63 @@ * @property {string} reason * @property {string} decidedAt * @property {string} [recheckAfter] YYYY-MM-DD + * + * ── report's output (T-2 v3, Session C1 2026-09-15 — ADDITIVE over the v2 + * shape; every key below is frozen once a skill quotes it). Readers: the + * setup skill (Phase 4 relay, Phase 5 close, Phase 6 precondition, --deep + * relay), the refresh skill (step 1 / step 4), describe-batch's progress + * probe (byDomain ONLY — it sums a row's values, so a byDomain row carries + * numbers and nothing else, ever). Pinned by test/contract-conformance.mjs. + * + * @typedef {object} GsReportDepth counts over DOCUMENTED entries + * @property {number} full a describe ran and a full doc was written + * (includes documented entries with no depth — + * pre-depth-field docs, never stubs) + * @property {number} metadata a stub under a domain with a describe + * template — incomplete, awaiting --deep + * @property {number} listOnly a stub under a domain recorded `none` — + * complete by definition + * @property {number} unrecorded a stub under a domain with NO describe + * recording — completeness unknown + * + * @typedef {object} GsReportDomain one row per domain holding entries or a stamp + * @property {boolean} stamped a domains_indexed stamp exists (false = entries + * registered into a never-listed domain, F-316) + * @property {"describable"|"list-only"|"unrecorded"} describeState the stub + * verb's three recorded states, same words + * @property {GsReportDepth} byDepth + * @property {"date"|"none"|"unrecorded"} changeDetection "date" = a date + * field is recorded; "none" = recorded-none + * (dateField null); "unrecorded" = legacy stamp + * with no dateField key, a bare-string stamp, + * or no stamp — refresh derives it once + * @property {number} datelessEntries entries with no modified_date (null or + * absent), whatever changeDetection says + * @property {number} docPathsUnknown documented entries with no doc_path + * + * @typedef {object} GsReport + * @property {true} ok + * @property {string} slug + * @property {string} baseUrl + * @property {?string} environment + * @property {number} total + * @property {?string} last_refresh + * @property {Object} byStatus + * @property {Object>} byDomain numbers only + * @property {Object} lookback + * @property {number} lookbackDefault + * @property {Object} domains_indexed + * @property {Object} domains_excluded + * @property {Object} domains_blocked + * @property {string[]} emptyDomains + * @property {GsReportDepth} byDepth tenant totals (v3) + * @property {{indexed: number, withAssets: number, empty: number}} domainCounts + * indexed = stamps; withAssets = domains holding + * entries; empty = emptyDomains.length. The three + * reconcile (withAssets + empty = indexed) unless + * an unstamped domain holds entries (v3) + * @property {number} docPathsUnknown tenant total (v3) + * @property {Object} domains keyed by domain name (v3) */ import { writeFileSync, mkdirSync, existsSync } from "node:fs"; import { dirname, join, resolve } from "node:path"; @@ -309,7 +428,7 @@ import { fileURLToPath } from "node:url"; // Doc filename for an asset id — the shared copy every doc writer imports // (sanitized ids get a short raw-id hash so distinct ids that clean to the // same base can never collide). -import { docNameClaimer, readJsonFile, writeFileAtomicSync, cmpKey, getPath as get, findItemsArray, extractIds, makeCliHelpers, STUB_MARKER, DESCRIBE_NONE, idPathHint, ZERO_RESOLVE_HEAD, findWorkspaceCatalog, makeCommandResolver } from "./doc-lib.mjs"; +import { docNameClaimer, docNameMatcher, readJsonFile, writeFileAtomicSync, cmpKey, getPath as get, findItemsArray, extractIds, makeCliHelpers, STUB_MARKER, DESCRIBE_NONE, idPathHint, ZERO_RESOLVE_HEAD, findWorkspaceCatalog, makeCommandResolver } from "./doc-lib.mjs"; const here = dirname(fileURLToPath(import.meta.url)); // The date-failure head refresh/SKILL.md's post-upgrade migration note quotes @@ -338,7 +457,7 @@ const { opt, out, finish } = helpers; /** @type {import("./doc-lib.mjs").FailFn} */ const fail = helpers.fail; -const VERBS = new Set(["init", "upsert-batch", "mark", "next", "stub", "crawl", "report", "remove", "exclude", "block"]); +const VERBS = new Set(["init", "upsert-batch", "mark", "next", "stub", "crawl", "report", "remove", "exclude", "block", "reconcile-docs"]); // Domain names are model-authored kebab identifiers (setup's naming rule). A // prototype-shaped name ("__proto__") survives the composite inventory keys @@ -376,6 +495,21 @@ function domainOrFail(domain) { // Hard cap on an exclude/block --reason (F-222). See the check site for why it // is set well above the observed working distribution rather than near it. const REASON_MAX = 1000; +// The recorded describe state of a domain stamp — the ONE reading (A-1) the +// stub banner and report's depth truth both derive from (F-334 → gate-3 F-346; +// F-455): the `none` sentinel is the operator's list-only decision, a template +// makes the domain describable, and null/absent (a legacy stamp, a bare-string +// stamp, an operator who never passed the flag, or no stamp at all) is +// UNRECORDED — nothing is known, so neither completeness nor a working --deep +// may be claimed for its stubs. +/** + * @param {GsDomainStamp|string|undefined} stamp + * @returns {"describable"|"list-only"|"unrecorded"} + */ +function describeStateOf(stamp) { + const recorded = stamp != null && typeof stamp === "object" && typeof stamp.describeCommand === "string" ? stamp.describeCommand : null; + return recorded === DESCRIBE_NONE ? "list-only" : recorded ? "describable" : "unrecorded"; +} // The three enumerations T-2 declares as unions. Each Set is TYPED AGAINST its // union, so a member added here but not to the typedef is red under both // configs (the sync mechanism A-2 asks for — measured at the 0.36.3 gate: with @@ -1299,6 +1433,28 @@ if (verb === "init") { const shown = unknown.slice(0, 5).join(", "); fail(`no inventory entry: ${shown}${unknown.length > 5 ? ` (+${unknown.length - 5} more)` : ""}${keysFile ? " — nothing marked" : ""}`); } + // A documented entry must know where its doc is (F-459). Every doc writer + // passes --doc-path on the mark that follows its write (stub and + // describe-batch by construction, the manual per-asset path by its + // reference), and describe-batch's --if-changed skip re-marks WITHOUT one + // only on an entry that already carries a path — so the only mark this + // refuses is the hand-run one that would create an entry no doc_path + // consumer can see (remove's cleanup list read 0 for a whole legacy domain + // whose stubs sat on disk). Checked before the loop: all-or-nothing, like + // the unknown-key rule above. + if (status === "documented" && docPath === undefined) { + const pathless = keys.filter((k) => typeof m.inventory[k].doc_path !== "string" || m.inventory[k].doc_path === ""); + if (pathless.length) { + const shown = pathless.slice(0, 5).join(", "); + fail( + `mark --status documented refused for ${shown}${pathless.length > 5 ? ` (+${pathless.length - 5} more)` : ""}: ` + + `the entr${pathless.length === 1 ? "y has" : "ies have"} no recorded doc_path and none was passed — a documented entry ` + + `with nowhere recorded is invisible to every doc_path consumer (F-459). Pass --doc-path ` + + `${keysFile ? " on a per-key mark (--keys-file marks carry no path; nothing was marked)" : ""}; for docs that already ` + + `exist on disk under the writers' naming, run reconcile-docs --domain first.` + ); + } + } for (const k of keys) { const e = m.inventory[k]; e.status = status; @@ -1393,13 +1549,7 @@ if (verb === "init") { // was avoiding.) The catalog cannot make the call for the operator: // scorecards describe through a different command group and connector // jobs through a flag, so a recorded decision is the only honest source. - const stubStamp = (m.domains_indexed ?? {})[domain]; - const recordedDescribe = - stubStamp != null && typeof stubStamp === "object" && typeof stubStamp.describeCommand === "string" - ? stubStamp.describeCommand - : null; - const describeState = - recordedDescribe === DESCRIBE_NONE ? "list-only" : recordedDescribe ? "describable" : "unrecorded"; + const describeState = describeStateOf((m.domains_indexed ?? {})[domain]); const banner = { describable: [ `${STUB_MARKER} (shallow crawl, captured ${now}) — full ingest:`, @@ -1493,6 +1643,46 @@ if (verb === "init") { const domainsIndexed = m.domains_indexed ?? {}; // Indexed but no inventory entries: the tenant genuinely has none of these. const emptyDomains = Object.keys(domainsIndexed).filter((d) => !byDomain[d]).sort(); + // Depth and change-detection truth per domain (F-455 / F-454 / F-451 / + // F-459 — header comment on `report` for the field semantics; typedef + // GsReportDomain). One row per domain that holds entries OR a stamp, so an + // empty stamp and an unstamped registration (F-316) both have a row; the + // folds are null-prototype like the ones above (F-225). NOTHING here is + // written into a byDomain row: describe-batch's progress probe sums a + // row's values, so those rows carry numbers only. + const newDepth = () => ({ full: 0, metadata: 0, listOnly: 0, unrecorded: 0 }); + const byDepth = newDepth(); + /** @type {Object} */ + const domains = Object.create(null); + const domainNames = [...new Set([...Object.keys(byDomain), ...Object.keys(domainsIndexed)])].sort(); + for (const d of domainNames) { + const stamp = Object.hasOwn(domainsIndexed, d) ? domainsIndexed[d] : undefined; + const stamped = stamp !== undefined; + const isObj = stamp != null && typeof stamp === "object"; + // dateField tri-state (T-2 GsDomainStamp): string = recorded, null = + // recorded-none, ABSENT = legacy/unknown — a bare-string stamp or no + // stamp at all reads as the third state, never as "none". + const changeDetection = isObj && "dateField" in stamp ? (typeof stamp.dateField === "string" ? "date" : "none") : "unrecorded"; + domains[d] = { stamped, describeState: describeStateOf(stamp), byDepth: newDepth(), changeDetection, datelessEntries: 0, docPathsUnknown: 0 }; + } + let docPathsUnknown = 0; + for (const e of Object.values(m.inventory)) { + const row = domains[e.domain]; + if (e.modified_date == null) row.datelessEntries++; + if (e.status !== "documented") continue; + if (typeof e.doc_path !== "string" || e.doc_path === "") { row.docPathsUnknown++; docPathsUnknown++; } + // A metadata stub's completeness is its DOMAIN's recorded describe state + // (the stub banner says the same three things); a full doc is full + // wherever it sits; no depth at all = a pre-depth-field doc = full (the + // stub verb's hasFullDoc rule — stubs have always carried a depth). + const bucket = + e.depth === "metadata" + ? { "list-only": "listOnly", describable: "metadata", unrecorded: "unrecorded" }[row.describeState] + : "full"; + row.byDepth[bucket]++; + byDepth[bucket]++; + } + const domainCounts = { indexed: Object.keys(domainsIndexed).length, withAssets: Object.keys(byDomain).length, empty: emptyDomains.length }; // Per-domain change window (F-417): days since the domain's OWN list stamp // — the `at` upsert-batch writes on every full list — never since a // workspace-wide last_refresh. A crawl that reached 2 of 17 domains leaves @@ -1525,6 +1715,10 @@ if (verb === "init") { domains_excluded: m.domains_excluded ?? {}, domains_blocked: m.domains_blocked ?? {}, emptyDomains, + byDepth, + domainCounts, + docPathsUnknown, + domains, }); } else if (verb === "exclude" || verb === "block") { const m = load(); @@ -1807,6 +2001,11 @@ if (verb === "init") { keys = [...new Set(keys)]; const missing = []; const docPaths = []; + // Documented entries removed with NO doc_path recorded (F-459): their docs + // may well sit on disk under the writers' naming, so the cleanup list is + // INCOMPLETE by this number — reported beside it, never folded into it, so + // an empty docPaths cannot read as "no docs" when the truth is "unknown". + let docPathsUnknown = 0; let removed = 0; for (const k of keys) { // hasOwn for the same reason as mark's presence check (F-168): a key like @@ -1818,11 +2017,95 @@ if (verb === "init") { } const e = m.inventory[k]; if (e.doc_path) docPaths.push(e.doc_path); + else if (e.status === "documented") docPathsUnknown++; delete m.inventory[k]; removed++; } save(m); // doc files are reported, not deleted — removal of tenant docs stays a // caller decision (they may be the only copy of a rekeyed asset's old doc) - out({ ok: true, removed, missing, docPaths, totalInventory: Object.keys(m.inventory).length }); + out({ ok: true, removed, missing, docPaths, docPathsUnknown, totalInventory: Object.keys(m.inventory).length }); +} else if (verb === "reconcile-docs") { + // Backfill doc_path from the docs already on disk (F-459; the + // reconcile-on-resume step issue #12's batched marks will reuse). The match + // is doc-lib's docNameMatcher — the claimer's read-side twin: it replays the + // doc writers' OWN naming rule (docBaseName, then the "-dup" chain a writer + // appends while the stem is claimed in-run or a DIFFERENT spelling of it + // sits on disk — case-folding filesystems, F-125/F-156) against the folder, + // so the file an entry is matched to is the file its writer would have + // chosen, and the suffix grammar stays in its one sanctioned home. Entries + // are walked in key order (cmpKey, the inventory's own order) so the replay + // is deterministic; the one residual the claimer documents (two ids whose + // docBaseName output is byte-identical) resolves to the first key here as + // it resolved to the first writer there. Read-then-write: the manifest is + // saved once, only when a path was recorded, and no doc file is touched. + const domain = opt("--domain") || fail("reconcile-docs requires --domain "); + domainOrFail(domain); + const dryRun = argv.includes("--dry-run"); + const detailOut = opt("--out"); + const m = load(); + // The folder every writer uses: /, and doc_path is + // recorded exactly as stub records it — the folder as passed, forward + // slashes, no trailing slash — so a path reconciled here is byte-identical + // to one a writer would have recorded (readers resolve it against the CWD). + const dirArg = opt("--dir") ?? join(dirname(manifestPath), domain); + const dirNorm = dirArg.replace(/\\/g, "/").replace(/\/$/, ""); + const dirAbs = resolve(dirArg); + const dirExists = existsSync(dirAbs); + const matcher = docNameMatcher(dirAbs); + const keys = Object.keys(m.inventory).filter((k) => m.inventory[k].domain === domain).sort(cmpKey); + /** @type {string[]} */ const recordedKeys = []; + /** @type {string[]} */ const recordedMissing = []; + /** @type {string[]} */ const unmatchedDocumented = []; + let alreadyRecorded = 0; + // Pass 1: entries that already carry a path claim their stem first (a + // recorded path is a fact; a name replay must never re-attribute it). + for (const k of keys) { + const e = m.inventory[k]; + if (typeof e.doc_path !== "string" || e.doc_path === "") continue; + if (existsSync(resolve(e.doc_path))) { + alreadyRecorded++; + const rel = e.doc_path.replace(/\\/g, "/"); + const slash = rel.lastIndexOf("/"); + const inDir = slash >= 0 && resolve(rel.slice(0, slash)) === dirAbs; + if (inDir && /\.md$/i.test(rel)) matcher.claim(rel.slice(slash + 1, -3)); + } else { + recordedMissing.push(k); + } + } + // Pass 2: the name replay for entries with no path. + for (const k of keys) { + const e = m.inventory[k]; + if (typeof e.doc_path === "string" && e.doc_path !== "") continue; + const match = matcher.match(e.id); + if (match === null) { + if (e.status === "documented") unmatchedDocumented.push(k); + continue; + } + if (!dryRun) e.doc_path = `${dirNorm}/${match}.md`; + recordedKeys.push(k); + } + const orphanFiles = matcher.unclaimed().map((s) => `${s}.md`); + if (!dryRun && recordedKeys.length) save(m); + const SAMPLE = 10; + /** @param {string[]} a */ + const sample = (a) => a.slice(0, SAMPLE); + const summary = { + ok: true, + domain, + dir: dirNorm, + dirExists, + dryRun, + entries: keys.length, + recorded: recordedKeys.length, + alreadyRecorded, + recordedMissing: recordedMissing.length, + unmatchedDocumented: unmatchedDocumented.length, + orphanFiles: orphanFiles.length, + samples: { recorded: sample(recordedKeys), recordedMissing: sample(recordedMissing), unmatchedDocumented: sample(unmatchedDocumented), orphanFiles: sample(orphanFiles) }, + detailFile: detailOut ?? null, + }; + // Full lists go to a file, never through model context (bulk-data rule). + if (detailOut) writeFileAtomicSync(resolve(detailOut), JSON.stringify({ ...summary, lists: { recorded: recordedKeys, recordedMissing, unmatchedDocumented, orphanFiles } }, null, 2) + "\n"); + out(summary); } diff --git a/plugins/gs-superadmin/skills/refresh/SKILL.md b/plugins/gs-superadmin/skills/refresh/SKILL.md index effe06e..137ac21 100644 --- a/plugins/gs-superadmin/skills/refresh/SKILL.md +++ b/plugins/gs-superadmin/skills/refresh/SKILL.md @@ -41,6 +41,11 @@ node .gs-superadmin/plugin/scripts/manifest.mjs report --manifest /_manife ``` If the manifest does not exist, or `total` is 0 with `last_refresh` null (setup was interrupted before Phase 4 completed), tell the user to run `/gs-superadmin:setup` first and stop. +Keep this report's `domains` block: step 4 names from it every domain whose changes +this run could not check (F-451) — `changeDetection` (`date` = a modified-date field +is recorded; `none` = recorded-none; `unrecorded` = a legacy stamp this run's upsert +records for the first time, step 3) and `datelessEntries` (entries with no stored +date, which the date comparison can never flag). **Environment backfill** (once per legacy manifest): if `report` above shows `environment: null` (a manifest that predates the field, or one initialised without it), @@ -247,12 +252,26 @@ passing `--list-command` on this refresh's upserts (above) is the backfill. ### 4 — Report +The `Not checked for change this run:` block is read from step 1's report `domains` +block — one line per domain whose `changeDetection` is `none` (no date field: the +whole domain sits outside change detection), one per domain whose `changeDetection` +is `unrecorded` (a legacy stamp; step 3's upsert recorded the field this run and +change detection starts next run), and one per domain with `changeDetection: date` +and `datelessEntries > 0` (rows the comparison could not see — the list-invisible +recovery path registers such entries). `` is that domain's `datelessEntries` and +`` its `byDomain` total. When no domain qualifies the block is the single word +`none` — never omitted, never silence: "Unchanged" above counts only what the date +comparison could see (F-451). ``` ✓ gs-superadmin refresh complete Lookback: days Changed: X assets marked stale New: Y assets added as pending Unchanged: Z assets + Not checked for change this run: + — no date field recorded; entries outside change detection + — date field recorded this run (legacy stamp); detection starts next refresh + — of entries carry no date under the recorded field ``` **Detect-only is the default posture — staleness detection is cheap; re-documenting is diff --git a/plugins/gs-superadmin/skills/setup/SKILL.md b/plugins/gs-superadmin/skills/setup/SKILL.md index aae65ba..cf4168e 100644 --- a/plugins/gs-superadmin/skills/setup/SKILL.md +++ b/plugins/gs-superadmin/skills/setup/SKILL.md @@ -453,11 +453,15 @@ Then: ``` node .gs-superadmin/plugin/scripts/manifest.mjs report --manifest /_manifest.json ``` -Relay the **per-domain counts** from the report, not just the total: "Indexed N assets -across M domains (domain: count, …). X pending, Y stale, Z documented." Include the -report's `emptyDomains` explicitly ("domain: 0 — listed, tenant has none") and the -exclusion ledger ("K list commands deliberately excluded — reasons recorded in -`domains_excluded`"). Treat any domain whose count lands exactly on a common default +Relay the **per-domain counts** from the report, not just the total, and take every +domain count from the report's `domainCounts` — `indexed` (coverage stamps), +`withAssets`, `empty` — never from a count of `byDomain`'s keys (F-454: the two differ +by exactly the empty domains, so a key count under-reports what the manifest holds): +"Indexed N assets across M domains — M = `domainCounts.indexed`, of which W hold +assets and E are listed but empty (domain: count, …). X pending, Y stale, Z +documented." Name each of the report's `emptyDomains` explicitly ("domain: 0 — listed, +tenant has none") and the exclusion ledger ("K list commands deliberately excluded — +reasons recorded in `domains_excluded`"). Treat any domain whose count lands exactly on a common default page size (20, 25, 50) as **suspect** — re-verify its pagination was exhausted before continuing — and ask the user whether the totals match their sense of the tenant before Phase 5 spends the documentation budget. @@ -505,7 +509,13 @@ block Phase 6, and the report names them "list-only (complete)", never **`--deep ` runs** — select the stubs awaiting full ingest with the `next --upgrade` invocation (`references/document-mechanics.md` §2), then follow the standard describe path below, overwriting each stub file (the entry's -`doc_path`) and marking `--status documented --depth full`. +`doc_path`) and marking `--status documented --depth full`. When the run's last +batch has exited, the statement that the domain is fully ingested is the quiescent +`report` quoted at the close of this phase (below), read at `domains..byDepth`: +fully ingested means `metadata: 0` and `unrecorded: 0` on that row with no `pending` +or `stale` in its `byDomain` row — relay the row, never a narrative "deep crawl +finished" (F-455: eight domains were entirely stubs when that sentence was last +relayed from memory). **All other domains (and deep crawl) — describe each entry in the batch.** Default execution is the sanctioned batch script (below); the manual per-asset path it @@ -586,19 +596,36 @@ first two and the three-level composite doc for designers. preference from the top of this phase — if the user opted in, pause and verify before starting the next domain. +**Every completeness statement comes from ONE `report` invocation at a quiescent +point** (F-455). When the last `describe-batch` or `stub` invocation of this run has +exited and nothing is still writing the manifest, run the report once: +``` +node .gs-superadmin/plugin/scripts/manifest.mjs report --manifest /_manifest.json +``` +and quote it rather than narrating: `byDepth` is the tenant's depth truth over +documented entries — `full` (describe docs), `metadata` (stubs awaiting `--deep`), +`listOnly` (stubs of a domain recorded `none`: complete by definition), `unrecorded` +(stubs of a domain with no describe recording: completeness unknown until Phase 4 +records a template or `none`) — and `domains..byDepth` is the same per +domain. "Documented" alone never means complete (a stub is documented); a domain is +fully ingested only when its row reads `metadata: 0` and `unrecorded: 0` with no +`pending` or `stale` in `byDomain`. A tally taken while a batch is still running is +a snapshot of a moving target — never quote one, and never re-derive these numbers +from `byStatus` or from what this run wrote. + After hitting the budget limit, report per the shapes in -`references/document-mechanics.md` §4 — keep documented / remaining / permanently -failed as separate counts (`failed` is a terminal state the skill deliberately -produces, not work still queued), and never advertise `--deep` for a list-only -domain. +`references/document-mechanics.md` §4 — every count in them comes from that one +quoted report; keep documented / remaining / permanently failed as separate counts +(`failed` is a terminal state the skill deliberately produces, not work still +queued), and never advertise `--deep` for a list-only domain. -**If any `pending` or `stale` entries remain, stop here.** Skip Phase 6 — relationship synthesis from partial data would produce misleading maps. Phase 6 runs automatically on the run that clears them, which is the same condition Phase 6 states as its precondition. Persistent `failed` entries do **not** block Phase 6: they are terminal, the `next` selection re-offers them so the remaining count would never clear on its own, and the maps' coverage headers report them ("N asset(s) failed describe and are not represented"). +**If the quoted report's `byStatus` carries any `pending` or `stale`, stop here.** Skip Phase 6 — relationship synthesis from partial data would produce misleading maps. Phase 6 runs automatically on the run that clears them, which is the same condition Phase 6 states as its precondition. Persistent `failed` entries do **not** block Phase 6: they are terminal, the `next` selection re-offers them so the remaining count would never clear on its own, and the maps' coverage headers report them ("N asset(s) failed describe and are not represented"). --- ## Phase 6 — Synthesize -**Precondition**: only run if all inventory entries are `documented` or `failed` (no `pending` or `stale` remaining). Metadata stubs count as documented — they never block this phase. +**Precondition**: only run if all inventory entries are `documented` or `failed` (no `pending` or `stale` remaining) — read from the quiescent `report` quoted at Phase 5's close, never re-tallied: its `byStatus` carries no `pending` and no `stale` key. Metadata stubs count as documented — they never block this phase (every `byDepth` bucket — `full`, `metadata`, `listOnly`, `unrecorded` — is a documented entry; the maps' coverage headers say which domains are stub-only). Build or refresh `/relationships/` — **default mechanics is the sanctioned script** (same rationale as describe-batch: deterministic parsing work, and the bulk @@ -647,11 +674,19 @@ a pointer from the relationships file if useful), never inline in generated file ## Final report +Every number below is read from the quiescent `report` quoted at Phase 5's close +(never from a running tally): `N` and `M` from `byStatus`; the `Depth:` line from +`byDepth`, all four buckets, each spelled even when 0; the `Not complete:` line lists +every domain whose `domains..byDepth` has `metadata > 0` or `unrecorded > 0`, +as ` ( metadata stubs)` or ` ( unrecorded — record a describe +command or none)`, comma-separated, or the word `none` when no domain qualifies. ``` ✓ gs-superadmin setup complete - Tenant: () - Documented: N assets - Remaining: M pending/stale - KB folder: / - Next steps: /gs-superadmin:refresh to detect changes + Tenant: () + Documented: N assets + Depth: F full · S metadata (awaiting --deep) · L list-only (complete) · U unrecorded (completeness unknown) + Not complete: + Remaining: M pending/stale + KB folder: / + Next steps: /gs-superadmin:refresh to detect changes ``` diff --git a/plugins/gs-superadmin/skills/setup/references/document-mechanics.md b/plugins/gs-superadmin/skills/setup/references/document-mechanics.md index 5f4a9ec..a8e0c64 100644 --- a/plugins/gs-superadmin/skills/setup/references/document-mechanics.md +++ b/plugins/gs-superadmin/skills/setup/references/document-mechanics.md @@ -71,15 +71,24 @@ reports `aborted.reason: "auth"` and leaves the entry's status untouched). ## §4 Budget-report shapes (after Phase 5 hits the budget limit) +Every count here is read from the ONE quiescent `report` Phase 5's close quotes +(F-455) — `byStatus` for the first shape, `domains..byDepth` for the domain +lines — never from a running tally or from what this run wrote. + - "Documented N assets. M remaining (pending/stale). F permanently failed. Re-run `/gs-superadmin:setup` to continue, or use `--budget N` / `--all`." Keep the two counts separate: `failed` entries are a terminal state this skill deliberately produces (un-describable assets, the known scorecard auth race), not work still queued. -- On a shallow crawl, also list the stubbed domains that have a describe command: - "Metadata-only (shallow): ( stubs) — full ingest: - `/gs-superadmin:setup --deep `." List list-only domains separately as - "List-only (complete): ()" — never advertise `--deep` for them. +- List every domain whose row has `metadata > 0` — stubs under a recorded describe + template: "Metadata-only (shallow): ( stubs) — full ingest: + `/gs-superadmin:setup --deep `." List every domain whose row has + `listOnly > 0` separately as "List-only (complete): ()" — never + advertise `--deep` for them. List every domain whose row has `unrecorded > 0` as + "Completeness unknown: () — no describe command recorded; re-run + its Phase 4 upsert with `--describe-command` (a template, or `none`)" — neither + complete nor a working `--deep` may be claimed for those stubs (their banners say + the same). ## §5 Hand-chaining several describes diff --git a/plugins/gs-superadmin/test/contract-conformance.mjs b/plugins/gs-superadmin/test/contract-conformance.mjs index fe22c36..212099f 100644 --- a/plugins/gs-superadmin/test/contract-conformance.mjs +++ b/plugins/gs-superadmin/test/contract-conformance.mjs @@ -197,6 +197,38 @@ try { const lg = m.domains_indexed["legacy-domain"]; check("legacy string stamp upgraded to an object on re-stamp", lg && typeof lg === "object" && "itemsPath" in lg && lg.itemsPath === null, lg); + + // T-2 v3 (Session C1, 2026-09-15): report's OUTPUT is part of the contract — + // the skills quote its fields and describe-batch reads byDomain. Key sets + // re-derived from the GsReport / GsReportDomain / GsReportDepth typedefs + // (never imported — the independent-derivation standard above), the v2 keys + // all still present (additive), byDomain rows numbers-only (the progress + // probe sums them), and remove's docPathsUnknown beside docPaths. + r = verb("report", []); + check("T-2 v3: report exits 0 with JSON", r.status === 0, r.stderr); + const rep = JSON.parse(r.stdout); + const T2_REPORT_KEYS = [ + "ok", "slug", "baseUrl", "environment", "total", "last_refresh", "byStatus", "byDomain", "lookback", "lookbackDefault", + "domains_indexed", "domains_excluded", "domains_blocked", "emptyDomains", + "byDepth", "domainCounts", "docPathsUnknown", "domains", + ].sort().join(","); + check("T-2 v3: report carries exactly the typedef keys (v2 set + byDepth/domainCounts/docPathsUnknown/domains)", Object.keys(rep).sort().join(",") === T2_REPORT_KEYS, Object.keys(rep).sort()); + const T2_DEPTH_KEYS = "full,listOnly,metadata,unrecorded"; + check("T-2 v3: byDepth is exactly {full, metadata, listOnly, unrecorded}, all numbers", Object.keys(rep.byDepth).sort().join(",") === T2_DEPTH_KEYS && Object.values(rep.byDepth).every((v) => typeof v === "number"), rep.byDepth); + check("T-2 v3: domainCounts is exactly {indexed, withAssets, empty}", Object.keys(rep.domainCounts).sort().join(",") === "empty,indexed,withAssets", rep.domainCounts); + check("T-2 v3: docPathsUnknown is a number", typeof rep.docPathsUnknown === "number", rep.docPathsUnknown); + const T2_DOMAIN_ROW_KEYS = "byDepth,changeDetection,datelessEntries,describeState,docPathsUnknown,stamped"; + check("T-2 v3: one domains row per stamped or populated domain, each exactly {stamped, describeState, byDepth, changeDetection, datelessEntries, docPathsUnknown}", + Object.keys(rep.domains).sort().join(",") === [...new Set([...Object.keys(rep.byDomain), ...Object.keys(rep.domains_indexed)])].sort().join(",") && + Object.values(rep.domains).every((d) => Object.keys(d).sort().join(",") === T2_DOMAIN_ROW_KEYS && Object.keys(d.byDepth).sort().join(",") === T2_DEPTH_KEYS), + rep.domains); + check("T-2 v3: describeState ∈ describable|list-only|unrecorded and changeDetection ∈ date|none|unrecorded on every row", + Object.values(rep.domains).every((d) => ["describable", "list-only", "unrecorded"].includes(d.describeState) && ["date", "none", "unrecorded"].includes(d.changeDetection) && typeof d.stamped === "boolean"), rep.domains); + check("T-2 v3: byDomain rows are numbers only — nothing new was nested into them", Object.values(rep.byDomain).every((row) => Object.values(row).every((v) => typeof v === "number")), rep.byDomain); + check("T-2 v3: the fixture's recorded-none domain reads changeDetection none; the recorded-field domain reads date; the legacy string-stamp domain reads unrecorded", + rep.domains["rules-engine-rules"]?.changeDetection === "none" && rep.domains["journey-email-templates"]?.changeDetection === "date" && rep.domains["legacy-domain"]?.changeDetection === "unrecorded", Object.fromEntries(Object.entries(rep.domains).map(([k, d]) => [k, d.changeDetection]))); + r = verb("remove", ["--key", "legacy-domain/l-1"]); + check("T-2 v3: remove's summary is exactly {ok, removed, missing, docPaths, docPathsUnknown, totalInventory}", r.status === 0 && Object.keys(JSON.parse(r.stdout)).sort().join(",") === "docPaths,docPathsUnknown,missing,ok,removed,totalInventory", r.stdout); } finally { removeTempDir(dir); } diff --git a/plugins/gs-superadmin/test/doc-lib-fixtures.mjs b/plugins/gs-superadmin/test/doc-lib-fixtures.mjs index 31d36e4..31891b4 100644 --- a/plugins/gs-superadmin/test/doc-lib-fixtures.mjs +++ b/plugins/gs-superadmin/test/doc-lib-fixtures.mjs @@ -28,7 +28,7 @@ import { pathToFileURL, fileURLToPath } from "node:url"; import { dirname } from "node:path"; import { makeTempDir, removeTempDir, writeFiles, runNode } from "../../../test/rig.mjs"; import { - stripBom, normalizeText, readJsonFile, codePointSlice, docBaseName, docNameClaimer, + stripBom, normalizeText, readJsonFile, codePointSlice, docBaseName, docNameClaimer, docNameMatcher, canonicalFingerprint, getPath, extractIds, findItemsArray, makeCliHelpers, requireKbDir, sq, shq, needsPosixQuoteCaveat, normTerm, termKey, cmpName, cmpKey, stripLauncherSuffix, extractFencedJson, topBullets, findWorkspaceCatalog, findWorkspaceDir, @@ -263,6 +263,75 @@ check("normalizeText: interior BOM preserved (F-113)", normalizeText("a" + BOM + } } +// ── docNameMatcher — the claimer's read-side twin (F-459 reconcile-docs) ───── +// Decision points, each pinned: exact stem on disk and unclaimed → match; +// a DIFFERENT spelling on disk → "-dup" hop (the claimer's foreignOnDisk); +// a stem claimed in-run (case-insensitively) → "-dup" hop; a free stem with +// no file → null; claim() of a recorded path excludes it from replay; +// unclaimed() is the orphan list; a missing folder matches nothing and orphans +// nothing; the hop bound stops a pathological chain. +{ + const dir = makeTempDir("doclib-matcher"); + try { + // What a claimer-driven writer left behind: Co.md then co-dup.md for the + // colliding pair, Own.md, and a stray file no id owns. + writeFileSync(join(dir, "Co.md"), "x"); + writeFileSync(join(dir, "co-dup.md"), "x"); + writeFileSync(join(dir, "Own.md"), "x"); + writeFileSync(join(dir, "stray.md"), "x"); + writeFileSync(join(dir, "notes.txt"), "x"); // not a doc — never a stem + const mm = docNameMatcher(dir); + check("matcher: exact on-disk stem matches and is claimed", mm.match("Own") === "Own"); + check("matcher: a second id resolving to a claimed exact stem hops to -dup and finds nothing", mm.match("own") === null); + check("matcher: no file under the writer's first free choice → null", mm.match("Missing") === null); + check("matcher: case-different spelling on disk forces the -dup hop, which finds co-dup", mm.match("co") === "co-dup"); + check("matcher: the exact-case owner still matches its own file after the pair's other half", mm.match("Co") === "Co"); + check("matcher: unclaimed() lists the orphans only, sorted, .md stems only", JSON.stringify(mm.unclaimed()) === JSON.stringify(["stray"]), mm.unclaimed()); + // claim() of a recorded path: the replay must not re-attribute it. + const m2 = docNameMatcher(dir); + m2.claim("Own"); + check("matcher: a pre-claimed recorded stem is not re-attributed by the replay", m2.match("Own") === null); + check("matcher: unclaimed() excludes pre-claimed stems", !m2.unclaimed().includes("Own") && m2.unclaimed().includes("stray")); + // Call order decides the byte-identical-name residual, as it does for + // the claimer: a hash-suffixed spelling of another id's clean output. + writeFileSync(join(dir, `${docBaseName("a/b")}.md`), "x"); + // The snapshot is taken at construction (one read per reconcile): both + // matchers below are built AFTER the file exists. + const m3 = docNameMatcher(dir); + const m4 = docNameMatcher(dir); + check("matcher: sanitized id matches its hash-suffixed file", m4.match("a/b") === docBaseName("a/b")); + check("matcher: an id spelled as another's sanitized output loses to the earlier call (documented residual)", m4.match(docBaseName("a/b")) === null && m3.match(docBaseName("a/b")) === docBaseName("a/b")); + // Missing folder: matches nothing, orphans nothing, never throws. + const m5 = docNameMatcher(join(dir, "not-created-yet")); + check("matcher: missing dir → null and no orphans", m5.match("anything") === null && m5.unclaimed().length === 0); + // The -dup chain resolves as far as files go (six stacked spellings on + // disk, six ids resolving to them), and the hop bound stops a chain that + // never frees: 64 stems claimed in-run (no files needed — a claimed stem + // hops exactly like a foreign spelling) and the next match returns null + // instead of spinning. (Stacking 65 suffixes as FILES would exceed the + // 255-character filename limit on Windows and NTFS, so the bound is + // proved through claims.) + const deep = makeTempDir("doclib-matcher-deep"); + try { + let stem = "d"; + for (let i = 0; i < 6; i++) { writeFileSync(join(deep, `${stem}.md`), "x"); stem += "-dup"; } + const m6 = docNameMatcher(deep); + let last = null; + for (let i = 0; i < 6; i++) last = m6.match("d"); + check("matcher: six ids resolve down the six-file -dup chain", last === "d" + "-dup".repeat(5), last); + check("matcher: the seventh finds no file past the chain → null", m6.match("d") === null); + const m7 = docNameMatcher(deep); + let claimedStem = "d"; + for (let i = 0; i < 64; i++) { m7.claim(claimedStem); claimedStem += "-dup"; } + check("matcher: 64 claimed hops hit the bound → null, no infinite loop", m7.match("d") === null); + } finally { + removeTempDir(deep); + } + } finally { + removeTempDir(dir); + } +} + // ── canonicalFingerprint — volatile-key drop + canonicalization ────────────── { const a = canonicalFingerprint({ b: 1, a: { modifiedDate: "x", v: 2 } }); diff --git a/plugins/gs-superadmin/test/manifest-ops.mjs b/plugins/gs-superadmin/test/manifest-ops.mjs index 0405dae..3a335be 100644 --- a/plugins/gs-superadmin/test/manifest-ops.mjs +++ b/plugins/gs-superadmin/test/manifest-ops.mjs @@ -11,9 +11,14 @@ // --allow-redate, baseline adoption over null-stored dates, epoch-ms // ordering in newerThan), the under-count warning and the --partial // deliberate-subset declaration (F-313), the mass-stale advisory, -// mark --fingerprint validation, and mark --keys-file/--limit batch marking -// (all-or-nothing, file-order limit, BOM tolerance — F-162). Fixtures live -// under the OS temp dir; no real state. +// mark --fingerprint validation, mark --keys-file/--limit batch marking +// (all-or-nothing, file-order limit, BOM tolerance — F-162), and the Session +// C1 truth set (F-455/F-454/F-451/F-459): report's byDepth four states per +// domain and in total, domainCounts, changeDetection + datelessEntries, +// docPathsUnknown; mark's refusal of a pathless documented mark; remove's +// docPathsUnknown; reconcile-docs over a legacy folder (matched, case- +// collision pair, recorded-missing, unmatched, orphans, dry-run, --out). +// Fixtures live under the OS temp dir; no real state. // // Run: node plugins/gs-superadmin/test/manifest-ops.mjs // Zero dependencies — Node built-ins only. @@ -80,8 +85,11 @@ r = run("next", ["--manifest", M, "--domain", "rules-engine-rules"]); check("next: --domain filters to that domain only", r.json?.count === 1 && r.json.entries[0].key === "rules-engine-rules/r-1", r); // ── stub: never downgrades a full doc ──────────────────────────────────────── -run("mark", ["--manifest", M, "--key", "journey-email-templates/t-1", "--status", "documented"]); const outDir = join(ROOT, "acme-sbx", "journey-email-templates"); +// A documented mark names where its doc landed (F-459) — the fixture's hand-run +// mark passes the path the writer would have recorded (the file need not exist +// for mark; existence is reconcile-docs' business). +run("mark", ["--manifest", M, "--key", "journey-email-templates/t-1", "--status", "documented", "--doc-path", join(outDir, "t-1.md")]); r = run("stub", ["--manifest", M, "--file", listFile, "--domain", "journey-email-templates", "--id-field", "templateId", "--name-field", "title", "--out-dir", outDir]); check("stub: stubs pending entries, skips the full-documented one", r.json?.stubbed === 2 && r.json?.skippedFull === 1, r); check("stub: no stub file written for the full-documented entry", !existsSync(join(outDir, "t-1.md")), readdirSync(outDir)); @@ -749,11 +757,11 @@ check("advisory: mass stale flip warns to verify the date field first", r.code = // ── mark --fingerprint: stored on documented marks, strict 40-hex shape ────── const FP = "a".repeat(40); -r = run("mark", ["--manifest", M, "--key", "pagedom/p-1", "--status", "documented", "--fingerprint", FP]); +r = run("mark", ["--manifest", M, "--key", "pagedom/p-1", "--status", "documented", "--fingerprint", FP, "--doc-path", "acme-sbx/pagedom/p-1.md"]); check("fingerprint: stored on a documented mark", r.code === 0 && dj().inventory["pagedom/p-1"].fingerprint === FP, r); r = run("mark", ["--manifest", M, "--key", "pagedom/p-1", "--status", "failed", "--error", "x"]); check("fingerprint: failed mark leaves the fingerprint untouched", r.code === 0 && dj().inventory["pagedom/p-1"].fingerprint === FP, dj().inventory["pagedom/p-1"]); -r = run("mark", ["--manifest", M, "--key", "pagedom/p-2", "--status", "documented", "--fingerprint", "B".repeat(40)]); +r = run("mark", ["--manifest", M, "--key", "pagedom/p-2", "--status", "documented", "--fingerprint", "B".repeat(40), "--doc-path", "acme-sbx/pagedom/p-2.md"]); check("fingerprint: uppercase hex accepted and stored lowercase", r.code === 0 && dj().inventory["pagedom/p-2"].fingerprint === "b".repeat(40), r); r = run("mark", ["--manifest", M, "--key", "pagedom/p-2", "--status", "documented", "--fingerprint", "not-a-fingerprint"]); check("fingerprint: malformed value rejected (exit 1)", r.code === 1 && /40-char hex/.test(r.stderr), r); @@ -1510,6 +1518,148 @@ check("fingerprint: malformed value rejected (exit 1)", r.code === 1 && /40-char check("f432 (control): a LIST-command collision (alias spelling of a recorded list) is still refused, whatever the describe command says", r.code === 1 && /--list-command is the recorded listCommand of domain jobs-shared-describe/.test(r.stderr), r.stderr.slice(0, 200)); } +// ── Session C1 (F-455 / F-454 / F-451 / F-459): report carries depth, domain +// counts, change-detection honesty and doc_path truth; mark refuses a pathless +// documented mark; remove counts unknown paths; reconcile-docs backfills ──── +{ + const MC = join(ROOT, "acme-c1", "_manifest.json"); + const djC = () => JSON.parse(readFileSync(MC, "utf8")); + const BSL = String.fromCharCode(92); + const fwd = (p) => p.split(BSL).join("/"); + const rows = (name, items) => { const f = join(ROOT, `c1-${name}.json`); writeFileSync(f, JSON.stringify({ data: items })); return f; }; + run("init", ["--manifest", MC, "--slug", "acme-c1", "--base-url", "https://c1.example", "--environment", "sandbox"]); + // full-dom: describable, dated — two full docs (one written before the depth + // field existed) and one pending entry. + const fullList = rows("full", [ + { id: "a-1", name: "Alpha", modifiedDate: "2026-01-01T00:00:00Z" }, + { id: "a-2", name: "Beta", modifiedDate: "2026-01-02T00:00:00Z" }, + { id: "a-3", name: "Gamma", modifiedDate: "2026-01-03T00:00:00Z" }, + ]); + run("upsert-batch", ["--manifest", MC, "--file", fullList, "--domain", "full-dom", "--id-field", "id", "--name-field", "name", "--date-field", "modifiedDate", "--describe-command", "gs-admin --json re r describe --id {id}", "--list-command", "gs-admin --json re r list"]); + run("mark", ["--manifest", MC, "--key", "full-dom/a-1", "--status", "documented", "--depth", "full", "--doc-path", "acme-c1/full-dom/a-1.md"]); + run("mark", ["--manifest", MC, "--key", "full-dom/a-2", "--status", "documented", "--depth", "full", "--doc-path", "acme-c1/full-dom/a-2.md"]); + // meta-dom: describable, a date field recorded under which every row is null + // (F-451 shape ii) — two metadata stubs awaiting --deep. + const metaList = rows("meta", [{ id: "m-1", name: "Meta One", modifiedDate: null }, { id: "m-2", name: "Meta Two", modifiedDate: null }]); + run("upsert-batch", ["--manifest", MC, "--file", metaList, "--domain", "meta-dom", "--id-field", "id", "--name-field", "name", "--date-field", "modifiedDate", "--describe-command", "gs-admin --json jo p describe --id {id}", "--list-command", "gs-admin --json jo p list"]); + run("stub", ["--manifest", MC, "--file", metaList, "--domain", "meta-dom", "--id-field", "id", "--name-field", "name", "--out-dir", join(ROOT, "acme-c1", "meta-dom")]); + // listonly-dom: `none` recorded, recorded-none date field (F-451 shape i). + const loList = rows("lo", [{ id: "c-1", name: "Conn One" }, { id: "c-2", name: "Conn Two" }]); + run("upsert-batch", ["--manifest", MC, "--file", loList, "--domain", "listonly-dom", "--id-field", "id", "--name-field", "name", "--no-date-field", "--describe-command", "none", "--list-command", "gs-admin --json cn list"]); + run("stub", ["--manifest", MC, "--file", loList, "--domain", "listonly-dom", "--id-field", "id", "--name-field", "name", "--out-dir", join(ROOT, "acme-c1", "listonly-dom")]); + // legacy-dom: no describe recording, and (below) no dateField key on the + // stamp — the July-crawl shape. Six stubs, two of them a case-colliding pair + // (the claimer writes Co.md and co-dup.md). + const legacyList = rows("legacy", ["l-1", "l-2", "l-3", "l-4", "Co", "co"].map((id) => ({ id, name: `Legacy ${id}`, modifiedDate: null }))); + const legacyDir = join(ROOT, "acme-c1", "legacy-dom"); + run("upsert-batch", ["--manifest", MC, "--file", legacyList, "--domain", "legacy-dom", "--id-field", "id", "--name-field", "name", "--date-field", "modifiedDate", "--list-command", "gs-admin --json sc list"]); + r = run("stub", ["--manifest", MC, "--file", legacyList, "--domain", "legacy-dom", "--id-field", "id", "--name-field", "name", "--out-dir", legacyDir]); + check("c1 rig: legacy stubs written under the claimer's naming (Co.md + co-dup.md)", r.json?.stubbed === 6 && existsSync(join(legacyDir, "Co.md")) && existsSync(join(legacyDir, "co-dup.md")), readdirSync(legacyDir)); + // empty-dom: listed, tenant has none. + const emptyFile = join(ROOT, "c1-empty.json"); + writeFileSync(emptyFile, JSON.stringify({})); + run("upsert-batch", ["--manifest", MC, "--file", emptyFile, "--domain", "empty-dom", "--id-field", "id", "--no-date-field", "--allow-empty", "--list-command", "gs-admin --json jo surveys list"]); + // The legacy shapes no verb can write any more, produced the way the + // workspaces got them (the fixture edits the JSON; skills never do): a doc + // written before path recording existed (documented, no doc_path — five of + // the six; l-2 keeps its path), a stamp from before dateField was recorded, + // and a full doc from before the depth field existed. + { + const m = djC(); + for (const k of ["legacy-dom/l-1", "legacy-dom/l-3", "legacy-dom/l-4", "legacy-dom/Co", "legacy-dom/co"]) delete m.inventory[k].doc_path; + delete m.domains_indexed["legacy-dom"].dateField; + delete m.inventory["full-dom/a-2"].depth; + writeFileSync(MC, JSON.stringify(m, null, 2)); + } + + // report — the four depth states, per domain and in total (F-455). + r = run("report", ["--manifest", MC]); + const rep = r.json; + check("c1 report: byDepth totals — full 2 (one pre-depth doc), metadata 2, listOnly 2, unrecorded 6", JSON.stringify(rep?.byDepth) === JSON.stringify({ full: 2, metadata: 2, listOnly: 2, unrecorded: 6 }), rep?.byDepth); + check("c1 report: full-dom row — describable, byDepth full 2, the pending entry under byStatus only", rep?.domains?.["full-dom"]?.describeState === "describable" && JSON.stringify(rep.domains["full-dom"].byDepth) === JSON.stringify({ full: 2, metadata: 0, listOnly: 0, unrecorded: 0 }) && rep.byDomain["full-dom"].pending === 1, rep?.domains?.["full-dom"]); + check("c1 report: meta-dom row — metadata stubs under a template read metadata (awaiting --deep)", rep?.domains?.["meta-dom"]?.byDepth?.metadata === 2 && rep.domains["meta-dom"].byDepth.listOnly === 0, rep?.domains?.["meta-dom"]); + check("c1 report: listonly-dom row — metadata stubs under `none` read listOnly, never metadata", rep?.domains?.["listonly-dom"]?.describeState === "list-only" && rep.domains["listonly-dom"].byDepth.listOnly === 2 && rep.domains["listonly-dom"].byDepth.metadata === 0, rep?.domains?.["listonly-dom"]); + check("c1 report: legacy-dom row — stubs under NO recording read unrecorded (the F-346 class), never listOnly or metadata", rep?.domains?.["legacy-dom"]?.describeState === "unrecorded" && rep.domains["legacy-dom"].byDepth.unrecorded === 6 && rep.domains["legacy-dom"].byDepth.listOnly === 0 && rep.domains["legacy-dom"].byDepth.metadata === 0, rep?.domains?.["legacy-dom"]); + check("c1 report: byDomain rows still carry numbers only (describe-batch sums them)", Object.values(rep?.byDomain ?? {}).every((row) => Object.values(row).every((v) => typeof v === "number")), rep?.byDomain); + check("c1 report: byStatus unchanged in shape — documented 12, pending 1", rep?.byStatus?.documented === 12 && rep?.byStatus?.pending === 1, rep?.byStatus); + // domainCounts (F-454): 5 stamps, 4 with assets, 1 empty — the empty domain + // is counted, never lost to a byDomain-key count. + check("c1 report: domainCounts 5 indexed / 4 withAssets / 1 empty, emptyDomains names it", JSON.stringify(rep?.domainCounts) === JSON.stringify({ indexed: 5, withAssets: 4, empty: 1 }) && JSON.stringify(rep?.emptyDomains) === JSON.stringify(["empty-dom"]), rep?.domainCounts); + check("c1 report: the empty domain has a row (stamped, no assets)", rep?.domains?.["empty-dom"]?.stamped === true && rep.domains["empty-dom"].byDepth.full === 0 && rep.domains["empty-dom"].datelessEntries === 0, rep?.domains?.["empty-dom"]); + // changeDetection + datelessEntries (F-451): recorded field → date; recorded- + // none → none; no key (legacy) → unrecorded; the dateless count is honest + // under every state. + check("c1 report: changeDetection date/none/unrecorded per stamp state", rep?.domains?.["full-dom"]?.changeDetection === "date" && rep.domains["meta-dom"].changeDetection === "date" && rep.domains["listonly-dom"].changeDetection === "none" && rep.domains["legacy-dom"].changeDetection === "unrecorded" && rep.domains["empty-dom"].changeDetection === "none", Object.fromEntries(Object.entries(rep?.domains ?? {}).map(([k, v]) => [k, v.changeDetection]))); + check("c1 report: datelessEntries — 0 under real dates, 2 null-under-recorded-field, 2 recorded-none, 6 legacy", rep?.domains?.["full-dom"]?.datelessEntries === 0 && rep.domains["meta-dom"].datelessEntries === 2 && rep.domains["listonly-dom"].datelessEntries === 2 && rep.domains["legacy-dom"].datelessEntries === 6, Object.fromEntries(Object.entries(rep?.domains ?? {}).map(([k, v]) => [k, v.datelessEntries]))); + // docPathsUnknown (F-459): five documented entries with no path, all in + // legacy-dom; the total is the tenant sum. + check("c1 report: docPathsUnknown 5 in legacy-dom, 0 elsewhere, total 5", rep?.docPathsUnknown === 5 && rep.domains["legacy-dom"].docPathsUnknown === 5 && rep.domains["full-dom"].docPathsUnknown === 0 && rep.domains["meta-dom"].docPathsUnknown === 0, rep?.docPathsUnknown); + // An unstamped registration (F-316) gets a row with stamped false and the + // counts say so: withAssets no longer equals indexed − empty. + const partialList = rows("partial", [{ id: "p-1", name: "Recovered" }]); + r = run("upsert-batch", ["--manifest", MC, "--file", partialList, "--domain", "partial-dom", "--id-field", "id", "--name-field", "name", "--partial"]); + const beforeRep = readFileSync(MC, "utf8"); + r = run("report", ["--manifest", MC]); + check("c1 report: an unstamped domain (partial registration) — stamped false, describe/changeDetection unrecorded, withAssets 5 vs indexed 5 / empty 1", r.json?.domains?.["partial-dom"]?.stamped === false && r.json.domains["partial-dom"].describeState === "unrecorded" && r.json.domains["partial-dom"].changeDetection === "unrecorded" && JSON.stringify(r.json.domainCounts) === JSON.stringify({ indexed: 5, withAssets: 5, empty: 1 }), r.json?.domainCounts); + check("c1 report: read-only — the report leaves the manifest byte-identical and records no path", r.code === 0 && readFileSync(MC, "utf8") === beforeRep && djC().inventory["legacy-dom/l-1"].doc_path === undefined, null); + + // mark refusal (F-459 part 1): a documented mark on a pathless entry with no + // --doc-path is refused, nothing written; every other shape still works. + const before = readFileSync(MC, "utf8"); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-1", "--status", "documented"]); + check("c1 mark: documented mark on a pathless entry with no --doc-path is refused (exit 1, names the remedy)", r.code === 1 && /no recorded doc_path and none was passed/.test(r.stderr) && /reconcile-docs/.test(r.stderr), r.stderr.slice(0, 200)); + check("c1 mark: the refusal wrote nothing", readFileSync(MC, "utf8") === before, null); + const pathlessKeys = join(ROOT, "c1-pathless-keys.json"); + writeFileSync(pathlessKeys, JSON.stringify(["legacy-dom/l-2", "legacy-dom/l-3"])); + r = run("mark", ["--manifest", MC, "--keys-file", pathlessKeys, "--status", "documented"]); + check("c1 mark: a --keys-file documented batch with ONE pathless key is refused whole — all-or-nothing, names the key", r.code === 1 && /legacy-dom\/l-3/.test(r.stderr) && !/legacy-dom\/l-2/.test(r.stderr) && /nothing was marked/.test(r.stderr) && readFileSync(MC, "utf8") === before, r.stderr.slice(0, 300)); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-2", "--status", "documented"]); + check("c1 mark (control): an entry that already carries a doc_path re-marks documented without one (the --if-changed skip)", r.code === 0 && djC().inventory["legacy-dom/l-2"].status === "documented", r); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-1", "--status", "failed", "--error", "x"]); + check("c1 mark (control): a failed mark on a pathless entry is not refused — only documented needs a path", r.code === 0 && djC().inventory["legacy-dom/l-1"].status === "failed", r); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-1", "--status", "documented", "--depth", "metadata", "--doc-path", join(legacyDir, "l-1.md")]); + check("c1 mark (control): the same mark WITH --doc-path is accepted and records it", r.code === 0 && /l-1\.md$/.test(djC().inventory["legacy-dom/l-1"].doc_path), r); + { const m = djC(); delete m.inventory["legacy-dom/l-1"].doc_path; writeFileSync(MC, JSON.stringify(m, null, 2)); } // back to the legacy shape for the reconcile arms + + // remove (F-459 part 2): a removed pathless documented entry is COUNTED, so + // docPaths [] never reads as "no docs". + r = run("remove", ["--manifest", MC, "--key", "legacy-dom/l-4"]); + check("c1 remove: pathless documented entry removed — docPaths [] but docPathsUnknown 1", r.code === 0 && r.json?.removed === 1 && r.json.docPaths.length === 0 && r.json.docPathsUnknown === 1, r.json); + r = run("remove", ["--manifest", MC, "--key", "full-dom/a-1"]); + check("c1 remove (control): an entry with a recorded path — docPaths names it, docPathsUnknown 0", r.code === 0 && r.json?.docPaths?.length === 1 && r.json.docPathsUnknown === 0, r.json); + + // reconcile-docs (F-459 part 3) over the legacy folder: l-1.md, Co.md and + // co-dup.md match pathless entries; l-2's recorded file is deleted (recorded + // missing, never rewritten); l-3.md is deleted (documented, no doc); l-4.md + // (entry removed above) and stray.md are orphans. + rmSync(join(legacyDir, "l-2.md")); + rmSync(join(legacyDir, "l-3.md")); + writeFileSync(join(legacyDir, "stray.md"), "# stray\n"); + const beforeRec = readFileSync(MC, "utf8"); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom", "--dry-run"]); + check("c1 reconcile --dry-run: computes recorded 3 / recordedMissing 1 / unmatchedDocumented 1 / orphanFiles 2 and writes nothing", r.code === 0 && r.json?.dryRun === true && r.json.recorded === 3 && r.json.recordedMissing === 1 && r.json.unmatchedDocumented === 1 && r.json.orphanFiles === 2 && readFileSync(MC, "utf8") === beforeRec, r.json); + const detail = join(ROOT, "c1-reconcile.json"); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom", "--out", detail]); + const inv = djC().inventory; + check("c1 reconcile: records doc_path on the three matched entries under the writers' naming — l-1.md, Co.md, co-dup.md (the claimer's collision suffix replayed)", r.code === 0 && r.json?.recorded === 3 && /\/l-1\.md$/.test(inv["legacy-dom/l-1"].doc_path) && /\/Co\.md$/.test(inv["legacy-dom/Co"].doc_path) && /\/co-dup\.md$/.test(inv["legacy-dom/co"].doc_path), { l1: inv["legacy-dom/l-1"].doc_path, Co: inv["legacy-dom/Co"].doc_path, co: inv["legacy-dom/co"].doc_path }); + check("c1 reconcile: doc_path is recorded forward-slashed under the default folder /, the spelling stub writes", inv["legacy-dom/l-1"].doc_path === `${fwd(legacyDir)}/l-1.md` && !inv["legacy-dom/Co"].doc_path.includes(BSL), inv["legacy-dom/l-1"].doc_path); + check("c1 reconcile: a recorded path whose file is gone is reported (recordedMissing 1, l-2) and left as recorded — never rewritten", r.json?.recordedMissing === 1 && r.json.samples.recordedMissing[0] === "legacy-dom/l-2" && /l-2\.md$/.test(inv["legacy-dom/l-2"].doc_path), r.json?.samples); + check("c1 reconcile: a documented entry with no doc on disk is named (unmatchedDocumented 1, l-3), its doc_path stays absent", r.json?.unmatchedDocumented === 1 && r.json.samples.unmatchedDocumented[0] === "legacy-dom/l-3" && inv["legacy-dom/l-3"].doc_path === undefined, r.json?.samples); + check("c1 reconcile: orphan files named — l-4.md (entry removed) and stray.md — and no doc file was created or deleted", r.json?.orphanFiles === 2 && JSON.stringify(r.json.samples.orphanFiles) === JSON.stringify(["l-4.md", "stray.md"]) && existsSync(join(legacyDir, "stray.md")) && existsSync(join(legacyDir, "l-4.md")) && !existsSync(join(legacyDir, "l-3.md")), r.json?.samples); + check("c1 reconcile: --out writes the full lists (bulk never through stdout)", existsSync(detail) && JSON.stringify(JSON.parse(readFileSync(detail, "utf8")).lists.recorded.slice().sort()) === JSON.stringify(["legacy-dom/Co", "legacy-dom/co", "legacy-dom/l-1"]), existsSync(detail) ? JSON.parse(readFileSync(detail, "utf8")).lists : null); + r = run("report", ["--manifest", MC]); + check("c1 reconcile → report: docPathsUnknown drops from 4 to 1 (l-3, the doc that is gone)", r.json?.docPathsUnknown === 1 && r.json.domains["legacy-dom"].docPathsUnknown === 1, r.json?.docPathsUnknown); + const afterRec = readFileSync(MC, "utf8"); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom"]); + check("c1 reconcile: idempotent — a re-run records 0, alreadyRecorded 3, same missing/unmatched/orphans, manifest untouched", r.code === 0 && r.json?.recorded === 0 && r.json.alreadyRecorded === 3 && r.json.recordedMissing === 1 && r.json.unmatchedDocumented === 1 && r.json.orphanFiles === 2 && readFileSync(MC, "utf8") === afterRec, r.json); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom", "--dir", join(ROOT, "acme-c1", "no-such-folder")]); + check("c1 reconcile --dir on a missing folder: dirExists false, nothing recorded, the pathless documented entry named — never an error that hides the state", r.code === 0 && r.json?.dirExists === false && r.json.recorded === 0 && r.json.unmatchedDocumented === 1, r.json); + r = run("reconcile-docs", ["--manifest", MC]); + check("c1 reconcile: --domain is required", r.code === 1 && /--domain/.test(r.stderr), r.stderr); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "__proto__"]); + check("c1 reconcile: the domain-name grammar applies (prototype-shaped name refused)", r.code === 1, r.stderr.slice(0, 120)); +} + rmSync(ROOT, { recursive: true, force: true }); console.log(failures ? `\n${failures} failure(s)` : "\nAll manifest ops checks passed"); diff --git a/plugins/gs-superadmin/test/relationships-build.mjs b/plugins/gs-superadmin/test/relationships-build.mjs index ffab512..3b71203 100644 --- a/plugins/gs-superadmin/test/relationships-build.mjs +++ b/plugins/gs-superadmin/test/relationships-build.mjs @@ -201,7 +201,7 @@ function buildWorkspace(slug, { rulesDepth = "full", withFailedRule = true, chai if (depth === "full") writeFileSync(join(rulesDir, `${r.id}.md`), doc(`rules-engine/${r.id}`, r.id, r.name, r.payload)); // metadata stubs carry no describe payload — an empty stub file stands in else writeFileSync(join(rulesDir, `${r.id}.md`), `# ${r.name}\n\n> Metadata-only stub.\n\n- key: rules-engine/${r.id}\n- id: ${r.id}\n- name: ${r.name}\n`); - manifest(M, "mark", ["--key", `rules-engine/${r.id}`, "--status", "documented", "--depth", depth]); + manifest(M, "mark", ["--key", `rules-engine/${r.id}`, "--status", "documented", "--depth", depth, "--doc-path", join(rulesDir, `${r.id}.md`)]); }); if (withFailedRule) manifest(M, "mark", ["--key", "rules-engine/rule-acme-0004", "--status", "failed", "--error", "simulated describe failure"]); @@ -211,7 +211,7 @@ function buildWorkspace(slug, { rulesDepth = "full", withFailedRule = true, chai manifest(M, "mark", ["--key", `rules-engine-chains/${CHAIN.id}`, "--status", "failed", "--error", "simulated describe failure"]); } else { writeFileSync(join(chainsDir, `${CHAIN.id}.md`), doc(`rules-engine-chains/${CHAIN.id}`, CHAIN.id, CHAIN.name, CHAIN.payload)); - manifest(M, "mark", ["--key", `rules-engine-chains/${CHAIN.id}`, "--status", "documented", "--depth", "full"]); + manifest(M, "mark", ["--key", `rules-engine-chains/${CHAIN.id}`, "--status", "documented", "--depth", "full", "--doc-path", join(chainsDir, `${CHAIN.id}.md`)]); } const scDir = join(ROOT, slug, "scorecard"); @@ -226,7 +226,7 @@ function buildWorkspace(slug, { rulesDepth = "full", withFailedRule = true, chai } else { writeFileSync(join(scDir, `${SCORECARD.id}.md`), `# ${SCORECARD.name}\n\n> Metadata-only stub.\n\n- key: scorecard/${SCORECARD.id}\n- id: ${SCORECARD.id}\n- name: ${SCORECARD.name}\n`); } - manifest(M, "mark", ["--key", `scorecard/${SCORECARD.id}`, "--status", "documented", "--depth", scorecardDepth]); + manifest(M, "mark", ["--key", `scorecard/${SCORECARD.id}`, "--status", "documented", "--depth", scorecardDepth, "--doc-path", join(scDir, `${SCORECARD.id}.md`)]); // journey programs + email-template inventory (program-to-template map). // Template entries stay pending on purpose: the map matches inventory ids @@ -242,13 +242,13 @@ function buildWorkspace(slug, { rulesDepth = "full", withFailedRule = true, chai // so a quoted key missed the inventory lookup, the depth gate was // skipped, and this stub was walked as a full program. writeFileSync(join(joDir, `${p.id}.md`), `# ${p.name}\n\n> Metadata-only stub.\n\n- key: \`journey/${p.id}\`\n- id: ${p.id}\n- name: ${p.name}\n\n\`\`\`json\n${JSON.stringify({ programId: p.id, name: p.name })}\n\`\`\`\n`); - manifest(M, "mark", ["--key", `journey/${p.id}`, "--status", "documented", "--depth", "metadata"]); + manifest(M, "mark", ["--key", `journey/${p.id}`, "--status", "documented", "--depth", "metadata", "--doc-path", join(joDir, `${p.id}.md`)]); continue; } if (p.corrupt) writeFileSync(join(joDir, `${p.id}.md`), `# ${p.name}\n\n- key: journey/${p.id}\n- id: ${p.id}\n- name: ${p.name}\n\n\`\`\`json\n{ corrupt\n\`\`\`\n`); else if (p.compact) writeFileSync(join(joDir, `${p.id}.md`), renderProgramDoc(p.payload, { key: `journey/${p.id}` }).doc); else writeFileSync(join(joDir, `${p.id}.md`), doc(`journey/${p.id}`, p.id, p.name, p.payload)); - manifest(M, "mark", ["--key", `journey/${p.id}`, "--status", "documented", "--depth", "full"]); + manifest(M, "mark", ["--key", `journey/${p.id}`, "--status", "documented", "--depth", "full", "--doc-path", join(joDir, `${p.id}.md`)]); } return M; } @@ -517,7 +517,7 @@ check( mkdirSync(join(slugDir, "jo-programs-x"), { recursive: true }); for (const p of PARSEABLE) { writeFileSync(join(slugDir, "jo-programs-x", `${p.id}.md`), doc(`jo-programs-x/${p.id}`, p.id, p.name, p.payload)); - manifest(M9, "mark", ["--key", `jo-programs-x/${p.id}`, "--status", "documented", "--depth", "full"]); + manifest(M9, "mark", ["--key", `jo-programs-x/${p.id}`, "--status", "documented", "--depth", "full", "--doc-path", join(slugDir, "jo-programs-x", `${p.id}.md`)]); } const OUT9 = join(slugDir, "relationships"); r = run(["--manifest", M9, "--out-dir", OUT9, "--date", "2026-01-15"]); From 3d556bacdfc1058b804c7b1d726e183e276ae2cb Mon Sep 17 00:00:00 2001 From: BradleyDB <23158057+BradleyDB@users.noreply.github.com> Date: Tue, 15 Sep 2026 09:21:05 -0700 Subject: [PATCH 02/10] review round (Session C1): one suffix grammar, blank --doc-path refused, reconcile-docs refuses a wrong CWD, one hasDocPath / docPathFor, report refuses a non-T-2 depth, six prose repairs /code-review medium (8 finders, 5 file-batched verifiers): 16 findings fixed. doc-lib: resolveStem is the ONE -dup grammar the claimer and the matcher both resolve through and both claim from (a matcher that skipped the claim on a null orphaned a deleted doc's collision partner); readDocStems agrees with listMdFiles on `.md`; docDirNorm/docPathFor are the one doc_path spelling (stub, describe-batch, the template writer, reconcile). manifest: hasDocPath type guard for mark/report/remove/reconcile; `--doc-path ""` refused; report refuses a depth outside metadata|full; reconcile-docs refuses a recorded path that does not resolve from this CWD while its file sits in the folder, and treats a different-case folder as the same folder on win32. Prose: F/S/L/U substitution rule, --deep paragraph is a pointer plus the --doc-path clause, Phase 4 relay names a stamped:false domain, refresh's block reads as the byDomain row sum, omits emptyDomains and states a legacy stamp's recording from step 3's dateFieldSource. Arms: doc-lib (+4), manifest-ops (+7). --- dev/FEEDBACK.md | 1 + plugins/gs-superadmin/CHANGELOG.md | 4 +- .../gs-superadmin/scripts/describe-batch.mjs | 4 +- plugins/gs-superadmin/scripts/doc-lib.mjs | 99 ++++++++++++------- plugins/gs-superadmin/scripts/manifest.mjs | 91 ++++++++++++----- plugins/gs-superadmin/skills/refresh/SKILL.md | 38 ++++--- plugins/gs-superadmin/skills/setup/SKILL.md | 36 ++++--- .../gs-superadmin/test/doc-lib-fixtures.mjs | 39 +++++++- plugins/gs-superadmin/test/manifest-ops.mjs | 51 ++++++++++ 9 files changed, 271 insertions(+), 92 deletions(-) diff --git a/dev/FEEDBACK.md b/dev/FEEDBACK.md index 08c6a08..2e72e45 100644 --- a/dev/FEEDBACK.md +++ b/dev/FEEDBACK.md @@ -466,6 +466,7 @@ Severity: normal — an entry marked documented with no recorded doc_path is inv What: `manifest.mjs remove` over a July-crawled sandbox domain (the F-449 sibling arm's step 2, 69 entries) reported `docPaths` 0 while the stub docs existed on disk — the entries carry status `documented` but no `doc_path` (docs written before path recording existed, then deleted by folder). Measured on the sandbox: 25 such entries across three July-crawled domains; zero on prod. `manifest.mjs mark --status documented` accepts a mark with no `doc_path` today (a stub or describe-batch always passes one; only a hand-run mark omits it), so the state is reachable by hand as well as by legacy. Expected: (1) `mark --status documented` refuses when the entry has no recorded `doc_path` and none is passed; (2) `report` and `remove` count documented entries carrying no `doc_path` and say so by number (`docPathsUnknown`), so `remove`'s `docPaths` can never read as "no docs" when the truth is "unknown"; (3) a backfill verb (`manifest.mjs reconcile-docs --domain `) that matches `//*.md` to entries through the doc-name claimer's naming, records `doc_path` for the matches and reports the unmatched on both sides. Scheduled as RP-5 (Session C1) in the round plan; the sandbox's three domains are the live judge. Fix: 2026-09-15 (builder, Session C1 / RP-5; ruling, Bradley 2026-09-15: refuse in mark, count in report and remove, backfill as its OWN verb — report stays read-only by contract) — all three parts as Expected. (1) `mark --status documented` refuses, all-or-nothing and naming the remedy, when an entry has no recorded `doc_path` and none is passed; measured before shipping: every doc writer passes `--doc-path` on the mark after its write (stub and describe-batch by construction, the manual path by its reference), and describe-batch's `--if-changed` skip re-marks WITHOUT a path only on an entry that already carries one (a `typeof entry.doc_path === "string"` conjunct guards that branch), so the only mark refused is the hand-run one — the fixture suites' own hand-run marks were that route and now pass the path. (2) `report` carries `docPathsUnknown` per domain and in total; `remove` carries it beside `docPaths`. (3) `reconcile-docs --domain [--dir ] [--out ] [--dry-run]` matches `//*.md` to the domain's entries through doc-lib's new `docNameMatcher` — the claimer's READ-side twin: `docBaseName`, then the `-dup` chain a writer appends while a stem is claimed in-run or a different spelling of it sits on disk — after pre-claiming every RECORDED path (a recorded doc is never re-attributed); records `doc_path` in the spelling stub writes (folder as passed, forward slashes) and reports `recorded`, `alreadyRecorded`, `recordedMissing` (recorded path, file gone — never rewritten: a deleted doc is the regeneration signal), `unmatchedDocumented` (documented, no doc on disk) and `orphanFiles` (docs no entry claims), samples on stdout and full lists under `--out`; writes the manifest once, only when a path was recorded, and never touches a doc file. Review-gate note: the first cut re-spelled the `-dup` loop inside manifest.mjs and check-doc-drift check 9 refused it — the grammar has one home, so the matcher moved into doc-lib beside the claimer (shared disk snapshot), pinned by 14 arms in test/doc-lib-fixtures.mjs including the hop bound. Legacy count re-measured locally on the sandbox (read-only) before the fixture was written: 20 documented entries with no `doc_path` today across the same three July-crawled domains (3 + 14 + 3), down from the 25 logged — the B-V round's #13 arm re-marked five chain entries with a path; prod 0. Class: `remove.docPaths` asserted "no docs to clean" from a field only the newer writers record. Fixtures: manifest-ops "c1" (the refusal incl. the all-or-nothing keys-file arm and three controls; remove's count; reconcile over a scratch folder — matched, the Co/co collision pair via `co-dup`, recorded-missing, unmatched, two orphans, dry-run, `--out`, idempotent re-run, missing folder). Version 0.40.0. MAINTAINERS.md's manifest row names the verb. +Review round (medium: 8 finders + 5 file-batched verifiers; 13 confirmed, 3 plausible, all fixed on the branch): (1) the matcher returned null on a free stem WITHOUT recording the claim, so a deleted doc's collision partner (co.md gone, Co-dup.md present) read as an orphan — the suffix grammar is now ONE private `resolveStem` in doc-lib that both the claimer and the matcher resolve through and both claim from (the doc-lib comment had also claimed check-doc-drift check 9 sanctions the second copy and that the claimer shares a 64-hop bound; check 9 never reads doc-lib and the claimer was unbounded — both statements gone with the second copy; pinned by a deleted-partner arm). (2) `--doc-path ""` walked past the refusal and recorded nothing — a blank value is now refused as malformed on every status, like every other blank mark value. (3) `reconcile-docs` derived its folder from `--manifest` but recorded paths resolve against the CWD (the writers' and readers' convention), so from any other working directory every doc read missing and every file an orphan — a recorded path that does not resolve while a file of that name sits in the folder is now refused before any write, naming the remedy; on Windows a different-case folder spelling is the same folder for the claim. (4) "has a doc_path" was spelled four ways and remove's truthiness test pushed a hand-edited non-string into the cleanup list — one `hasDocPath` predicate (non-blank string) for mark, report, remove and reconcile; a depth outside T-2's metadata|full is refused loud by report instead of counted as full. (5) the doc_path spelling lived in four places (stub, describe-batch, the template writer, reconcile) — one `docPathFor` in doc-lib, byte-identical; `readDocStems` now agrees with `listMdFiles` on `.md` exact case. (6) prose: the final report's `F S L U` placeholders gained their substitution rule, the `--deep` paragraph's second copy of the fully-ingested test became a pointer and gained the `--doc-path` clause, the Phase 4 relay names an unstamped `--partial` domain separately instead of contradicting its own arithmetic, and refresh's block reads `` as the `byDomain` row sum (there is no total key), omits `emptyDomains`, and states a legacy stamp's recording outcome from step 3's `dateFieldSource` instead of asserting it. Carried, not built: a quiescence sequence on the manifest (`save()` bumps a counter the report echoes) would make "quoted at a quiescent point" checkable instead of prose — a design change for a later round. Class: outcome-from-proxy Judge: independent of the fixer, banked in dev/VALIDATION.md (F-459 section, Session C1): the tester reads `report`'s `docPathsUnknown` on the sandbox (expected 20 across three domains), runs `reconcile-docs --dry-run` then `reconcile-docs` over each of the three (the ONE manifest write this round asks of the tester), reads `docPathsUnknown` again (expected 0, every orphan named), and re-runs the F-449 sibling arm's step 2 SHAPE as a read — `report`'s row for one of those domains showing a complete `docPaths` picture (no `docPathsUnknown`), no `remove` needed. A `docPathsUnknown` that does not reach 0 without an explanation in `unmatchedDocumented` REOPENS this entry. Sibling sweep: outcome-from-proxy recipe, shared with F-455's line above (`remove.docPaths` is this entry's site; `report.byStatus.documented` F-455's; `upsert-batch.unchanged` named there). consumer-parity check on the naming grammar: `git grep -n '"-dup"' -- plugins build` — doc-lib.mjs only (claimer + matcher); check-doc-drift check 9 sanctions no other file, and the first cut's second copy in manifest.mjs is what that check refused. Cost: seconds. diff --git a/plugins/gs-superadmin/CHANGELOG.md b/plugins/gs-superadmin/CHANGELOG.md index 8d56083..47a249a 100644 --- a/plugins/gs-superadmin/CHANGELOG.md +++ b/plugins/gs-superadmin/CHANGELOG.md @@ -36,7 +36,9 @@ quiescent point instead of narrating. [--dry-run]` backfills `doc_path` from the docs on disk through the writers' own naming (doc-lib's `docNameMatcher`, the claimer's read-side twin — same `-dup` collision chain), and names recorded-missing, unmatched and orphan files; it writes only `doc_path`, - never a doc. This is the reconcile-on-resume step batched marks (#12) will reuse. + never a doc, and refuses — before any write — a run from a working directory the + recorded paths do not resolve from. This is the reconcile-on-resume step batched + marks (#12) will reuse. A blank `--doc-path` on `mark` is refused as malformed. - Contract: report's output is now part of T-2 (`GsReport` in manifest.mjs's header, pinned by `test/contract-conformance.mjs`); every addition is additive — `byDomain` rows stay numbers-only because describe-batch's progress probe sums them. diff --git a/plugins/gs-superadmin/scripts/describe-batch.mjs b/plugins/gs-superadmin/scripts/describe-batch.mjs index 5bdbe92..b288860 100644 --- a/plugins/gs-superadmin/scripts/describe-batch.mjs +++ b/plugins/gs-superadmin/scripts/describe-batch.mjs @@ -178,7 +178,7 @@ import { renderDesignerDoc, designerDocProgress, designerDrilldownStats, designerTaskFieldLabels, splitTrailingGroup, parseDocJson, normalizeText, writeFileAtomicSync, readJsonFile, makeCliHelpers, findWorkspaceCatalog, makeCommandResolver, - assertReadOnlyCommand, assertPlainGsAdminCommand, resolveCliArgv, isDescribeRead, isAuthDeath, DESCRIBE_NONE, RECORDED_LANES } from "./doc-lib.mjs"; + assertReadOnlyCommand, assertPlainGsAdminCommand, resolveCliArgv, isDescribeRead, isAuthDeath, docPathFor, DESCRIBE_NONE, RECORDED_LANES } from "./doc-lib.mjs"; const here = dirname(fileURLToPath(import.meta.url)); const MANIFEST_SCRIPT = join(here, "manifest.mjs"); @@ -739,7 +739,7 @@ for (const entry of batch.entries) { // recorded doc_path yet, and the claimer's exact-case reuse is how the next // run finds it. const base = claimBaseName(entry.id); - const relPath = `${outDir.replace(/\\/g, "/").replace(/\/$/, "")}/${base}.md`; + const relPath = docPathFor(outDir, base); // the ONE doc_path spelling (doc-lib; stub and reconcile-docs record the same) // The full-depth documented mark, one spelling for the raw/template/program // path and the designer composite (A-1); a success resets the within-run // failure count (issue #13). diff --git a/plugins/gs-superadmin/scripts/doc-lib.mjs b/plugins/gs-superadmin/scripts/doc-lib.mjs index fc7d1eb..f98675a 100644 --- a/plugins/gs-superadmin/scripts/doc-lib.mjs +++ b/plugins/gs-superadmin/scripts/doc-lib.mjs @@ -245,6 +245,17 @@ export function canonicalFingerprint(payload) { return createHash("sha1").update(JSON.stringify(canon(payload))).digest("hex"); } +// The doc_path SPELLING every writer records and every reader resolves against +// the CWD (T-2 GsInventoryEntry.doc_path): the folder as passed, forward +// slashes, no trailing slash, then `/.md`. One home (review round of +// F-459: stub, describe-batch, the template writer and reconcile-docs each +// carried the same two-replace chain) — a path reconciled from disk is +// byte-identical to one a writer recorded because both call this. +/** @param {string} dir */ +export const docDirNorm = (dir) => dir.replace(/\\/g, "/").replace(/\/$/, ""); +/** @param {string} dir @param {string} stem */ +export const docPathFor = (dir, stem) => `${docDirNorm(dir)}/${stem}.md`; + // Doc filename for an asset id — the single copy; import, never re-implement. // Beyond the charset whitelist, two Windows filename-legality rules force the // hash suffix (F-126): reserved DOS device names (NUL/CON/COM1/…, with or @@ -309,7 +320,12 @@ function readDocStems(dir) { /* dir not created yet — nothing on disk to respect */ } for (const n of names) { - if (!/\.md$/i.test(n)) continue; + // The same extension test as listMdFiles (the readers' listing): a doc is + // `.md`, exact case — every writer emits that, and a `.MD` file is + // invisible to every reader, so it must be no stem here either (review + // round: the matcher would have recorded `.md` for a file spelled + // `.MD`, a path that exists on Windows only). + if (!n.endsWith(".md")) continue; const stem = n.slice(0, -3); const lower = stem.toLowerCase(); if (!disk.has(lower)) disk.set(lower, new Set()); @@ -319,37 +335,61 @@ function readDocStems(dir) { } return disk; } +// The ONE suffix grammar (F-125/F-156; review round of F-459): starting from +// docBaseName(id), append "-dup" while the stem is claimed in-run +// (case-insensitively) or a DIFFERENT spelling of it sits on disk, and stop +// at the first stem that is free — which is either this id's own exact-case +// file (reuse) or a name no file carries (a fresh write). The write side +// (docNameClaimer) and the read side (docNameMatcher) both resolve through +// this function and both record the claim, so a replay lands on the stem +// the writer chose. Termination: each hop lengthens the stem, and both the +// disk snapshot and the claim set are finite. Not exported — the grammar +// has no third home (check-doc-drift check 9 refuses a copy in any script). +/** + * @param {Map>} disk lower-cased stem -> exact on-disk stems + * @param {Set} claimedLower lower-cased stems claimed this run + * @param {string} id + * @returns {string} + */ +function resolveStem(disk, claimedLower, id) { + let base = docBaseName(id); + for (;;) { + const spellings = disk.get(base.toLowerCase()); + const foreignOnDisk = spellings != null && !spellings.has(base); + if (!claimedLower.has(base.toLowerCase()) && !foreignOnDisk) break; + base += "-dup"; + } + return base; +} export function docNameClaimer(dir) { const claimedLower = new Set(); /** @type {Map> | null} */ let disk = null; // lower-cased stem -> Set of exact on-disk stems return (id) => { if (disk === null) disk = readDocStems(dir); - let base = docBaseName(id); - for (;;) { - const spellings = disk.get(base.toLowerCase()); - const foreignOnDisk = spellings != null && !spellings.has(base); - if (!claimedLower.has(base.toLowerCase()) && !foreignOnDisk) break; - base += "-dup"; - } + const base = resolveStem(disk, claimedLower, id); claimedLower.add(base.toLowerCase()); return base; }; } // The claimer's READ-side twin (F-459, manifest.mjs reconcile-docs): which file -// on disk did a writer running the claimer's rule give this id? It replays the -// same grammar — docBaseName, then "-dup" while the stem is claimed in-run or a -// DIFFERENT spelling of it sits on disk — and answers with the exact stem when -// that file exists and no earlier id took it, or null when the writer's choice -// is a name no file carries (no doc for this id). Callers claim() the stems of -// RECORDED paths first, so a replay never re-attributes a doc whose owner is a -// fact; unclaimed() is what remains — the orphans. Same residual as the -// claimer (two ids whose docBaseName output is byte-identical resolve in call -// order), and the same 64-hop bound a pathological folder could otherwise -// defeat. The suffix loop is written once more here on purpose: it is the -// grammar's second, read-side home, in the file check-doc-drift check 9 -// sanctions for it — never copy it into a script. +// on disk did a writer running the claimer's rule give this id? It resolves +// through the same resolveStem and answers with that stem when the file exists +// (exact case) or null when the writer's choice is a name no file carries (no +// doc for this id) — and it records the claim EITHER way, exactly as the +// writer did, so a later id whose stem collides hops past it even when the +// earlier id's doc was deleted (review round: a matcher that skipped the claim +// on a null read a deleted doc's collision partner as an orphan). Callers +// claim() the stems of RECORDED paths first, so a replay never re-attributes +// a doc whose owner is a fact; unclaimed() is what remains — the orphans. +// Residual, shared with the claimer: the replay agrees with the writer only +// in the writer's claim ORDER — two ids whose docBaseName output collides +// case-insensitively were named in list-payload order, and manifest.mjs +// replays in key order; the pair resolves to the same two files either way +// when both docs exist, and to the surviving file when one was deleted only +// if the replay order matches the writing order. Exact-name residual as the +// claimer's (byte-identical docBaseName output resolves in call order). /** * @param {string} dir * @returns {{ match: (id: string) => string|null, claim: (stem: string) => void, unclaimed: () => string[] }} @@ -368,20 +408,9 @@ export function docNameMatcher(dir) { return { claim, match(id) { - let base = docBaseName(id); - for (let hops = 0; hops < 64; hops++) { - const lower = base.toLowerCase(); - const spellings = disk.get(lower); - if (!claimedLower.has(lower)) { - if (spellings !== undefined && spellings.has(base)) { - claim(base); - return base; - } - if (spellings === undefined) return null; // the writer's first free choice — and no file carries it - } - base += "-dup"; - } - return null; + const base = resolveStem(disk, claimedLower, id); + claim(base); + return disk.get(base.toLowerCase())?.has(base) ? base : null; }, unclaimed: () => [...disk.values()].flatMap((s) => [...s]).filter((stem) => !claimed.has(stem)).sort(), }; @@ -1386,7 +1415,7 @@ export function runDocGenerator({ scriptName, render, argv }) { const base = claimName(id); const path = resolve(outDir, `${base}.md`); writeFileSync(path, doc, "utf8"); - written.push({ id, path: `${outDir.replace(/\\/g, "/").replace(/\/$/, "")}/${base}.md`, bytes: Buffer.byteLength(doc) }); + written.push({ id, path: docPathFor(outDir, base), bytes: Buffer.byteLength(doc) }); } catch (e) { failed.push({ file, error: e instanceof Error ? e.message : String(e) }); } diff --git a/plugins/gs-superadmin/scripts/manifest.mjs b/plugins/gs-superadmin/scripts/manifest.mjs index 49eba19..1b22fce 100644 --- a/plugins/gs-superadmin/scripts/manifest.mjs +++ b/plugins/gs-superadmin/scripts/manifest.mjs @@ -428,7 +428,7 @@ import { fileURLToPath } from "node:url"; // Doc filename for an asset id — the shared copy every doc writer imports // (sanitized ids get a short raw-id hash so distinct ids that clean to the // same base can never collide). -import { docNameClaimer, docNameMatcher, readJsonFile, writeFileAtomicSync, cmpKey, getPath as get, findItemsArray, extractIds, makeCliHelpers, STUB_MARKER, DESCRIBE_NONE, idPathHint, ZERO_RESOLVE_HEAD, findWorkspaceCatalog, makeCommandResolver } from "./doc-lib.mjs"; +import { docNameClaimer, docNameMatcher, docPathFor, docDirNorm, readJsonFile, writeFileAtomicSync, cmpKey, getPath as get, findItemsArray, extractIds, makeCliHelpers, STUB_MARKER, DESCRIBE_NONE, idPathHint, ZERO_RESOLVE_HEAD, findWorkspaceCatalog, makeCommandResolver } from "./doc-lib.mjs"; const here = dirname(fileURLToPath(import.meta.url)); // The date-failure head refresh/SKILL.md's post-upgrade migration note quotes @@ -510,6 +510,16 @@ function describeStateOf(stamp) { const recorded = stamp != null && typeof stamp === "object" && typeof stamp.describeCommand === "string" ? stamp.describeCommand : null; return recorded === DESCRIBE_NONE ? "list-only" : recorded ? "describable" : "unrecorded"; } +// "This entry knows where its doc is" — the ONE predicate (review round of +// F-459: mark, report, remove and reconcile-docs each spelled it, and remove's +// truthiness test pushed a hand-edited non-string into the cleanup list). A +// usable doc_path is a non-blank string; anything else is unknown. +/** @param {GsInventoryEntry} e @returns {e is GsInventoryEntry & {doc_path: string}} */ +const hasDocPath = (e) => typeof e.doc_path === "string" && e.doc_path.trim() !== ""; +// The describe-state → byDepth bucket for a metadata stub (F-455): frozen and +// null-prototype (F-225 tenet) — the stub banner says the same three things. +/** @type {Readonly>} */ +const STUB_BUCKET = Object.freeze(Object.assign(Object.create(null), { "list-only": "listOnly", describable: "metadata", unrecorded: "unrecorded" })); // The three enumerations T-2 declares as unions. Each Set is TYPED AGAINST its // union, so a member added here but not to the typedef is red under both // configs (the sync mechanism A-2 asks for — measured at the 0.36.3 gate: with @@ -1399,6 +1409,10 @@ if (verb === "init") { fail("--fingerprint must be a 40-char hex sha1 digest"); } const docPath = opt("--doc-path"); + // A blank value is a malformed one, like every other value-bearing mark flag + // (review round: `--doc-path ""` walked past the F-459 refusal below and + // recorded nothing — the state the refusal exists to prevent). + if (docPath !== undefined && docPath.trim() === "") fail("--doc-path requires a path — pass where the doc landed"); // A fingerprint identifies ONE describe payload and a doc_path ONE file — // recording the same value on a whole batch is always a corruption, so the // batch mode refuses rather than fans out. @@ -1443,7 +1457,7 @@ if (verb === "init") { // whose stubs sat on disk). Checked before the loop: all-or-nothing, like // the unknown-key rule above. if (status === "documented" && docPath === undefined) { - const pathless = keys.filter((k) => typeof m.inventory[k].doc_path !== "string" || m.inventory[k].doc_path === ""); + const pathless = keys.filter((k) => !hasDocPath(m.inventory[k])); if (pathless.length) { const shown = pathless.slice(0, 5).join(", "); fail( @@ -1587,7 +1601,7 @@ if (verb === "init") { // its name, whatever order the passes ran in. (The old "~" suffix here // also sat outside docBaseName's A-Za-z0-9._- charset.) const base = claimName(id); - const relPath = `${outDir.replace(/\\/g, "/").replace(/\/$/, "")}/${base}.md`; + const relPath = docPathFor(outDir, base); const name = (nameField ? get(it, nameField) : null) ?? e.name ?? String(id); const fields = Object.entries(it) .filter(([, v]) => v === null || ["string", "number", "boolean"].includes(typeof v)) @@ -1636,7 +1650,8 @@ if (verb === "init") { // the output shape is unchanged.) const byStatus = Object.create(null); const byDomain = Object.create(null); - for (const e of Object.values(m.inventory)) { + const entries = Object.values(m.inventory); // one materialization for both folds (report is read-only; nothing mutates the map between them) + for (const e of entries) { byStatus[e.status] = (byStatus[e.status] ?? 0) + 1; (byDomain[e.domain] ??= Object.create(null))[e.status] = (byDomain[e.domain][e.status] ?? 0) + 1; } @@ -1666,19 +1681,22 @@ if (verb === "init") { domains[d] = { stamped, describeState: describeStateOf(stamp), byDepth: newDepth(), changeDetection, datelessEntries: 0, docPathsUnknown: 0 }; } let docPathsUnknown = 0; - for (const e of Object.values(m.inventory)) { + for (const e of entries) { const row = domains[e.domain]; if (e.modified_date == null) row.datelessEntries++; if (e.status !== "documented") continue; - if (typeof e.doc_path !== "string" || e.doc_path === "") { row.docPathsUnknown++; docPathsUnknown++; } + if (!hasDocPath(e)) { row.docPathsUnknown++; docPathsUnknown++; } // A metadata stub's completeness is its DOMAIN's recorded describe state - // (the stub banner says the same three things); a full doc is full - // wherever it sits; no depth at all = a pre-depth-field doc = full (the - // stub verb's hasFullDoc rule — stubs have always carried a depth). - const bucket = - e.depth === "metadata" - ? { "list-only": "listOnly", describable: "metadata", unrecorded: "unrecorded" }[row.describeState] - : "full"; + // (STUB_BUCKET — the stub banner says the same three things); a full doc + // is full wherever it sits; no depth at all = a pre-depth-field doc = full + // (the stub verb's hasFullDoc rule — stubs have always carried a depth). + // Any other value is outside T-2 — only a hand edit can put one there — + // and is refused loud rather than bucketed as full (review round: a + // catch-all read `depth: "partial"` as a completed describe). + if (e.depth != null && !isDepth(e.depth)) { + fail(`inventory entry ${e.domain}/${e.id} carries depth "${String(e.depth).slice(0, 40)}" — T-2 allows metadata|full (or none); the manifest was hand-edited, and report will not guess its completeness`); + } + const bucket = e.depth === "metadata" ? STUB_BUCKET[row.describeState] : "full"; row.byDepth[bucket]++; byDepth[bucket]++; } @@ -1705,7 +1723,7 @@ if (verb === "init") { // own output (F-427): a legacy manifest with no key reads null here, // the same as one initialised without --environment. environment: m.environment ?? null, - total: Object.keys(m.inventory).length, + total: entries.length, last_refresh: m.last_refresh, byStatus, byDomain, @@ -2016,7 +2034,7 @@ if (verb === "init") { continue; } const e = m.inventory[k]; - if (e.doc_path) docPaths.push(e.doc_path); + if (hasDocPath(e)) docPaths.push(e.doc_path); else if (e.status === "documented") docPathsUnknown++; delete m.inventory[k]; removed++; @@ -2049,40 +2067,67 @@ if (verb === "init") { // slashes, no trailing slash — so a path reconciled here is byte-identical // to one a writer would have recorded (readers resolve it against the CWD). const dirArg = opt("--dir") ?? join(dirname(manifestPath), domain); - const dirNorm = dirArg.replace(/\\/g, "/").replace(/\/$/, ""); + const dirNorm = docDirNorm(dirArg); const dirAbs = resolve(dirArg); const dirExists = existsSync(dirAbs); const matcher = docNameMatcher(dirAbs); + // Folder identity for the "is this recorded path inside the folder" test: + // case-insensitive on Windows, where `acme/DOM` and `acme/dom` are one + // directory (review round: a stub written under a model-typed `--out-dir` + // of different case read as an orphan of itself). + /** @param {string} a @param {string} b */ + const sameDir = (a, b) => (process.platform === "win32" ? a.toLowerCase() === b.toLowerCase() : a === b); const keys = Object.keys(m.inventory).filter((k) => m.inventory[k].domain === domain).sort(cmpKey); /** @type {string[]} */ const recordedKeys = []; /** @type {string[]} */ const recordedMissing = []; /** @type {string[]} */ const unmatchedDocumented = []; let alreadyRecorded = 0; + /** @type {string[]} recorded paths that do not resolve from this CWD while their file sits in the folder */ + const cwdMismatch = []; // Pass 1: entries that already carry a path claim their stem first (a // recorded path is a fact; a name replay must never re-attribute it). + // Recorded paths resolve against the CWD — the convention every writer + // records by and every reader (describe-batch's --if-changed gate) resolves + // by — while the folder above was derived from --manifest; the two agree + // only from the workspace root, so a recorded path that does NOT resolve + // here while a file of that name sits in the folder is a wrong CWD, not a + // deleted doc, and the verb refuses rather than reporting every doc missing + // and every file an orphan (review round, reproduced). for (const k of keys) { const e = m.inventory[k]; - if (typeof e.doc_path !== "string" || e.doc_path === "") continue; + if (!hasDocPath(e)) continue; + const rel = docDirNorm(e.doc_path); + const slash = rel.lastIndexOf("/"); + const stem = slash >= 0 && rel.endsWith(".md") ? rel.slice(slash + 1, -3) : null; if (existsSync(resolve(e.doc_path))) { alreadyRecorded++; - const rel = e.doc_path.replace(/\\/g, "/"); - const slash = rel.lastIndexOf("/"); - const inDir = slash >= 0 && resolve(rel.slice(0, slash)) === dirAbs; - if (inDir && /\.md$/i.test(rel)) matcher.claim(rel.slice(slash + 1, -3)); + const inDir = slash >= 0 && sameDir(resolve(rel.slice(0, slash)), dirAbs); + if (inDir && stem !== null) matcher.claim(stem); + } else if (stem !== null && existsSync(join(dirAbs, `${stem}.md`))) { + cwdMismatch.push(k); } else { recordedMissing.push(k); } } + if (cwdMismatch.length) { + const shown = cwdMismatch.slice(0, 3).map((k) => `${k} → ${m.inventory[k].doc_path}`).join("; "); + fail( + `${cwdMismatch.length} recorded doc_path${cwdMismatch.length === 1 ? "" : "s"} in domain ${domain} do not resolve from this working directory ` + + `while a file of that name sits in ${dirNorm} (${shown}${cwdMismatch.length > 3 ? "; …" : ""}). doc_path values are recorded relative ` + + `to the workspace root: run reconcile-docs from there with a relative --manifest, or pass --dir spelled the way the recorded ` + + `paths are — nothing was written.` + ); + } // Pass 2: the name replay for entries with no path. for (const k of keys) { const e = m.inventory[k]; - if (typeof e.doc_path === "string" && e.doc_path !== "") continue; + if (hasDocPath(e)) continue; const match = matcher.match(e.id); if (match === null) { if (e.status === "documented") unmatchedDocumented.push(k); continue; } - if (!dryRun) e.doc_path = `${dirNorm}/${match}.md`; + if (!dryRun) e.doc_path = docPathFor(dirArg, match); recordedKeys.push(k); } const orphanFiles = matcher.unclaimed().map((s) => `${s}.md`); diff --git a/plugins/gs-superadmin/skills/refresh/SKILL.md b/plugins/gs-superadmin/skills/refresh/SKILL.md index 137ac21..f45f4fe 100644 --- a/plugins/gs-superadmin/skills/refresh/SKILL.md +++ b/plugins/gs-superadmin/skills/refresh/SKILL.md @@ -41,11 +41,9 @@ node .gs-superadmin/plugin/scripts/manifest.mjs report --manifest /_manife ``` If the manifest does not exist, or `total` is 0 with `last_refresh` null (setup was interrupted before Phase 4 completed), tell the user to run `/gs-superadmin:setup` first and stop. -Keep this report's `domains` block: step 4 names from it every domain whose changes -this run could not check (F-451) — `changeDetection` (`date` = a modified-date field -is recorded; `none` = recorded-none; `unrecorded` = a legacy stamp this run's upsert -records for the first time, step 3) and `datelessEntries` (entries with no stored -date, which the date comparison can never flag). +Keep this report's `domains` block: step 4 reads its `changeDetection` (`date` / +`none` / `unrecorded`) and `datelessEntries` to name every domain whose changes this +run could not check (F-451) — the glossary and the per-line rules are at step 4. **Environment backfill** (once per legacy manifest): if `report` above shows `environment: null` (a manifest that predates the field, or one initialised without it), @@ -253,15 +251,24 @@ passing `--list-command` on this refresh's upserts (above) is the backfill. ### 4 — Report The `Not checked for change this run:` block is read from step 1's report `domains` -block — one line per domain whose `changeDetection` is `none` (no date field: the -whole domain sits outside change detection), one per domain whose `changeDetection` -is `unrecorded` (a legacy stamp; step 3's upsert recorded the field this run and -change detection starts next run), and one per domain with `changeDetection: date` -and `datelessEntries > 0` (rows the comparison could not see — the list-invisible -recovery path registers such entries). `` is that domain's `datelessEntries` and -`` its `byDomain` total. When no domain qualifies the block is the single word -`none` — never omitted, never silence: "Unchanged" above counts only what the date -comparison could see (F-451). +block, one line per qualifying domain, domains sorted by name within each kind: +- `changeDetection: none` (recorded-none: no date field, the whole domain sits outside + change detection) — the first line. +- `changeDetection: unrecorded` (a legacy stamp: no date field when this run started, + so nothing could be compared this run) — whether the field got recorded THIS run is + not knowable from step 1's report; read that domain's step 3 `upsert-batch` output: + `dateFieldSource: "explicit"` or `"explicit-none"` = recorded this run, emit the + second line with `` = the field name, or the word `none` for + `"explicit-none"`; `"unrecorded"`, or no step 3 upsert for that domain at all = still + unrecorded, emit the third line (it will appear in this block next run too). +- `changeDetection: date` with `datelessEntries > 0` (rows the comparison could not + see — the list-invisible recovery path registers such entries) — the fourth line. +`` is that domain's `datelessEntries`; `` is the sum of that domain's `byDomain` +row values (`byDomain.` is a status→count map and carries no total key). A +domain in `emptyDomains` has no `byDomain` row — it holds zero entries — and gets no +line at all. When no domain qualifies the block is the single word `none` — never +omitted, never silence: "Unchanged" above counts only what the date comparison could +see (F-451). ``` ✓ gs-superadmin refresh complete Lookback: days @@ -270,7 +277,8 @@ comparison could see (F-451). Unchanged: Z assets Not checked for change this run: — no date field recorded; entries outside change detection - — date field recorded this run (legacy stamp); detection starts next refresh + — no date field when this run started (legacy stamp); recorded this run as ; detection starts next refresh + — no date field recorded (legacy stamp); still unrecorded — this run's upsert passed neither --date-field nor --no-date-field — of entries carry no date under the recorded field ``` diff --git a/plugins/gs-superadmin/skills/setup/SKILL.md b/plugins/gs-superadmin/skills/setup/SKILL.md index cf4168e..c1a8cf3 100644 --- a/plugins/gs-superadmin/skills/setup/SKILL.md +++ b/plugins/gs-superadmin/skills/setup/SKILL.md @@ -455,11 +455,15 @@ node .gs-superadmin/plugin/scripts/manifest.mjs report --manifest /_manife ``` Relay the **per-domain counts** from the report, not just the total, and take every domain count from the report's `domainCounts` — `indexed` (coverage stamps), -`withAssets`, `empty` — never from a count of `byDomain`'s keys (F-454: the two differ -by exactly the empty domains, so a key count under-reports what the manifest holds): -"Indexed N assets across M domains — M = `domainCounts.indexed`, of which W hold -assets and E are listed but empty (domain: count, …). X pending, Y stale, Z -documented." Name each of the report's `emptyDomains` explicitly ("domain: 0 — listed, +`withAssets`, `empty` — never from a count of `byDomain`'s keys (F-454: a key count +omits every listed-but-empty domain and can include an unstamped one, so it is not the +manifest's domain count): "Indexed N assets across M domains — M = +`domainCounts.indexed`, of which W hold assets and E are listed but empty (domain: +count, …). X pending, Y stale, Z documented." When W + E exceeds M, a domain holds +entries with no coverage stamp (a `--partial` registration into a never-listed domain): +find it as the `domains` row with `stamped: false` and name it separately ("domain: +count — registered, never listed; no coverage stamp"), never folded into W. Name each +of the report's `emptyDomains` explicitly ("domain: 0 — listed, tenant has none") and the exclusion ledger ("K list commands deliberately excluded — reasons recorded in `domains_excluded`"). Treat any domain whose count lands exactly on a common default page size (20, 25, 50) as **suspect** — re-verify its pagination was exhausted before @@ -509,13 +513,15 @@ block Phase 6, and the report names them "list-only (complete)", never **`--deep ` runs** — select the stubs awaiting full ingest with the `next --upgrade` invocation (`references/document-mechanics.md` §2), then follow the standard describe path below, overwriting each stub file (the entry's -`doc_path`) and marking `--status documented --depth full`. When the run's last -batch has exited, the statement that the domain is fully ingested is the quiescent -`report` quoted at the close of this phase (below), read at `domains..byDepth`: -fully ingested means `metadata: 0` and `unrecorded: 0` on that row with no `pending` -or `stale` in its `byDomain` row — relay the row, never a narrative "deep crawl -finished" (F-455: eight domains were entirely stubs when that sentence was last -relayed from memory). +`doc_path`) and marking `--status documented --depth full` — with `--doc-path`, per +`references/document-mechanics.md` §3 (a legacy stub may carry no recorded path, and +`mark` refuses a pathless documented mark; the batch script passes it itself). When +the run's last batch has exited, the statement that the domain is fully ingested is +the quiescent `report` quoted at the close of this phase (below), read at +`domains..byDepth` — the row must show no stubs left awaiting `--deep` and +nothing unrecorded; that close states the exact bucket test. Relay the row, never a +narrative "deep crawl finished" (F-455: eight domains were entirely stubs when that +sentence was last relayed from memory). **All other domains (and deep crawl) — describe each entry in the batch.** Default execution is the sanctioned batch script (below); the manual per-asset path it @@ -675,8 +681,10 @@ a pointer from the relationships file if useful), never inline in generated file ## Final report Every number below is read from the quiescent `report` quoted at Phase 5's close -(never from a running tally): `N` and `M` from `byStatus`; the `Depth:` line from -`byDepth`, all four buckets, each spelled even when 0; the `Not complete:` line lists +(never from a running tally): `N` and `M` from `byStatus`; on the `Depth:` line +substitute `F` · `S` · `L` · `U` with `byDepth.full` · `byDepth.metadata` · +`byDepth.listOnly` · `byDepth.unrecorded` respectively, each spelled even when 0, +leaving the words that follow each number exactly as written; the `Not complete:` line lists every domain whose `domains..byDepth` has `metadata > 0` or `unrecorded > 0`, as ` ( metadata stubs)` or ` ( unrecorded — record a describe command or none)`, comma-separated, or the word `none` when no domain qualifies. diff --git a/plugins/gs-superadmin/test/doc-lib-fixtures.mjs b/plugins/gs-superadmin/test/doc-lib-fixtures.mjs index 31891b4..132b0d1 100644 --- a/plugins/gs-superadmin/test/doc-lib-fixtures.mjs +++ b/plugins/gs-superadmin/test/doc-lib-fixtures.mjs @@ -28,7 +28,7 @@ import { pathToFileURL, fileURLToPath } from "node:url"; import { dirname } from "node:path"; import { makeTempDir, removeTempDir, writeFiles, runNode } from "../../../test/rig.mjs"; import { - stripBom, normalizeText, readJsonFile, codePointSlice, docBaseName, docNameClaimer, docNameMatcher, + stripBom, normalizeText, readJsonFile, codePointSlice, docBaseName, docNameClaimer, docNameMatcher, docPathFor, docDirNorm, canonicalFingerprint, getPath, extractIds, findItemsArray, makeCliHelpers, requireKbDir, sq, shq, needsPosixQuoteCaveat, normTerm, termKey, cmpName, cmpKey, stripLauncherSuffix, extractFencedJson, topBullets, findWorkspaceCatalog, findWorkspaceDir, @@ -323,10 +323,45 @@ check("normalizeText: interior BOM preserved (F-113)", normalizeText("a" + BOM + const m7 = docNameMatcher(deep); let claimedStem = "d"; for (let i = 0; i < 64; i++) { m7.claim(claimedStem); claimedStem += "-dup"; } - check("matcher: 64 claimed hops hit the bound → null, no infinite loop", m7.match("d") === null); + check("matcher: 64 claimed stems resolve past the chain to a name no file carries → null, terminating", m7.match("d") === null); } finally { removeTempDir(deep); } + // Review round (F-459): the matcher records its claim on a null too, as the + // writer did — so a deleted doc's collision partner is still found. Writer + // order "co" then "Co" (the replay's cmpKey order): co.md, Co-dup.md; then + // co.md is deleted (the regeneration signal). Replay: co → null (claimed), + // Co → hops past the claimed "co" → Co-dup — its real file; no orphans. + const gone = makeTempDir("doclib-matcher-gone"); + try { + const w = docNameClaimer(gone); + writeFileSync(join(gone, `${w("co")}.md`), "x"); + writeFileSync(join(gone, `${w("Co")}.md`), "x"); + rmSync(join(gone, "co.md")); + const m8 = docNameMatcher(gone); + const first = m8.match("co"); + const second = m8.match("Co"); + check("matcher: a null read still claims the stem — the collision partner's -dup doc is attributed, not orphaned", first === null && second === "Co-dup" && m8.unclaimed().length === 0, { first, second, unclaimed: m8.unclaimed() }); + } finally { + removeTempDir(gone); + } + // Extension agreement with listMdFiles: a `.MD` file is no stem (every + // writer emits `.md`; every reader lists `.md`), so the matcher neither + // attributes it nor records a `.md` path for a file spelled otherwise. + const ext = makeTempDir("doclib-matcher-ext"); + try { + writeFileSync(join(ext, "Foo.MD"), "x"); + writeFileSync(join(ext, "bar.md"), "x"); + const m9 = docNameMatcher(ext); + check("matcher: .MD is invisible (listMdFiles agreement) — Foo → null, bar matches, no .MD orphan", m9.match("Foo") === null && m9.match("bar") === "bar" && m9.unclaimed().length === 0, m9.unclaimed()); + } finally { + removeTempDir(ext); + } + // The ONE doc_path spelling (docPathFor): folder as passed, forward + // slashes, no trailing slash, then /.md — the writers' exact form. + const BSL = String.fromCharCode(92); + check("docPathFor: backslashes folded, trailing slash dropped", docPathFor(`acme${BSL}dom${BSL}`, "a-1") === "acme/dom/a-1.md" && docPathFor("acme/dom/", "a-1") === "acme/dom/a-1.md" && docPathFor("acme/dom", "a-1") === "acme/dom/a-1.md"); + check("docPathFor: degenerate folders keep the writers' spelling (root and empty)", docPathFor("/", "x") === "/x.md" && docPathFor("", "x") === "/x.md" && docDirNorm("C:/") === "C:"); } finally { removeTempDir(dir); } diff --git a/plugins/gs-superadmin/test/manifest-ops.mjs b/plugins/gs-superadmin/test/manifest-ops.mjs index 3a335be..a8a6128 100644 --- a/plugins/gs-superadmin/test/manifest-ops.mjs +++ b/plugins/gs-superadmin/test/manifest-ops.mjs @@ -1619,6 +1619,13 @@ check("fingerprint: malformed value rejected (exit 1)", r.code === 1 && /40-char check("c1 mark (control): a failed mark on a pathless entry is not refused — only documented needs a path", r.code === 0 && djC().inventory["legacy-dom/l-1"].status === "failed", r); r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-1", "--status", "documented", "--depth", "metadata", "--doc-path", join(legacyDir, "l-1.md")]); check("c1 mark (control): the same mark WITH --doc-path is accepted and records it", r.code === 0 && /l-1\.md$/.test(djC().inventory["legacy-dom/l-1"].doc_path), r); + // Review round: a blank --doc-path is malformed, like every other blank mark + // value — it neither bypasses the refusal nor records a blank. + const beforeBlank = readFileSync(MC, "utf8"); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-3", "--status", "documented", "--doc-path", ""]); + check("c1 mark: --doc-path \"\" is refused as malformed (exit 1), nothing written", r.code === 1 && /--doc-path requires a path/.test(r.stderr) && readFileSync(MC, "utf8") === beforeBlank, r.stderr.slice(0, 120)); + r = run("mark", ["--manifest", MC, "--key", "legacy-dom/l-3", "--status", "failed", "--error", "x", "--doc-path", " "]); + check("c1 mark: a whitespace --doc-path is refused on any status", r.code === 1 && /--doc-path requires a path/.test(r.stderr), r.stderr.slice(0, 120)); { const m = djC(); delete m.inventory["legacy-dom/l-1"].doc_path; writeFileSync(MC, JSON.stringify(m, null, 2)); } // back to the legacy shape for the reconcile arms // remove (F-459 part 2): a removed pathless documented entry is COUNTED, so @@ -1627,6 +1634,26 @@ check("fingerprint: malformed value rejected (exit 1)", r.code === 1 && /40-char check("c1 remove: pathless documented entry removed — docPaths [] but docPathsUnknown 1", r.code === 0 && r.json?.removed === 1 && r.json.docPaths.length === 0 && r.json.docPathsUnknown === 1, r.json); r = run("remove", ["--manifest", MC, "--key", "full-dom/a-1"]); check("c1 remove (control): an entry with a recorded path — docPaths names it, docPathsUnknown 0", r.code === 0 && r.json?.docPaths?.length === 1 && r.json.docPathsUnknown === 0, r.json); + // Review round: report, remove and mark decide "has a doc_path" through ONE + // predicate — a hand-edited non-string is unknown everywhere, never pushed + // into the cleanup list; a depth outside T-2 is refused, never read as full. + { + const m = djC(); + m.inventory["full-dom/a-3"].status = "documented"; + m.inventory["full-dom/a-3"].doc_path = 5; + writeFileSync(MC, JSON.stringify(m, null, 2)); + r = run("report", ["--manifest", MC]); + check("c1 report: a non-string doc_path counts as unknown", r.json?.domains?.["full-dom"]?.docPathsUnknown === 1, r.json?.domains?.["full-dom"]); + r = run("remove", ["--manifest", MC, "--key", "full-dom/a-3"]); + check("c1 remove: a non-string doc_path is counted unknown, never pushed into docPaths", r.code === 0 && r.json?.docPaths?.length === 0 && r.json.docPathsUnknown === 1, r.json); + const m2 = djC(); + m2.inventory["full-dom/a-2"].depth = "partial"; + writeFileSync(MC, JSON.stringify(m2, null, 2)); + r = run("report", ["--manifest", MC]); + check("c1 report: a depth outside metadata|full is refused loud (exit 1, names the entry), never bucketed as full", r.code === 1 && /full-dom\/a-2/.test(r.stderr) && /"partial"/.test(r.stderr), r.stderr.slice(0, 160)); + delete m2.inventory["full-dom/a-2"].depth; + writeFileSync(MC, JSON.stringify(m2, null, 2)); + } // reconcile-docs (F-459 part 3) over the legacy folder: l-1.md, Co.md and // co-dup.md match pathless entries; l-2's recorded file is deleted (recorded @@ -1652,6 +1679,30 @@ check("fingerprint: malformed value rejected (exit 1)", r.code === 1 && /40-char const afterRec = readFileSync(MC, "utf8"); r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom"]); check("c1 reconcile: idempotent — a re-run records 0, alreadyRecorded 3, same missing/unmatched/orphans, manifest untouched", r.code === 0 && r.json?.recorded === 0 && r.json.alreadyRecorded === 3 && r.json.recordedMissing === 1 && r.json.unmatchedDocumented === 1 && r.json.orphanFiles === 2 && readFileSync(MC, "utf8") === afterRec, r.json); + // Review round: a recorded path that does not resolve from this CWD while + // its file sits in the folder is a wrong working directory, not a deleted + // doc — refused before any write, never reported as missing + orphan. + { + const m = djC(); + m.inventory["legacy-dom/Co"].doc_path = "acme-c1/legacy-dom/Co.md"; // workspace-relative spelling; this suite's CWD is not the workspace + writeFileSync(MC, JSON.stringify(m, null, 2)); + const beforeCwd = readFileSync(MC, "utf8"); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom"]); + check("c1 reconcile: CWD mismatch refused (exit 1, names the entry and the remedy), nothing written", r.code === 1 && /do not resolve from this working directory/.test(r.stderr) && /legacy-dom\/Co/.test(r.stderr) && readFileSync(MC, "utf8") === beforeCwd, r.stderr.slice(0, 200)); + m.inventory["legacy-dom/Co"].doc_path = `${fwd(legacyDir)}/Co.md`; + writeFileSync(MC, JSON.stringify(m, null, 2)); + } + if (process.platform === "win32") { + // Windows: a recorded path under a different-case folder spelling is the + // same directory — its stem is claimed, never an orphan of itself. + const m = djC(); + m.inventory["legacy-dom/Co"].doc_path = `${fwd(legacyDir).replace(/legacy-dom$/, "LEGACY-DOM")}/Co.md`; + writeFileSync(MC, JSON.stringify(m, null, 2)); + r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom", "--dry-run"]); + check("c1 reconcile (win32): a case-different folder spelling claims its stem — Co.md is not an orphan", r.code === 0 && r.json?.alreadyRecorded === 3 && !r.json.samples.orphanFiles.includes("Co.md"), r.json?.samples); + m.inventory["legacy-dom/Co"].doc_path = `${fwd(legacyDir)}/Co.md`; + writeFileSync(MC, JSON.stringify(m, null, 2)); + } r = run("reconcile-docs", ["--manifest", MC, "--domain", "legacy-dom", "--dir", join(ROOT, "acme-c1", "no-such-folder")]); check("c1 reconcile --dir on a missing folder: dirExists false, nothing recorded, the pathless documented entry named — never an error that hides the state", r.code === 0 && r.json?.dirExists === false && r.json.recorded === 0 && r.json.unmatchedDocumented === 1, r.json); r = run("reconcile-docs", ["--manifest", MC]); From 368664dec6b8015faaab493589eae9bfbfe3cc12 Mon Sep 17 00:00:00 2001 From: BradleyDB <23158057+BradleyDB@users.noreply.github.com> Date: Tue, 15 Sep 2026 09:23:18 -0700 Subject: [PATCH 03/10] handoff hb-20260915-01 (round-c1-report-truth) --- dev/FEEDBACK.md | 4 ++-- plugins/gs-superadmin/skills/dev-canary/SKILL.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/dev/FEEDBACK.md b/dev/FEEDBACK.md index 2e72e45..84ff5f8 100644 --- a/dev/FEEDBACK.md +++ b/dev/FEEDBACK.md @@ -146,8 +146,8 @@ what the gate counts (F-444). The gate refuses both. Sections from F-360 on; the rule also reads the archive. -Under test: dev · hb-20260914-02 · 2026-09-14 -Blind spots (hb-20260914-02): post-merge close-out of PR 19 on dev: nothing new is under test; the Session B arms were measured on the sandbox by the tester round (F-456 and F-458 CLEARED). Still unmeasured on any tenant, all fixture-only: the tester's own blind spots — F-456's --upgrade selection over a metadata-depth chain and the refuse direction (trimmed catalog, PUT twin); F-458's designer mid-drilldown death, the three sibling auth literals, and an oversized-batch rig from a fresh token. F-459 (section logged today, OPEN) awaits Session C1, plus everything Sessions C1 onward bank. +Under test: round-c1-report-truth · hb-20260915-01 · 2026-09-15 +Blind spots (hb-20260915-01): no live tenant call was needed and none was made: the report reads over both real manifests were measured locally (read-only) and the counts reproduced, but the ONE manifest write this round adds — reconcile-docs over the sandbox's three legacy domains (20 pathless entries today) — is the tester's, banked in dev/VALIDATION.md; the setup and refresh walks (both slash-only) are unwalked — no CI executes skill prose, so the quoted-report relay shapes rest on the fixtures and the prose; the mark refusal's only reachable route (a hand-run documented mark on a pathless legacy entry) and the reconcile CWD-mismatch refusal are fixture-only; the win32 folder-case arm ran on this host only, the Linux/macOS legs rest on CI