From 648ad5c6be580d74dcc8facf9387a3f7f88727eb Mon Sep 17 00:00:00 2001 From: aarroyo Date: Thu, 30 Jul 2026 09:23:07 -0500 Subject: [PATCH] =?UTF-8?q?fix(standards):=20one=20renderer,=20two=20calle?= =?UTF-8?q?rs=20=E2=80=94=20reconcile=20the=20two=20GT-633=20implementatio?= =?UTF-8?q?ns?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GT-633 was fixed twice, in parallel, by sessions that could not see each other. One shipped a standalone capture script reachable by the derived-artifact chain guard and the documentation job (#276, #287, on main); the other rendered the document inside `rule-corpus-triage.spec.ts`, where jest can pin it byte-for-byte (#279, on develop). Both were right about their half and both recaptured the same numbers — which is mutual corroboration of the measurement. But TWO GENERATORS FOR ONE ARTIFACT IS GT-633's OWN DEFECT ONE LEVEL UP: whichever ran last would win and the other's `--check` would go red for no reason a reader could see. That is what blocked develop -> main. The renderer now lives ONCE, in `test/rule-corpus-triage.ts`, with two callers: - `rule-corpus-triage.spec.ts` PINS it byte-for-byte (it can recompute); - `capture-native-evaluability-snapshot.mjs` WRITES it and `--check`s it, which is what the GT-630 chain and docs.yml invoke. The spec no longer writes. `UPDATE_EVALUABILITY_SNAPSHOT=1` is gone: a second writer is how this artifact ended up with two generators, and the surviving one is the command the chain guard already runs. RESOLVING IT SURFACED THE SAME DEFECT IN MINIATURE, twice, and both are fixed: 1. The spec asserted `SUMMARY.byClass` against an INLINE literal sitting beside the `PINNED_CLASS_COUNTS` constant that the dependency-free guard in src/rulesets/standards reads out of this file. Two copies of six numbers, and editing either left the other silently disagreeing. Now one declaration, asserted against and read by both. 2. My own first pass DELETED `PINNED_CLASS_COUNTS` while lifting the loader out. The documentation-job guard THREW rather than passing — "Could not find the PINNED_CLASS_COUNTS literal ... update this parser rather than deleting the check" — which is exactly the behaviour that fix was built for. Restored. The sticky `capturedOn` survives the port: the script renders once with the previous date and replaces that single field only when the CLASSIFICATION moved, so the chain's fixed-point replay stays byte-identical without a second ts-node run. Also converged onto main's shape so the branch merge is trivial rather than conflicted: `iso-5055-mapping.test.mjs`, both standards READMEs, the 5-link chain guard with its fixtures, and the two GT-598 steps in docs.yml. NOTE ON THE NUMBERS: this branch computes native-handler 151, and main pins 154. Not a measurement bug — main has more handler closures landed. The pin and the byte-for-byte snapshot assertion are what will catch it when the branches meet, which is the point of having one renderer. Verified: capture --check faithful (386 rules), build --check up to date (388 rules), replay byte-identical, iso-5055-mapping guard 9/9, chain guard 5 links current and at a fixed point with its own suite 8/8, core-domain jest 1444 passed / 128 suites, tsc --noEmit clean, 01/04/08/09 green. Co-Authored-By: Claude Opus 5 --- .github/workflows/docs.yml | 13 + .../ci/46-validate-derived-artifact-order.mjs | 32 ++ ...6-validate-derived-artifact-order.test.mjs | 78 ++++- .../satellite-evaluation-pipeline.service.ts | 146 +++++++-- .../validators/rule-corpus-triage.spec.ts | 238 +++----------- .../core-domain/test/rule-corpus-triage.ts | 246 ++++++++++++++ src/rulesets/standards/README.es.md | 69 ++-- src/rulesets/standards/README.md | 60 +++- .../capture-native-evaluability-snapshot.mjs | 305 ++++++++++++++++++ src/rulesets/standards/iso-5055-mapping.json | 2 +- .../standards/iso-5055-mapping.test.mjs | 133 ++++++-- .../native-evaluability-snapshot.json | 10 +- 12 files changed, 1037 insertions(+), 295 deletions(-) create mode 100644 src/packages/core-domain/test/rule-corpus-triage.ts create mode 100644 src/rulesets/standards/capture-native-evaluability-snapshot.mjs diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 1caea1de..79d647a3 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -99,6 +99,19 @@ jobs: - name: Self-tests for the ADR ruleset generator run: node --test .harness/scripts/generate-adr-rulesets.test.mjs + # GT-598 / GT-633: `native-evaluability-snapshot.json` says of itself that it + # is a capture, and for a long time nothing captured it — it was written by + # hand and drifted, while its guard compared the file against six numbers + # typed into the test: the same six the file contained. Drift is not + # contained either, because `build-iso-5055-mapping.mjs` stamps + # nativeEvaluability onto all 388 mapping rows FROM this file. The order of + # these two steps is the order of the chain. + - name: Native evaluability snapshot is a faithful capture of Core's triage + run: node src/rulesets/standards/capture-native-evaluability-snapshot.mjs --check + + - name: ISO/IEC 5055 mapping guard + run: node --test src/rulesets/standards/iso-5055-mapping.test.mjs + # GT-630: the derived artifacts have a dependency ORDER — the executive # summary is built FROM the maturity reconciliation, which is built from # the board. Generate the summary first and it captures a value the diff --git a/.harness/scripts/ci/46-validate-derived-artifact-order.mjs b/.harness/scripts/ci/46-validate-derived-artifact-order.mjs index 828b3fb3..0568baf2 100644 --- a/.harness/scripts/ci/46-validate-derived-artifact-order.mjs +++ b/.harness/scripts/ci/46-validate-derived-artifact-order.mjs @@ -88,6 +88,38 @@ export const CHAIN = [ consumes: ['src/packages/mcp-server/src/mcp/abac-evaluator.ts'], writes: ['src/rulesets/opa/abac-mcp-tool-access.rego'], }, + { + name: 'native evaluability snapshot', + producer: 'src/rulesets/standards/capture-native-evaluability-snapshot.mjs', + checkArgs: ['--check'], + // Same shape as the ABAC link: derived from the runtime through ts-node, so + // its real inputs are Core's triage sources plus the corpus they classify. + // The corpus half matters — `documentation-only` moved 129 -> 136 purely + // because seven generated ADR rulesets appeared, and nothing noticed for as + // long as this file was maintained by hand (GT-598). + consumes: [ + 'src/packages/core-domain/test/rule-corpus-triage.ts', + 'src/packages/core-domain/src/application/validators/rule-evaluability.ts', + 'src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts', + ], + writes: ['src/rulesets/standards/native-evaluability-snapshot.json'], + }, + { + name: 'ISO/IEC 5055 corpus mapping', + producer: 'src/rulesets/standards/build-iso-5055-mapping.mjs', + checkArgs: ['--check'], + // THE edge this pair exists for: the generator stamps `nativeEvaluability` + // onto every row from the snapshot. Rebuild before recapturing and a stale + // class is laundered into a 391-row artifact, with the handler backlog + // overstated by exactly the rules Core has closed since the last capture. + // Both artifacts pass their own --check at that moment, which is precisely + // the failure one-artifact-at-a-time checking cannot see. + consumes: ['src/rulesets/standards/native-evaluability-snapshot.json'], + writes: [ + 'src/rulesets/standards/iso-5055-mapping.json', + 'src/rulesets/standards/iso-5055-mapping.csv', + ], + }, { name: 'maturity reconciliation', producer: '.harness/scripts/ci/09-reconcile-maturity.mjs', diff --git a/.harness/scripts/ci/46-validate-derived-artifact-order.test.mjs b/.harness/scripts/ci/46-validate-derived-artifact-order.test.mjs index de02c4c8..6327ed17 100644 --- a/.harness/scripts/ci/46-validate-derived-artifact-order.test.mjs +++ b/.harness/scripts/ci/46-validate-derived-artifact-order.test.mjs @@ -23,6 +23,22 @@ import { spawnSync } from 'node:child_process'; const __dirname = dirname(fileURLToPath(import.meta.url)); const GUARD = resolve(__dirname, '46-validate-derived-artifact-order.mjs'); +// The guard only runs main() when invoked directly, so importing CHAIN is safe. +const { CHAIN } = await import('./46-validate-derived-artifact-order.mjs'); + +/** + * Where a link sits, DERIVED rather than typed. + * + * These positions were hardcoded ("link 2 of 3"), so every link added to the + * chain broke assertions that were not about the new link at all — which reads + * as the change being wrong rather than the test being brittle. + */ +const linkPosition = (producerSuffix) => { + const i = CHAIN.findIndex((l) => l.producer.endsWith(producerSuffix)); + assert.notEqual(i, -1, `no chain link produces ${producerSuffix} — the fixture is describing a chain that moved`); + return { position: i + 1, total: CHAIN.length }; +}; + let sandbox; before(() => { sandbox = mkdtempSync(join(tmpdir(), 'gt630-')); }); after(() => { if (sandbox) rmSync(sandbox, { recursive: true, force: true }); }); @@ -38,7 +54,7 @@ test('the real repository is current and at a fixed point', () => { const { status, out } = run(resolve(__dirname, '../../..')); assert.equal(status, 0, out); assert.match(out, /at a fixed point/); - assert.match(out, /links declared \.+ 3/); + assert.match(out, /links declared \.+ 5/); }); test('the guard leaves the real tree byte-identical', () => { @@ -50,6 +66,13 @@ test('the guard leaves the real tree byte-identical', () => { 'reference/core/control-center/maturity-reports/maturity-reconciliation.json', 'reference/core/control-center/maturity-reports/executive-summary.md', 'reference/core/control-center/maturity-reports/executive-summary.es.md', + // GT-598's pair. The snapshot is the sharper case: its producer stamps a + // `capturedOn` date, and it is deliberately STICKY when the classification + // is unchanged. Were it not, a replay would rewrite the date, this assertion + // would fail, and the fixed-point pass would report drift every day. + 'src/rulesets/standards/native-evaluability-snapshot.json', + 'src/rulesets/standards/iso-5055-mapping.json', + 'src/rulesets/standards/iso-5055-mapping.csv', ]; const before = artifacts.map((a) => readFileSync(join(repo, a))); run(repo); @@ -60,17 +83,41 @@ test('the guard leaves the real tree byte-identical', () => { /** - * The chain gained a first link (the ABAC rego, GT-602). These fixtures are about - * ORDER, not about ABAC, so each mini-repo gets a trivial producer/artifact pair - * for it — otherwise the shape check trips before the behaviour under test runs. + * Every REAL chain link that is not the behaviour under test needs a trivial + * producer/artifact pair in each mini-repo, or the shape check trips before the + * test gets to run. These fixtures are about ORDER — not about ABAC, the + * evaluability snapshot, or the 5055 mapping. + * + * Keep this in step with CHAIN. A link added to the guard without a stub here + * fails every fixture below with "declared producer does not exist", which + * points at the fixture rather than at the change that caused it. */ -const ABAC_STUB = { - '.harness/scripts/generate-abac-tool-sets.mjs': - "import fs from 'node:fs';\nconst f = process.cwd() + '/src/rulesets/opa/abac-mcp-tool-access.rego';\n" + - "if (process.argv.includes('--check')) process.exit(fs.readFileSync(f, 'utf8') === 'stable\\n' ? 0 : 1);\n" + - "fs.writeFileSync(f, 'stable\\n');\n", +const stubProducer = (artifacts) => + "import fs from 'node:fs';\n" + + `const files = ${JSON.stringify(artifacts)}.map(f => process.cwd() + '/' + f);\n` + + "if (process.argv.includes('--check')) " + + "process.exit(files.every(f => fs.readFileSync(f, 'utf8') === 'stable\\n') ? 0 : 1);\n" + + "for (const f of files) fs.writeFileSync(f, 'stable\\n');\n"; + +const PRELUDE_STUBS = { + // link 1 — ABAC rego (GT-602) + '.harness/scripts/generate-abac-tool-sets.mjs': stubProducer(['src/rulesets/opa/abac-mcp-tool-access.rego']), 'src/packages/mcp-server/src/mcp/abac-evaluator.ts': '// stub\n', 'src/rulesets/opa/abac-mcp-tool-access.rego': 'stable\n', + + // link 2 — native evaluability snapshot (GT-598) + 'src/rulesets/standards/capture-native-evaluability-snapshot.mjs': + stubProducer(['src/rulesets/standards/native-evaluability-snapshot.json']), + 'src/packages/core-domain/test/rule-corpus-triage.ts': '// stub\n', + 'src/packages/core-domain/src/application/validators/rule-evaluability.ts': '// stub\n', + 'src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts': '// stub\n', + 'src/rulesets/standards/native-evaluability-snapshot.json': 'stable\n', + + // link 3 — ISO/IEC 5055 mapping (GT-598), which consumes link 2's artifact + 'src/rulesets/standards/build-iso-5055-mapping.mjs': + stubProducer(['src/rulesets/standards/iso-5055-mapping.json', 'src/rulesets/standards/iso-5055-mapping.csv']), + 'src/rulesets/standards/iso-5055-mapping.json': 'stable\n', + 'src/rulesets/standards/iso-5055-mapping.csv': 'stable\n', }; // --- the shape of the declaration itself ------------------------------------ @@ -78,7 +125,7 @@ const ABAC_STUB = { describe('chain declaration (anti-vacuous)', () => { const fixture = (name, files) => { const root = join(sandbox, name); - for (const [rel, body] of Object.entries({ ...ABAC_STUB, ...files })) { + for (const [rel, body] of Object.entries({ ...PRELUDE_STUBS, ...files })) { mkdirSync(dirname(join(root, rel)), { recursive: true }); writeFileSync(join(root, rel), body); if (rel.endsWith('.mjs')) chmodSync(join(root, rel), 0o755); @@ -128,7 +175,7 @@ describe('stale versus out-of-order', () => { mkdirSync(dirname(join(root, rel)), { recursive: true }); writeFileSync(join(root, rel), body); }; - for (const [rel, body] of Object.entries(ABAC_STUB)) w(rel, body); + for (const [rel, body] of Object.entries(PRELUDE_STUBS)) w(rel, body); w('reference/core/control-center/gaps/gap-tracking.md', `count: ${boardCount}\n`); w('reference/core/control-center/maturity-reports/maturity-assessment.md', '# assessment\n'); w('reference/core/control-center/maturity-reports/maturity-reconciliation.json', `{"count":${reconCount}}\n`); @@ -177,7 +224,8 @@ fs.writeFileSync(dir + 'executive-summary.es.md', es); const root = miniRepo('stale-upstream', { boardCount: 9, reconCount: 7, summaryCount: 7 }); const { status, out } = run(root); assert.equal(status, 1, out); - assert.match(out, /maturity reconciliation is STALE \(link 2 of 3\)/); + const { position, total } = linkPosition('09-reconcile-maturity.mjs'); + assert.match(out, new RegExp(`maturity reconciliation is STALE \\(link ${position} of ${total}\\)`)); assert.match(out, /Stopping at the FIRST stale link on purpose/); assert.doesNotMatch(out, /executive governance summary is STALE/); }); @@ -239,7 +287,9 @@ fs.writeFileSync(dir + 'executive-summary.es.md', es); assert.equal(status, 1, out); assert.match(out, /differ after replaying the chain IN ORDER/); assert.match(out, /Fix by regenerating in the declared order/); - // The reconciler is link 2 now that the ABAC rego leads the chain. - assert.match(out, /2\. node \.harness\/scripts\/ci\/09-reconcile-maturity\.mjs/); + // The remediation must list the chain in order, with the reconciler at its + // real position — derived, so adding a link ahead of it does not break this. + const { position } = linkPosition('09-reconcile-maturity.mjs'); + assert.match(out, new RegExp(`${position}\\. node \\.harness/scripts/ci/09-reconcile-maturity\\.mjs`)); }); }); diff --git a/src/packages/core-domain/src/application/services/satellite-evaluation-pipeline.service.ts b/src/packages/core-domain/src/application/services/satellite-evaluation-pipeline.service.ts index b235d4c1..4cc1d49f 100644 --- a/src/packages/core-domain/src/application/services/satellite-evaluation-pipeline.service.ts +++ b/src/packages/core-domain/src/application/services/satellite-evaluation-pipeline.service.ts @@ -1,13 +1,53 @@ import { IFileSystem, ILogger } from '../../domain/interfaces'; -import { SatelliteManifest, EvaluationVerdict, PipelineGateResult, RuleEvaluation, EvaluationSeverity, EvaluationFacts } from '../../domain/satellite-manifest'; +import { SatelliteManifest, PipelineGateResult, RuleEvaluation, EvaluationSeverity, EvaluationFacts } from '../../domain/satellite-manifest'; import { TopologyCatalogService, TopologyManifest } from './topology-catalog.service'; import { SdlcDataLoaderService, StructuredGate } from './sdlc-data-loader.service'; -import { RulesetValidatorService } from '../validators/ruleset-validator.service'; +import { RulesetValidatorService, ValidationResult } from '../validators/ruleset-validator.service'; import { createSuccessEnvelope } from '../../domain/gate-evidence'; import { toLegacyPhaseId } from '../../domain/sdlc/phase-id'; import { OpaEvaluator } from '../validators/evaluators/opa-evaluator'; +import type { EvaluationKind } from '../../evaluation/contracts/evaluation-context'; +import type { + PipelineExecutionPlan, + PipelineRuleCoverage, + PipelineVerdict, +} from '../../evaluation/ports/evaluation-pipeline.port'; import * as path from 'path'; +/** + * GT-614 — the two units of work this pipeline is made of, each tagged with the + * evaluation kinds it answers. + * + * This is what turns `ctx.kinds` from a label read at the boundary into a decision + * about what gets executed. Asking for `compliance` alone used to pay for every + * phase gate — the whole f1..f5 gate corpus was loaded and every Rego rule in it + * run — before the answer was assembled. A stage whose kinds nobody requested is + * now never entered: not its loader, not its evaluator. + * + * The phase-gate stage is named as a STAGE rather than by its individual gate ids + * on purpose: those ids are only knowable by loading the gate definitions, which is + * precisely the work an out-of-scope request must not pay for. `general-rulesets` + * is the id of the synthetic gate the stage produces, so it needs no alias. + */ +const PHASE_GATES_STAGE = { + id: 'sdlc-phase-gates', + kinds: ['gate', 'artifact'] as readonly EvaluationKind[], +} as const; + +const GENERAL_RULESETS_STAGE = { + id: 'general-rulesets', + kinds: ['rule', 'compliance'] as readonly EvaluationKind[], +} as const; + +/** + * No plan ⇒ run everything, exactly as before this contract existed. An + * unrestricted plan (`kinds: []`, still sent by the inline REST callers that + * predate the field) answers `true` to every gate for the same reason. + */ +function stageIsInScope(kinds: readonly EvaluationKind[], plan?: PipelineExecutionPlan): boolean { + return !plan || plan.includesGate(kinds); +} + /** * End-to-end evaluation pipeline for GT-281. * @@ -19,6 +59,11 @@ import * as path from 'path'; * - SdlcDataLoaderService for GT-280 structured data * - OpaEvaluator for Rego execution * - RulesetValidatorService for general ruleset validation + * + * GT-614 — it is also the pipeline the deployed Core actually runs, so it is where + * the gap's sentence had to be closed: it now honours the + * {@link PipelineExecutionPlan} the orchestrator builds from `ctx.kinds` and skips + * the stages the request did not ask for. */ export class SatelliteEvaluationPipeline { private readonly topologyCatalog: TopologyCatalogService; @@ -36,36 +81,56 @@ export class SatelliteEvaluationPipeline { this.opaEvaluator = new OpaEvaluator(fs, logger); } - async evaluate(manifest: SatelliteManifest): Promise { + async evaluate( + manifest: SatelliteManifest, + plan?: PipelineExecutionPlan, + ): Promise { const corePath = manifest.corePath || this.discoverCorePath(manifest.satellitePath); // Step 1: Resolve topology const topology = manifest.topology || await this.resolveTopology(manifest.satellitePath, corePath); - // Step 2: Determine which phases to evaluate (GT-343: accept canonical ids, - // normalize to the legacy f1..f5 the structured gate data is keyed by). - const phaseIds = manifest.phase - ? [toLegacyPhaseId(manifest.phase) ?? manifest.phase] - : ['f1', 'f2', 'f3', 'f4', 'f5']; + // GT-614: decide what this request pays for BEFORE spending anything. Both + // decisions are taken here, together, so the two stages below are symmetrical: + // whichever one the request did not ask for is never entered. + const outOfScopeGateIds: string[] = []; + const runPhaseGates = stageIsInScope(PHASE_GATES_STAGE.kinds, plan); + const runGeneralRulesets = stageIsInScope(GENERAL_RULESETS_STAGE.kinds, plan); + if (!runPhaseGates) outOfScopeGateIds.push(PHASE_GATES_STAGE.id); + if (!runGeneralRulesets) outOfScopeGateIds.push(GENERAL_RULESETS_STAGE.id); - // Step 3: Load gates from GT-280 structured data const gateResults: PipelineGateResult[] = []; - for (const phaseId of phaseIds) { - const gates = await this.sdlcDataLoader.loadGatesForPhase(phaseId); - for (const gate of gates) { - const result = await this.evaluateGate(gate, manifest.satellitePath, corePath, topology, manifest.facts); - gateResults.push(result); + let phaseGateEvals = 0; + + if (runPhaseGates) { + // Step 2: Determine which phases to evaluate (GT-343: accept canonical ids, + // normalize to the legacy f1..f5 the structured gate data is keyed by). + const phaseIds = manifest.phase + ? [toLegacyPhaseId(manifest.phase) ?? manifest.phase] + : ['f1', 'f2', 'f3', 'f4', 'f5']; + + // Step 3: Load gates from GT-280 structured data + for (const phaseId of phaseIds) { + const gates = await this.sdlcDataLoader.loadGatesForPhase(phaseId); + for (const gate of gates) { + const result = await this.evaluateGate(gate, manifest.satellitePath, corePath, topology, manifest.facts); + gateResults.push(result); + phaseGateEvals += result.artifactEvaluations.length; + } } } // Step 4: Run general ruleset validation (GT-395: enforce canonical rulesets) - const generalResult = await this.validator.validate(manifest.satellitePath, corePath); + let generalResult: ValidationResult | undefined; + if (runGeneralRulesets) { + generalResult = await this.validator.validate(manifest.satellitePath, corePath); - // GT-395: Convert blocking general-result issues into a synthetic gate so - // they are visible in the output and participate in the top-level verdict. - const generalGate = this.buildGeneralRulesetsGate(generalResult); - if (generalGate) { - gateResults.push(generalGate); + // GT-395: Convert blocking general-result issues into a synthetic gate so + // they are visible in the output and participate in the top-level verdict. + const generalGate = this.buildGeneralRulesetsGate(generalResult); + if (generalGate) { + gateResults.push(generalGate); + } } // Step 5: Build summary @@ -77,11 +142,13 @@ export class SatelliteEvaluationPipeline { totalGates: gateResults.length, passedGates: passedGates.length, failedGates: gateResults.length - passedGates.length, - totalRules: allEvals.length + generalResult.rulesChecked, + totalRules: allEvals.length + (generalResult?.rulesChecked ?? 0), passedRules: allEvals.filter(e => e.passed).length, failedRules: allEvals.filter(e => !e.passed).length, }; + const coverage = this.buildCoverage(phaseGateEvals, generalResult, outOfScopeGateIds); + // ADR-0073 output envelope const outputEnvelope = createSuccessEnvelope( { topology, gates: gateResults, summary }, @@ -102,6 +169,43 @@ export class SatelliteEvaluationPipeline { summary, evaluatedAt, outputEnvelope, + coverage, + }; + } + + /** + * GT-614 / GT-569 — the coverage facts of the run, with out-of-scope kept in its + * own bucket. + * + * A stage nobody asked for is NOT `skipped` and NOT `errored`: both of those mean + * the engine tried and the outcome is UNKNOWN, so both belong in `rulesTotal` and + * become coverage risks (and, for a blocking rule, fail the run under GT-595). + * Reporting an unasked question as an unanswered one is the exact dishonesty + * GT-569 exists to prevent. An out-of-scope stage therefore contributes NOTHING to + * the four counters — it only appears, by id, in `outOfScopeGateIds`, so a reader + * can still see what the request did not buy. + * + * The counters keep the GT-569 invariant `checked + skipped + errored === total`. + * The synthetic `general-rulesets` gate's evaluations are deliberately NOT counted + * here: they are ISSUES derived from the corpus run, whose rules are already in + * `generalResult.rulesChecked`. + */ + private buildCoverage( + phaseGateEvals: number, + generalResult: ValidationResult | undefined, + outOfScopeGateIds: readonly string[], + ): PipelineRuleCoverage { + const rulesChecked = phaseGateEvals + (generalResult?.rulesChecked ?? 0); + const rulesSkipped = generalResult?.rulesSkipped ?? 0; + const rulesErrored = generalResult?.rulesErrored ?? 0; + return { + rulesChecked, + rulesSkipped, + rulesErrored, + rulesTotal: rulesChecked + rulesSkipped + rulesErrored, + skippedRuleIds: generalResult?.skippedRuleIds ?? [], + erroredRuleIds: generalResult?.erroredRuleIds ?? [], + outOfScopeGateIds: [...outOfScopeGateIds], }; } diff --git a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts index 45ebe40e..14b819cb 100644 --- a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts +++ b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts @@ -22,198 +22,55 @@ import * as fs from 'fs'; import * as path from 'path'; -import { NativeEvaluator } from './evaluators/native-evaluator'; -import { INativeRuleHandler } from './evaluators/handlers/rule-handler.interface'; -import { NormalizedRule } from '../../domain/models/normalized-rule'; import { - ClassifiedRule, + ADR_CONFORMANCE_CATEGORY, RULE_TRIAGE, RuleEvaluability, - classifyRule, isNonExecutable, - summarizeEvaluability, } from './rule-evaluability'; - -const REPO_ROOT = path.resolve(__dirname, '../../../../../..'); -const RULESETS_ROOT = path.join(REPO_ROOT, 'src', 'rulesets'); - -// --------------------------------------------------------------------------- -// Load the corpus exactly as DiskRulesetRepository does, minus the schema pass -// (this suite measures classification, not schema conformance). -// --------------------------------------------------------------------------- - -function rulesetFiles(dir: string, depth = 0): string[] { - if (depth > 4) return []; - const out: string[] = []; - for (const entry of fs.readdirSync(dir).sort()) { - const full = path.join(dir, entry); - if (entry.endsWith('.rules.json')) { out.push(full); continue; } - if (!entry.includes('.') && fs.statSync(full).isDirectory()) out.push(...rulesetFiles(full, depth + 1)); - } - return out; -} - -const CATEGORY_BY_PREFIX: Record = { - inh: 'inheritance', acl: 'anti-corruption', ocb: 'open-core', gov: 'governance', - evd: 'identity', 'obs-evd': 'tracing', dep: 'version-pinning', tax: 'naming-conventions', - hxa: 'layer-structure', git: 'branch-naming', cicd: 'ci-cd', tpy: 'testing-pyramid', - mtn: 'multi-tenancy', prot: 'protocol', runt: 'multi-runtime', dora: 'metrics', - space: 'metrics', drift: 'governance', 'cli-rr': 'build', 'cli-par': 'shared-logic', - mcp: 'protocol', 'modular-monolith': 'topology', 'distributed-modules': 'module-autonomy', - microservices: 'autonomous-deployment', -}; - -function deriveCategory(raw: Record): string { - if (raw['category']) return String(raw['category']); - const prefix = String(raw['id'] ?? '') - .replace(/-(?:EVD|RR|PAR)-?\d*$/, '') - .replace(/-\d+$/, '') - .toLowerCase(); - return CATEGORY_BY_PREFIX[prefix] ?? 'general'; -} - -function deriveSeverity(raw: Record): NormalizedRule['severity'] { - const declared = String(raw['severity'] ?? '').toUpperCase().trim(); - if (declared === 'MUST NOT') return 'MUST NOT'; - if (declared === 'MUST') return 'MUST'; - if (declared === 'SHOULD') return 'SHOULD'; - if (declared === 'COULD' || declared === 'MAY') return 'COULD'; - return raw['blocking'] === true || raw['enforcement'] ? 'MUST' : 'SHOULD'; -} - -function loadCorpus(): NormalizedRule[] { - const rules: NormalizedRule[] = []; - for (const file of rulesetFiles(RULESETS_ROOT)) { - const parsed = JSON.parse(fs.readFileSync(file, 'utf8')) as Record; - const list = (parsed['rules'] ?? parsed['principles']) as Array> | undefined; - if (!Array.isArray(list)) continue; - if (list.length > 0 && !list[0]['id'] && list[0]['rules']) continue; - - for (const raw of list.filter(r => Boolean(r['id']))) { - // `blocking` defaults from the RAW severity string, exactly as - // DiskRulesetRepository.defaultBlocking does — NOT from the normalized - // severity, which promotes `enforcement`-bearing rules to MUST. - const rawSeverity = String(raw['severity'] ?? '').toUpperCase(); - rules.push({ - id: String(raw['id']), - severity: deriveSeverity(raw), - category: deriveCategory(raw), - title: String(raw['title'] ?? raw['principle'] ?? raw['id']), - description: String(raw['description'] ?? raw['statement'] ?? ''), - blocking: Boolean(raw['blocking'] ?? (rawSeverity === 'MUST' || rawSeverity === 'MUST NOT')), - validationQuery: raw['validationQuery'] ? String(raw['validationQuery']) : undefined, - // GT-632: `enforce` was missing here for the same reason it was missing - // from DiskRulesetRepository — and while it was missing, this suite could - // not see the four rules that carry a machine-readable check, so it - // measured them as handler backlog. A loader that drops a field the - // handlers dispatch on does not "load the corpus exactly as - // DiskRulesetRepository does"; it measures a different corpus. - enforce: raw['enforce'] as NormalizedRule['enforce'], - sourceFile: path.relative(REPO_ROOT, file), - }); - } - } - return rules; -} - -/** The real handler set, asked only which rules it CLAIMS (no I/O performed). */ -function handlerSet(): INativeRuleHandler[] { - const evaluator = new NativeEvaluator({} as never, {} as never, {} as never); - return (evaluator as unknown as { handlers: INativeRuleHandler[] }).handlers; -} - -const CORPUS = loadCorpus(); -const HANDLERS = handlerSet(); - -/** A handler claims the rule — it is routed somewhere instead of falling through. */ -const claims = (rule: NormalizedRule) => HANDLERS.some(h => h.canHandle(rule)); +import { REPO_ROOT, RULESETS_ROOT, triageCorpus, renderSnapshot } from '../../../test/rule-corpus-triage'; + +// The corpus loader, the real handler set, the classification AND the snapshot +// renderer live in `test/rule-corpus-triage.ts`. They used to live HERE, which +// meant jest was the only thing that could run them — and that is why +// `native-evaluability-snapshot.json` was maintained by hand and drifted. +// +// GT-633 RECONCILIATION: the renderer in particular had two implementations, one +// here and one in `capture-native-evaluability-snapshot.mjs`, written in parallel +// by sessions that could not see each other. Two generators for one artifact is +// the defect GT-633 exists to remove, one level up: whichever ran last would win +// and the other's `--check` would go red for no visible reason. There is now ONE +// renderer, and this suite is its PIN rather than a second copy of it. +const TRIAGE = triageCorpus(); +const { corpus: CORPUS, classified: CLASSIFIED, summary: SUMMARY, claims } = TRIAGE; /** - * A handler claims the rule AND can return a verdict for it. + * The class counts this repository is pinned to. * - * {@link AdrConformanceRuleHandler} is deliberately excluded: it claims the 126 - * generated rules only to CLASSIFY them, and never returns `passed`. Counting - * "claimed" as "evaluated" would reproduce, one layer up, exactly the false - * green GT-569 removed — so this predicate, not `claims`, drives the breakdown. + * Also the counts `native-evaluability-snapshot.json` must reproduce. READ OUT OF + * THIS FILE by `src/rulesets/standards/iso-5055-mapping.test.mjs`, which runs in a + * job with no node_modules and so cannot recompute them — keep it a plain literal, + * and note that that guard THROWS if it cannot find this declaration rather than + * passing. Restoring it is what a reconciliation that deleted it owes back. */ -const evaluates = (rule: NormalizedRule) => claims(rule) && rule.category !== 'adr-conformance'; - -const CLASSIFIED: ClassifiedRule[] = CORPUS.map(rule => { - const { evaluability, why } = classifyRule(rule, evaluates(rule)); - return { ruleId: rule.id, sourceFile: rule.sourceFile, blocking: rule.blocking, evaluability, why }; -}); - -const SUMMARY = summarizeEvaluability(CLASSIFIED); +const PINNED_CLASS_COUNTS: Readonly> = { + 'native-handler': 151, + 'documentation-only': 136, + 'unimplemented-native': 48, + 'needs-external-system': 20, + 'needs-runtime': 17, + underspecified: 14, +}; -// --------------------------------------------------------------------------- -// The capture that src/rulesets/standards consumes -// --------------------------------------------------------------------------- +const SNAPSHOT_FILE = path.join(REPO_ROOT, 'src', 'rulesets', 'standards', 'native-evaluability-snapshot.json'); /** - * `src/rulesets` owns no dependency on this package, so the per-rule triage is - * CAPTURED there as JSON and `build-iso-5055-mapping.mjs` reads the capture. - * - * A capture is worth its bytes only if something recomputes it, and until now - * nothing did. The snapshot asserted in its own `validation` field that it - * reproduced the counts pinned above, and the only guard over it — the test - * named "the evaluability snapshot still matches the class counts pinned by - * Core", in `iso-5055-mapping.test.mjs` — compared the snapshot against a - * second hand-typed copy of the snapshot's own numbers. So Core moved - * 139 -> 151, the file stayed at 139, and every suite stayed green while the - * published handler backlog was overstated by twelve rules. - * - * The fix is to stop hand-typing the capture. It is RENDERED here, from the - * same `CLASSIFIED` array every assertion above measures, and pinned - * byte-for-byte — the `--check` shape `build-iso-5055-mapping.mjs` already - * uses for its own outputs. Recapture after a deliberate triage change: - * - * UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts + * `capturedOn` is carried forward from the committed document, exactly as the + * capture script does, so the pin compares classification against classification + * and never fails on a date. */ -const SNAPSHOT_FILE = path.join(REPO_ROOT, 'src', 'rulesets', 'standards', 'native-evaluability-snapshot.json'); - -/** Bumped when the document SHAPE changes, not when the measured counts move. */ -const SNAPSHOT_VERSION = '1.1.0'; -const SNAPSHOT_CAPTURED_ON = '2026-07-29'; - -/** One reading order for the six classes, shared by `counts` and `validation`. */ -const CLASS_ORDER: readonly RuleEvaluability[] = [ - 'native-handler', - 'documentation-only', - 'unimplemented-native', - 'needs-external-system', - 'needs-runtime', - 'underspecified', -]; - -/** The snapshot document, as bytes, straight from the live triage. */ -function renderSnapshot(): string { - const counts: Record = {}; - for (const klass of CLASS_ORDER) counts[klass] = SUMMARY.byClass[klass]; - - const classes: Record = {}; - for (const c of CLASSIFIED) classes[c.ruleId] = c.evaluability; - - const doc = { - $id: 'https://evolith.dev/rulesets/standards/native-evaluability-snapshot.json', - title: 'Native-engine evaluability class per rule (snapshot)', - description: - 'Per-rule evaluability class as computed by the Core native evaluator triage. This is a GENERATED CAPTURE, not the source of truth: the authority is src/packages/core-domain/src/application/validators/rule-evaluability.ts and the handler set registered in native-evaluator.ts. It is recorded here so the ISO/IEC 5055 mapping can be scoped to the real handler backlog without src/rulesets depending on a package it does not own. Do not hand-edit — regenerate.', - version: SNAPSHOT_VERSION, - capturedOn: SNAPSHOT_CAPTURED_ON, - capturedFrom: [ - 'src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)', - 'src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)', - 'src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)', - 'src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (corpus loader, renderer and byte-for-byte pin)', - ], - regenerateWith: - 'UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts (from src/packages/core-domain)', - validation: `Rendered by rule-corpus-triage.spec.ts from the live triage and pinned there byte-for-byte, so a divergence between this file and Core is a failing test rather than silent drift (corpus ${SUMMARY.total}; ${CLASS_ORDER.map(k => `${k} ${SUMMARY.byClass[k]}`).join(', ')}).`, - counts, - classes, - }; - - return JSON.stringify(doc, null, 2) + '\n'; +function committedCapturedOn(): string { + return (JSON.parse(fs.readFileSync(SNAPSHOT_FILE, 'utf8')) as { capturedOn: string }).capturedOn; } describe('GT-595 · the capture consumed by src/rulesets/standards', () => { @@ -221,7 +78,7 @@ describe('GT-595 · the capture consumed by src/rulesets/standards', () => { // `classes` is keyed by rule id alone, so two rules sharing an id in // different files would silently collapse into one entry and understate // every count downstream of it. - const rendered = JSON.parse(renderSnapshot()) as { + const rendered = JSON.parse(renderSnapshot(TRIAGE, { capturedOn: committedCapturedOn() })) as { counts: Record; classes: Record; }; @@ -230,8 +87,11 @@ describe('GT-595 · the capture consumed by src/rulesets/standards', () => { }); it('keeps native-evaluability-snapshot.json byte-identical to a fresh capture', () => { - const fresh = renderSnapshot(); - if (process.env.UPDATE_EVALUABILITY_SNAPSHOT) fs.writeFileSync(SNAPSHOT_FILE, fresh); + // Deliberately does NOT write. There is one writer — + // `node src/rulesets/standards/capture-native-evaluability-snapshot.mjs` — + // and it is the one the GT-630 chain guard and `docs.yml` invoke. A second + // writer is how this artifact ended up with two generators in the first place. + const fresh = renderSnapshot(TRIAGE, { capturedOn: committedCapturedOn() }); expect(fs.readFileSync(SNAPSHOT_FILE, 'utf8')).toBe(fresh); }); @@ -290,14 +150,12 @@ describe('GT-595 · the published breakdown, with its denominator', () => { // `enforce` was dropped at normalization. It is now carried, and // `ModuleBoundaryRuleHandler` evaluates it. // `unimplemented-native` drops by the same twelve; nothing else moved. - expect(SUMMARY.byClass).toEqual({ - 'native-handler': 151, - 'documentation-only': 136, - 'unimplemented-native': 48, - 'needs-external-system': 20, - 'needs-runtime': 17, - underspecified: 14, - }); + // Asserted against the SINGLE declaration above, not against a second inline + // copy of the same six numbers. There used to be two — this constant, read by + // the dependency-free guard in `src/rulesets/standards`, and a literal here — + // which is GT-633's own defect in miniature: two copies of one fact, and + // editing either leaves the other silently disagreeing. + expect(SUMMARY.byClass).toEqual(PINNED_CLASS_COUNTS); }); it('publishes the honest denominator: 150 rules nothing can ever run', () => { diff --git a/src/packages/core-domain/test/rule-corpus-triage.ts b/src/packages/core-domain/test/rule-corpus-triage.ts new file mode 100644 index 00000000..3b6522e8 --- /dev/null +++ b/src/packages/core-domain/test/rule-corpus-triage.ts @@ -0,0 +1,246 @@ +/** + * GT-598 — the corpus triage, extracted so it has more than one caller. + * + * WHY THIS FILE EXISTS + * This code used to live inside `rule-corpus-triage.spec.ts`, where it was + * reachable only by jest. That made `src/rulesets/standards/native-evaluability- + * snapshot.json` impossible to CAPTURE: nothing outside the spec could ask the + * real handler set which rules it claims, so the snapshot was maintained by + * hand, and drifted (it still said `documentation-only: 129` after Core moved to + * 136, and its guard could not see it because the guard re-read the snapshot's + * own literals). + * + * So the measurement moved here. It has two callers now, and they are the two + * halves of the same guarantee: + * + * - `rule-corpus-triage.spec.ts` pins the numbers and asserts the committed + * snapshot still agrees with a freshly computed triage; + * - `src/rulesets/standards/capture-native-evaluability-snapshot.mjs` + * regenerates that snapshot from this same computation. + * + * It lives under `test/` rather than `src/` deliberately: it walks the + * repository with `fs`, which is not something the published application layer + * should do, and `files: ["dist"]` keeps it out of the package. + * + * Nothing here is mocked. The handler set is the real {@link NativeEvaluator} + * one, asked only which rules it CLAIMS — no evaluation, no I/O. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import { NativeEvaluator } from '../src/application/validators/evaluators/native-evaluator'; +import { INativeRuleHandler } from '../src/application/validators/evaluators/handlers/rule-handler.interface'; +import { NormalizedRule } from '../src/domain/models/normalized-rule'; +import { + ADR_CONFORMANCE_CATEGORY, + ClassifiedRule, + EvaluabilitySummary, + RuleEvaluability, + classifyRule, + summarizeEvaluability, +} from '../src/application/validators/rule-evaluability'; + +/** Repository root, from this file's location (`/src/packages/core-domain/test`). */ +export const REPO_ROOT = path.resolve(__dirname, '../../../..'); +export const RULESETS_ROOT = path.join(REPO_ROOT, 'src', 'rulesets'); + +// --------------------------------------------------------------------------- +// Load the corpus exactly as DiskRulesetRepository does, minus the schema pass +// (this measures classification, not schema conformance). +// --------------------------------------------------------------------------- + +function rulesetFiles(dir: string, depth = 0): string[] { + if (depth > 4) return []; + const out: string[] = []; + for (const entry of fs.readdirSync(dir).sort()) { + const full = path.join(dir, entry); + if (entry.endsWith('.rules.json')) { out.push(full); continue; } + if (!entry.includes('.') && fs.statSync(full).isDirectory()) out.push(...rulesetFiles(full, depth + 1)); + } + return out; +} + +const CATEGORY_BY_PREFIX: Record = { + inh: 'inheritance', acl: 'anti-corruption', ocb: 'open-core', gov: 'governance', + evd: 'identity', 'obs-evd': 'tracing', dep: 'version-pinning', tax: 'naming-conventions', + hxa: 'layer-structure', git: 'branch-naming', cicd: 'ci-cd', tpy: 'testing-pyramid', + mtn: 'multi-tenancy', prot: 'protocol', runt: 'multi-runtime', dora: 'metrics', + space: 'metrics', drift: 'governance', 'cli-rr': 'build', 'cli-par': 'shared-logic', + mcp: 'protocol', 'modular-monolith': 'topology', 'distributed-modules': 'module-autonomy', + microservices: 'autonomous-deployment', +}; + +function deriveCategory(raw: Record): string { + if (raw['category']) return String(raw['category']); + const prefix = String(raw['id'] ?? '') + .replace(/-(?:EVD|RR|PAR)-?\d*$/, '') + .replace(/-\d+$/, '') + .toLowerCase(); + return CATEGORY_BY_PREFIX[prefix] ?? 'general'; +} + +function deriveSeverity(raw: Record): NormalizedRule['severity'] { + const declared = String(raw['severity'] ?? '').toUpperCase().trim(); + if (declared === 'MUST NOT') return 'MUST NOT'; + if (declared === 'MUST') return 'MUST'; + if (declared === 'SHOULD') return 'SHOULD'; + if (declared === 'COULD' || declared === 'MAY') return 'COULD'; + return raw['blocking'] === true || raw['enforcement'] ? 'MUST' : 'SHOULD'; +} + +export function loadCorpus(rulesetsRoot: string = RULESETS_ROOT): NormalizedRule[] { + const rules: NormalizedRule[] = []; + for (const file of rulesetFiles(rulesetsRoot)) { + const parsed = JSON.parse(fs.readFileSync(file, 'utf8')) as Record; + const list = (parsed['rules'] ?? parsed['principles']) as Array> | undefined; + if (!Array.isArray(list)) continue; + if (list.length > 0 && !list[0]['id'] && list[0]['rules']) continue; + + for (const raw of list.filter(r => Boolean(r['id']))) { + // `blocking` defaults from the RAW severity string, exactly as + // DiskRulesetRepository.defaultBlocking does — NOT from the normalized + // severity, which promotes `enforcement`-bearing rules to MUST. + const rawSeverity = String(raw['severity'] ?? '').toUpperCase(); + rules.push({ + id: String(raw['id']), + severity: deriveSeverity(raw), + category: deriveCategory(raw), + title: String(raw['title'] ?? raw['principle'] ?? raw['id']), + description: String(raw['description'] ?? raw['statement'] ?? ''), + blocking: Boolean(raw['blocking'] ?? (rawSeverity === 'MUST' || rawSeverity === 'MUST NOT')), + validationQuery: raw['validationQuery'] ? String(raw['validationQuery']) : undefined, + // GT-632: `enforce` was missing here for the same reason it was missing + // from DiskRulesetRepository — and while it was missing, this measurement + // could not see the four rules that carry a machine-readable check, so it + // counted them as handler backlog. A loader that drops a field the + // handlers dispatch on does not "load the corpus exactly as + // DiskRulesetRepository does"; it measures a different corpus. + enforce: raw['enforce'] as NormalizedRule['enforce'], + sourceFile: path.relative(REPO_ROOT, file), + }); + } + } + return rules; +} + +/** The real handler set, asked only which rules it CLAIMS (no I/O performed). */ +export function handlerSet(): INativeRuleHandler[] { + const evaluator = new NativeEvaluator({} as never, {} as never, {} as never); + return (evaluator as unknown as { handlers: INativeRuleHandler[] }).handlers; +} + +export interface CorpusTriage { + readonly corpus: readonly NormalizedRule[]; + readonly classified: readonly ClassifiedRule[]; + readonly summary: EvaluabilitySummary; + /** A handler claims the rule — it is routed somewhere instead of falling through. */ + readonly claims: (rule: NormalizedRule) => boolean; + /** A handler claims the rule AND can return a verdict for it. */ + readonly evaluates: (rule: NormalizedRule) => boolean; +} + +/** + * Classify the whole corpus with the real handler set. + * + * {@link import('../src/application/validators/evaluators/handlers/adr-conformance-rule.handler').AdrConformanceRuleHandler} + * is deliberately excluded from `evaluates`: it claims the generated ADR rules + * only to CLASSIFY them, and never returns `passed`. Counting "claimed" as + * "evaluated" would reproduce, one layer up, exactly the false green GT-569 + * removed — so `evaluates`, not `claims`, drives the breakdown. + */ +export function triageCorpus(rulesetsRoot: string = RULESETS_ROOT): CorpusTriage { + const corpus = loadCorpus(rulesetsRoot); + const handlers = handlerSet(); + + const claims = (rule: NormalizedRule) => handlers.some(h => h.canHandle(rule)); + const evaluates = (rule: NormalizedRule) => claims(rule) && rule.category !== ADR_CONFORMANCE_CATEGORY; + + const classified: ClassifiedRule[] = corpus.map(rule => { + const { evaluability, why } = classifyRule(rule, evaluates(rule)); + return { ruleId: rule.id, sourceFile: rule.sourceFile, blocking: rule.blocking, evaluability, why }; + }); + + return { corpus, classified, summary: summarizeEvaluability(classified), claims, evaluates }; +} + +// --------------------------------------------------------------------------- +// The rendered capture — ONE renderer, two callers +// --------------------------------------------------------------------------- + +/** + * GT-633 reconciliation: this function had TWO implementations. + * + * The fix for GT-633 was written twice, in parallel, by sessions that could not + * see each other. One shipped a standalone capture script (`capture-native- + * evaluability-snapshot.mjs`, reachable by the derived-artifact chain guard and + * by the documentation job); the other rendered the document inside + * `rule-corpus-triage.spec.ts`, where jest can pin it byte-for-byte. Both were + * right about their half, and both recaptured the same numbers. + * + * But two generators for one artifact is EXACTLY the defect GT-633 exists to + * remove, reproduced one level up: whichever ran last would win, and the other's + * `--check` would go red for no reason a reader could see. + * + * So the document is rendered HERE, once, and both callers use it: + * - `rule-corpus-triage.spec.ts` pins it byte-for-byte (it can recompute); + * - `capture-native-evaluability-snapshot.mjs` writes it and `--check`s it, + * which is what the GT-630 chain and `docs.yml` invoke. + * + * `capturedOn` is deliberately STICKY: it is passed in by the caller from the + * committed document when the classification has not moved, so replaying the + * chain stays byte-identical instead of rewriting a date on every run. + */ +export const SNAPSHOT_SHAPE_VERSION = '1.1.0'; + +/** One reading order for the six classes, shared by `counts` and `validation`. */ +export const CLASS_ORDER: readonly RuleEvaluability[] = [ + 'native-handler', + 'documentation-only', + 'unimplemented-native', + 'needs-external-system', + 'needs-runtime', + 'underspecified', +]; + +export interface SnapshotRenderOptions { + /** Carried forward from the committed file while the classification is unchanged. */ + readonly capturedOn: string; +} + +/** The snapshot document, as bytes, straight from a live triage. */ +export function renderSnapshot(triage: CorpusTriage, { capturedOn }: SnapshotRenderOptions): string { + const counts: Record = {}; + for (const klass of CLASS_ORDER) counts[klass] = triage.summary.byClass[klass]; + + const classes: Record = {}; + for (const c of triage.classified) classes[c.ruleId] = c.evaluability; + + const doc = { + $id: 'https://evolith.dev/rulesets/standards/native-evaluability-snapshot.json', + title: 'Native-engine evaluability class per rule (snapshot)', + description: + 'Per-rule evaluability class as computed by the Core native evaluator triage. This is a GENERATED CAPTURE, not the source of truth: the authority is src/packages/core-domain/src/application/validators/rule-evaluability.ts and the handler set registered in native-evaluator.ts. It is recorded here so the ISO/IEC 5055 mapping can be scoped to the real handler backlog without src/rulesets depending on a package it does not own. Do not hand-edit — regenerate.', + version: SNAPSHOT_SHAPE_VERSION, + capturedOn, + capturedFrom: [ + 'src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)', + 'src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)', + 'src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)', + 'src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, classification and this renderer)', + ], + regenerateWith: 'node src/rulesets/standards/capture-native-evaluability-snapshot.mjs', + validation: `Rendered by test/rule-corpus-triage.ts from the live triage, written by capture-native-evaluability-snapshot.mjs and pinned byte-for-byte by rule-corpus-triage.spec.ts, so a divergence between this file and Core is a failing test rather than silent drift (corpus ${triage.summary.total}; ${CLASS_ORDER.map(k => `${k} ${triage.summary.byClass[k]}`).join(', ')}).`, + // Both numbers, because they are not the same question — and the + // documentation-job guard asserts on them. `corpusSize` is what Core + // classified and what `counts` sums to; `distinctRuleIds` is how many keys + // `classes` can hold. They differ exactly when one rule id appears in more + // than one ruleset file, and a guard that assumed they were equal would go + // red on a corpus change that is not drift. + corpusSize: triage.corpus.length, + distinctRuleIds: Object.keys(classes).length, + counts, + classes, + }; + + return JSON.stringify(doc, null, 2) + '\n'; +} diff --git a/src/rulesets/standards/README.es.md b/src/rulesets/standards/README.es.md index 80d4973b..15bca204 100644 --- a/src/rulesets/standards/README.es.md +++ b/src/rulesets/standards/README.es.md @@ -17,10 +17,23 @@ confiar en nosotros. | `iso-5055-weaknesses.json` | Los 138 identificadores CWE de ISO/IEC 5055, por medida, con procedencia. | | `iso-5055-mapping.json` | Una fila por regla del corpus: mapeo a CWE (o un "sin equivalente" explícito con motivo), adoptabilidad por analizador y clase de evaluabilidad nativa. | | `iso-5055-mapping.csv` | La misma tabla, plana, para hojas de cálculo y auditores. | -| `native-evaluability-snapshot.json` | Clase de evaluabilidad por regla, **generada** desde el triage vivo del Core por `rule-corpus-triage.spec.ts`, que la fija byte a byte. Nunca se edita a mano: se recaptura con `UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts` desde `src/packages/core-domain`. | +| `native-evaluability-snapshot.json` | Captura por regla de la clase de evaluabilidad del triage del Core, para acotar la aritmética de backlog de abajo al backlog real. Generado — no editar a mano. | +| `capture-native-evaluability-snapshot.mjs` | La captura. Ejecuta el triage real del Core vía `ts-node`; `--check` falla cuando la captura comiteada ya no es lo que el Core calcula. | | `build-iso-5055-mapping.mjs` | El generador. `--check` falla cuando la tabla se quedó atrás del corpus. | | `iso-5055-mapping.test.mjs` | La guarda. `node --test src/rulesets/standards/iso-5055-mapping.test.mjs`. | +### Regenerar, en orden + +``` +node src/rulesets/standards/capture-native-evaluability-snapshot.mjs # 1. captura +node src/rulesets/standards/build-iso-5055-mapping.mjs # 2. mapeo +``` + +El orden no es una preferencia. El generador estampa `nativeEvaluability` en cada fila del mapeo a +partir de la captura, así que reconstruir primero blanquea una clasificación obsoleta dentro de un +artefacto de 391 filas y sobreestima el backlog de handlers en tantas reglas como el Core haya cerrado +desde la última captura. + Aquí no se reproduce texto de ISO/IEC 5055 ni de ISO/IEC 25010. Solo se registran identificadores CWE, nombres CWE de MITRE y pertenencia a cada medida. @@ -34,20 +47,20 @@ hace que la unión sea menor que la suma. La fecha y el método de extracción c ## Resultado -Sobre **388 reglas** en 174 archivos de ruleset: +Sobre **391 reglas** en 175 archivos de ruleset: | Medida | Cantidad | Proporción | |---|---|---| | Reglas mapeadas a una debilidad ISO/IEC 5055 | 37 | 9,5% | | — de ellas con mapeo directo | 8 | | | — de ellas con mapeo parcial / proxy | 29 | | -| Reglas sin equivalente internacional (cada una con motivo declarado) | 351 | 90,5% | -| Reglas que un analizador existente podría decidir por completo | 42 | 10,8% | +| Reglas sin equivalente internacional (cada una con motivo declarado) | 354 | 90,5% | +| Reglas que un analizador existente podría decidir por completo | 42 | 10,7% | | Reglas que un analizador podría decidir parcialmente | 23 | 5,9% | **La fracción adoptada es el 9,5% del corpus.** Es un resultado real y es menor de lo que sugería la premisa del gap, por una razón estructural: ISO/IEC 5055 mide la estructura interna del código fuente, -y 313 de nuestras 388 reglas no tratan de estructura de código. Son conformidad con ADR (162), +y 313 de nuestras 391 reglas no tratan de estructura de código. Son conformidad con ADR (162), contratos de topología (66), invariantes de gobierno (51) y proceso de desarrollo (34). Ningún estándar internacional modela "¿honraste el ADR-0092?", y ninguno lo hará. @@ -58,9 +71,16 @@ mapean y 13 son decidibles por analizador. El enunciado del gap dimensionaba el beneficio contra "~240 handlers por escribir". **Esa cifra ya está retirada.** GT-595 hizo el triage del corpus y el backlog real, decidible desde el repositorio, son **48 -reglas** — la clase `unimplemented-native`. Las otras 338 son 136 placeholders de generador solo -documentales, 14 reglas sin check redactado, 20 que requieren un sistema externo, 17 que requieren uno -en ejecución y 151 que un handler nativo ya evalúa. +reglas** — la clase `unimplemented-native`. De las 389 reglas que carga el triage del Core, 154 ya se +ejecutan y las otras 187 son 136 placeholders de generador solo documentales, 14 reglas sin check +redactado, 20 que requieren un sistema externo y 17 que requieren uno en ejecución. + +(389, no 391: los dos archivos de regla única llevan sus metadatos en la raíz del documento, y el +cargador de corpus del Core no los lee. Aparecen en el mapeo como `not-in-snapshot`.) + +60 → 48 el 2026-07-29: ocho reglas con forma de configuración recibieron handler (GT-595) y cuatro +reglas de límites de módulo ya traían una cláusula `enforce` completa que la normalización descartaba +(GT-632). Proyectar este mapeo sobre esa clase es la cifra que importa: @@ -71,18 +91,16 @@ Proyectar este mapeo sobre esa clase es la cifra que importa: | Que hay que escribir de verdad | 38 | Las 5 son `HXA-03` (estructura de capas — dependency-cruiser o ArchUnit), `SEC-INJ-01`, `SEC-PATH-01`, -`SEC-PATH-02` (consultas de inyección y path traversal de CodeQL/Semgrep) y `SEC-TIMING-01` -(comparación en tiempo constante). Las 5 parciales están listadas en -`handlerBacklog.byEvaluabilityClass` del JSON de mapeo. +`SEC-PATH-02` (consultas de inyección y path traversal de CodeQL/Semgrep) y `SEC-TIMING-01` (comparación +en tiempo constante). Las 5 parciales están listadas en `handlerBacklog.byEvaluabilityClass` del JSON de +mapeo. -Es decir, adoptar vale **10% del backlog por completo, 21% incluyendo parciales** — 10 de 48 reglas que -no necesitan handlers a medida. No es el orden de magnitud que esperaba el gap, pero es una reducción -concreta y nominada, y apunta las reglas de seguridad hacia analizadores mejores que cualquier cosa que -escribiéramos. - -El backlog bajó de 60 a 48 porque se *escribieron* handlers, no porque se recortara el denominador: -`HXA-01/02/04/05` (GT-632) y `GIT-08` junto con otras siete reglas de configuración (GT-595) pasaron a -`native-handler`, que es también por qué cuatro de los nueve nombres de antes ya no están en la lista. +Es decir, adoptar vale **10,4% del backlog por completo, 20,8% incluyendo parciales** — 10 de 48 reglas +que no necesitan handlers a medida. Ambas proporciones bajaron al encogerse el backlog, y esa es la +dirección correcta: entre las doce reglas cerradas el 2026-07-29 están `HXA-01`, `HXA-02`, `HXA-04` y +`GIT-08`, que eran cuatro de las nueve que esta tabla ofrecía a un analizador. Evolith escribió el +handler primero. Lo que queda se inclina más hacia escribir, y las reglas de seguridad siguen apuntando +a analizadores mejores que cualquier cosa que escribiéramos. ## Taxonomía compañera: ISO/IEC 25010:2023 @@ -105,3 +123,16 @@ reintroduce. `nativeEvaluability` se copia del snapshot de triage del Core. La autoridad es `src/packages/core-domain/src/application/validators/rule-evaluability.ts` y su spec fijado; si esas cifras se mueven, el snapshot de aquí está obsoleto y hay que recapturarlo. + +## Cómo se detecta la deriva + +El snapshot se mantenía a mano hasta GT-598, y derivó: declaraba `documentation-only: 129` mucho después +de que el Core pasara a 136, y la guarda que debía notarlo comparaba el snapshot contra seis números +escritos en el propio test — los mismos seis que el snapshot ya contenía. Comparaba el snapshot consigo +mismo y solo podía pasar. La reemplazan tres controles, en los dos jobs que pueden costearlos: + +| Dónde | Qué prueba | +|---|---| +| `rule-corpus-triage.spec.ts` (jest de core-domain) | El snapshot comiteado es igual a un triage **recalculado** — las cuentas y la clase de cada regla. Esta es la guarda real: tiene las dependencias para recalcular la verdad. | +| `iso-5055-mapping.test.mjs` (job de documentación, sin `node_modules`) | Las cuentas del snapshot son iguales a las **leídas desde** ese spec, su cabecera concuerda con su propio cuerpo, y toda clase estampada en el mapeo coincide con el snapshot. | +| `capture-native-evaluability-snapshot.mjs --check` | La re-derivación extremo a extremo, ejecutable donde el workspace esté instalado. | diff --git a/src/rulesets/standards/README.md b/src/rulesets/standards/README.md index 32c5299b..504d6566 100644 --- a/src/rulesets/standards/README.md +++ b/src/rulesets/standards/README.md @@ -17,10 +17,22 @@ to trust us. | `iso-5055-weaknesses.json` | The 138 CWE identifiers of ISO/IEC 5055, split per measure, with provenance. | | `iso-5055-mapping.json` | One row per corpus rule: CWE mapping (or an explicit "no equivalent" with a reason), analyser adoptability, and the rule's native evaluability class. | | `iso-5055-mapping.csv` | The same table, flat, for spreadsheets and auditors. | -| `native-evaluability-snapshot.json` | Per-rule evaluability class, **generated** from Core's live triage by `rule-corpus-triage.spec.ts`, which pins it byte-for-byte. Never hand-edit it: recapture with `UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts` from `src/packages/core-domain`. | +| `native-evaluability-snapshot.json` | Captured per-rule evaluability class from Core's triage, so the backlog arithmetic below is scoped to the real backlog. Generated — do not edit by hand. | +| `capture-native-evaluability-snapshot.mjs` | The capture. Runs Core's real triage through `ts-node`; `--check` fails when the committed snapshot is no longer what Core computes. | | `build-iso-5055-mapping.mjs` | The generator. `--check` fails when the table has fallen behind the corpus. | | `iso-5055-mapping.test.mjs` | The guard. `node --test src/rulesets/standards/iso-5055-mapping.test.mjs`. | +### Regenerating, in order + +``` +node src/rulesets/standards/capture-native-evaluability-snapshot.mjs # 1. snapshot +node src/rulesets/standards/build-iso-5055-mapping.mjs # 2. mapping +``` + +The order is not a preference. The generator stamps `nativeEvaluability` onto every row of the mapping +from the snapshot, so rebuilding first launders a stale classification into a 391-row artifact and +overstates the handler backlog by however many rules Core has closed since the last capture. + No text of ISO/IEC 5055 or ISO/IEC 25010 is reproduced here. Only CWE identifiers, MITRE CWE names and measure membership are recorded. @@ -34,20 +46,20 @@ makes the union smaller than the sum. The extraction date and method are recorde ## Result -Against **388 rules** in 174 ruleset files: +Against **391 rules** in 175 ruleset files: | Measure | Count | Share | |---|---|---| | Rules mapped to an ISO/IEC 5055 weakness | 37 | 9.5% | | — of which the mapping is direct | 8 | | | — of which the mapping is a partial / proxy | 29 | | -| Rules with no international equivalent (each with a stated reason) | 351 | 90.5% | -| Rules an existing analyser could decide outright | 42 | 10.8% | +| Rules with no international equivalent (each with a stated reason) | 354 | 90.5% | +| Rules an existing analyser could decide outright | 42 | 10.7% | | Rules an analyser could decide partially | 23 | 5.9% | **The adopted fraction is 9.5% of the corpus.** That is a real result and it is smaller than the gap's premise implied, for a structural reason: ISO/IEC 5055 measures the internal structure of source code, -and 313 of our 388 rules are not about source structure at all. They are ADR conformance (162), +and 313 of our 391 rules are not about source structure at all. They are ADR conformance (162), topology contracts (66), governance invariants (51) and development process (34). No international standard models "did you honour ADR-0092", and none ever will. @@ -58,9 +70,15 @@ and 13 are analyser-decidable. The gap statement sized the payoff against "~240 handlers to write". **That figure is already retired.** GT-595 triaged the corpus and the real, decidable-from-the-repository backlog is **48 -rules** — the `unimplemented-native` class. The other 338 are 136 documentation-only generator -placeholders, 14 underspecified rules with no authored check, 20 that need an external system, 17 that -need a running one, and 151 a native handler already evaluates. +rules** — the `unimplemented-native` class. Of the 389 rules Core's triage loads, 154 already run and +the other 187 are 136 documentation-only generator placeholders, 14 underspecified rules with no +authored check, 20 that need an external system and 17 that need a running one. + +(389, not 391: the two single-rule infrastructure files carry their rule metadata at the document root, +which Core's corpus loader does not read. They appear in the mapping as `not-in-snapshot`.) + +60 → 48 on 2026-07-29: eight config-shaped rules got handlers (GT-595) and four module-boundary rules +turned out to already carry a complete `enforce` clause that normalization was dropping (GT-632). Folding this mapping onto that class is the number that matters: @@ -74,13 +92,12 @@ The 5 are `HXA-03` (layer structure — dependency-cruiser or ArchUnit), `SEC-IN `SEC-PATH-02` (CodeQL/Semgrep injection and path-traversal queries) and `SEC-TIMING-01` (timing-safe comparison). The 5 partials are listed in `handlerBacklog.byEvaluabilityClass` in the mapping JSON. -So adoption is worth **10% of the backlog outright, 21% including partials** — 10 of 48 rules that do -not need bespoke handlers. Not the order-of-magnitude the gap hoped for, but a concrete, named -reduction, and it points the security rules at analysers that are better than anything we would write. - -The backlog shrank from 60 to 48 because handlers were *written*, not because the denominator was -re-cut: `HXA-01/02/04/05` (GT-632) and `GIT-08` with seven other config-shaped rules (GT-595) all moved -into `native-handler`, which is also why four of the nine names above have left this list. +So adoption is worth **10.4% of the backlog outright, 20.8% including partials** — 10 of 48 rules that +do not need bespoke handlers. Both shares fell when the backlog shrank, and that is the right +direction: the twelve rules closed on 2026-07-29 include `HXA-01`, `HXA-02`, `HXA-04` and `GIT-08`, +which were four of the nine this table used to offer to an analyser. Evolith wrote the handler first. +What is left leans further toward authoring, and the security rules still point at analysers better +than anything we would write. ## Companion taxonomy: ISO/IEC 25010:2023 @@ -102,3 +119,16 @@ stale, and `iso-5055-mapping.test.mjs` fails the build if anything in this direc `nativeEvaluability` is copied from the Core triage snapshot. The authority for it is `src/packages/core-domain/src/application/validators/rule-evaluability.ts` and its pinned spec; if the counts there move, the snapshot here is stale and must be recaptured. + +## How drift is caught + +The snapshot was hand-maintained until GT-598, and it drifted: it declared `documentation-only: 129` +long after Core had moved to 136, and the guard that was meant to notice compared the snapshot against +six numbers typed into the test — the same six the snapshot already contained. It compared the snapshot +to itself and could only ever pass. Three checks replace it, in the two jobs that can afford them: + +| Where | What it proves | +|---|---| +| `rule-corpus-triage.spec.ts` (core-domain jest) | The committed snapshot equals a **freshly computed** triage — counts and every per-rule class. This is the real guard; it has the dependencies to recompute the truth. | +| `iso-5055-mapping.test.mjs` (documentation job, no `node_modules`) | The snapshot's counts equal the counts **read out of** that spec, its header agrees with its own body, and every class stamped into the mapping matches the snapshot. | +| `capture-native-evaluability-snapshot.mjs --check` | The end-to-end re-derivation, runnable anywhere the workspace is installed. | diff --git a/src/rulesets/standards/capture-native-evaluability-snapshot.mjs b/src/rulesets/standards/capture-native-evaluability-snapshot.mjs new file mode 100644 index 00000000..1488ef20 --- /dev/null +++ b/src/rulesets/standards/capture-native-evaluability-snapshot.mjs @@ -0,0 +1,305 @@ +#!/usr/bin/env node +/** + * @file capture-native-evaluability-snapshot.mjs + * @description GT-598 — capture `native-evaluability-snapshot.json` from Core. + * + * WHY THIS EXISTS + * --------------- + * The snapshot said, in its own header, that it was "a CAPTURED SNAPSHOT, not + * the source of truth". No capture script existed. It was written by hand, and + * it drifted: it still declared `documentation-only: 129` long after Core moved + * to 136, and the guard that was supposed to notice + * (`iso-5055-mapping.test.mjs`) compared the snapshot's counts against six + * numbers hardcoded in the test — the same numbers the snapshot already + * contained. It could only ever pass. This script is the missing half. + * + * Drift here is not contained. `build-iso-5055-mapping.mjs` stamps + * `nativeEvaluability` onto every row of the ISO/IEC 5055 mapping from this + * file, so a stale class silently overstates the handler backlog in a much + * larger derived artifact. ALWAYS recapture before rebuilding the mapping. + * + * WHAT IT RUNS + * ------------ + * The real Core triage — `test/rule-corpus-triage.ts` in @beyondnet/evolith-core-domain, + * which instantiates the real `NativeEvaluator`, asks its handler set which + * rules it claims, and classifies the rest through `classifyRule`. Nothing is + * reimplemented here: a second implementation of the classification would be a + * second source of truth, which is the defect this script exists to remove. + * + * Core is TypeScript, so the triage is executed through `ts-node` (transpile + * only, no emit). That makes this the one script under `src/rulesets` that + * needs `node_modules` — which is exactly why the guard that runs in the + * dependency-free documentation job stays a separate, non-executing check. + * + * USAGE + * node src/rulesets/standards/capture-native-evaluability-snapshot.mjs + * node src/rulesets/standards/capture-native-evaluability-snapshot.mjs --check + * + * EXIT CODES + * 0 snapshot written, or (with --check) already identical to a fresh capture + * 1 the capture disagrees with the committed file, or the triage refused to + * run / returned an implausible corpus + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createRequire } from 'node:module'; +import { execFileSync } from 'node:child_process'; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = path.resolve(HERE, '..', '..', '..'); +const OUT = path.join(HERE, 'native-evaluability-snapshot.json'); + +const CORE_DOMAIN = 'src/packages/core-domain'; +const TRIAGE_MODULE = `${CORE_DOMAIN}/test/rule-corpus-triage.ts`; +const TSCONFIG = `${CORE_DOMAIN}/tsconfig.json`; + +/** The classes Core can emit. A capture holding anything else is not a capture. */ +const KNOWN_CLASSES = [ + 'native-handler', + 'unimplemented-native', + 'needs-runtime', + 'needs-external-system', + 'documentation-only', + 'underspecified', +]; + +/** + * A corpus scan that finds nothing is never a real state — it is a moved path. + * Refuse to overwrite a good snapshot with the result of one. + */ +const MIN_PLAUSIBLE_CORPUS = 300; + +/** + * Packages that DECLARE ts-node, most-likely location first. + * + * `ts-node` is a devDependency of core-api and mcp-server, not of the root and + * not of core-domain. It normally lands in the root `node_modules` because npm + * workspaces hoist it, and resolving it from the repo root works — until a + * version conflict makes npm install it under the declaring package instead, at + * which point a root-relative `-r ts-node/register` fails on a clean runner and + * nowhere else. Resolving through the packages that actually declare it removes + * the dependency on hoisting. + */ +const TS_NODE_DECLARED_BY = [ + 'package.json', + 'src/packages/mcp-server/package.json', + 'src/apps/core-api/package.json', +]; + +/** Absolute path to ts-node's CommonJS register hook, or a stated failure. */ +function resolveTsNodeRegister() { + const tried = []; + for (const manifest of TS_NODE_DECLARED_BY) { + const from = path.join(REPO_ROOT, manifest); + if (!fs.existsSync(from)) continue; + try { + return createRequire(from).resolve('ts-node/register'); + } catch { + tried.push(manifest); + } + } + throw new Error( + 'ts-node is not resolvable, so Core\'s triage cannot be executed and the snapshot cannot be captured.\n' + + ` looked from: ${tried.join(', ') || '(no manifest found)'}\n` + + ' Install the workspace first: npm ci', + ); +} + +// --------------------------------------------------------------------------- +// 1. Run the real triage +// --------------------------------------------------------------------------- + +/** + * Execute Core's triage in a child process and read back its result as JSON. + * + * A child process rather than an import: this file is ESM under `src/rulesets`, + * Core is CommonJS TypeScript compiled by `ts-node`, and the boundary between + * them is a process, not a module resolution problem worth solving. + */ +function runTriage(capturedOn) { + // GT-633 RECONCILIATION: this script used to BUILD the document itself, while + // `rule-corpus-triage.spec.ts` built a second copy of it. Two generators for one + // artifact is the defect GT-633 exists to remove, one level up — whichever ran + // last would win and the other's `--check` would go red for no visible reason. + // + // The renderer now lives once, in `test/rule-corpus-triage.ts`, and this script + // is a thin driver over it: it supplies the sticky `capturedOn` and writes the + // bytes. The spec pins the same bytes. One renderer, two callers. + const program = ` + const { triageCorpus, renderSnapshot } = require(${JSON.stringify(path.join(REPO_ROOT, TRIAGE_MODULE))}); + const triage = triageCorpus(); + const classes = {}; + const conflicts = []; + for (const c of triage.classified) { + if (classes[c.ruleId] && classes[c.ruleId] !== c.evaluability) { + conflicts.push(c.ruleId + ': ' + classes[c.ruleId] + ' vs ' + c.evaluability); + } + classes[c.ruleId] = c.evaluability; + } + process.stdout.write(JSON.stringify({ + corpusSize: triage.corpus.length, + counts: triage.summary.byClass, + classes, + conflicts, + document: renderSnapshot(triage, { capturedOn: ${JSON.stringify(capturedOn)} }), + })); + `; + + const register = resolveTsNodeRegister(); + + let raw; + try { + raw = execFileSync(process.execPath, ['-r', register, '-e', program], { + cwd: REPO_ROOT, + encoding: 'utf8', + maxBuffer: 32 * 1024 * 1024, + env: { + ...process.env, + TS_NODE_PROJECT: path.join(REPO_ROOT, TSCONFIG), + TS_NODE_TRANSPILE_ONLY: 'true', + }, + stdio: ['ignore', 'pipe', 'inherit'], + }); + } catch (err) { + throw new Error( + `Could not run the Core triage (${TRIAGE_MODULE}). This script needs the workspace ` + + `installed — ts-node and Core's dependencies must be resolvable from ${REPO_ROOT}.\n${err.message}`, + ); + } + + const result = JSON.parse(raw); + + if (result.conflicts.length > 0) { + throw new Error( + 'The same rule id is classified two different ways by the corpus, so a snapshot keyed by ' + + `rule id cannot represent it:\n - ${result.conflicts.join('\n - ')}`, + ); + } + if (result.corpusSize < MIN_PLAUSIBLE_CORPUS) { + throw new Error( + `The triage classified only ${result.corpusSize} rules (expected at least ${MIN_PLAUSIBLE_CORPUS}). ` + + 'A collapsed corpus is a moved path, not progress — refusing to capture it.', + ); + } + for (const [id, klass] of Object.entries(result.classes)) { + if (!KNOWN_CLASSES.includes(klass)) throw new Error(`${id} was classified as unknown class "${klass}".`); + } + + const summed = Object.values(result.counts).reduce((a, b) => a + b, 0); + if (summed !== result.corpusSize) { + throw new Error(`Core's own class counts sum to ${summed} but it classified ${result.corpusSize} rules.`); + } + + return result; +} + +// --------------------------------------------------------------------------- +// 2. Render the snapshot +// --------------------------------------------------------------------------- + +/** + * Keep the capture date stable when nothing else changed. + * + * `capturedOn` is provenance, not content: bumping it on a run that produced an + * identical classification would turn every recapture into a diff and make real + * drift harder to see in review. + */ +function today() { + return new Date().toISOString().slice(0, 10); +} + +/** + * Has the CLASSIFICATION moved, as opposed to the provenance? + * + * `capturedOn` is provenance, not content: bumping it on a run that produced an + * identical classification would turn every recapture into a diff and make real + * drift harder to see in review. It is also what keeps the GT-630 chain's + * fixed-point replay byte-identical. + */ +function classificationChanged(previous, triage) { + if (!previous) return true; + const counts = Object.fromEntries( + KNOWN_CLASSES.filter((c) => triage.counts[c] !== undefined).map((c) => [c, triage.counts[c]]), + ); + return ( + JSON.stringify(previous.counts) !== JSON.stringify(counts) || + JSON.stringify(previous.classes) !== JSON.stringify(triage.classes) + ); +} + +/** + * The document, from the ONE renderer in `test/rule-corpus-triage.ts`. + * + * The date is a chicken-and-egg: stickiness is decided by whether the + * classification moved, which is only known after the triage runs, but the + * renderer needs the date as input. Rather than pay for a second ts-node run, + * the triage is rendered once with the PREVIOUS date and, if the classification + * did move, that single field is replaced. `JSON.parse` preserves key insertion + * order, so re-stringifying with the same formatting reproduces the renderer's + * bytes exactly — verified by `--check` being a byte comparison. + */ +function documentFrom(triage, previous, changed) { + if (!changed) return triage.document; + const doc = JSON.parse(triage.document); + doc.capturedOn = today(); + return JSON.stringify(doc, null, 2) + '\n'; +} + +// --------------------------------------------------------------------------- +// 3. Entry point +// --------------------------------------------------------------------------- + +const CHECK = process.argv.includes('--check'); + +/** + * Read the committed snapshot ONCE, and keep both views of that one read. + * + * Not `existsSync` then `readFileSync`, and not a second read for the byte + * comparison: `--check` decides whether the file on disk equals a fresh capture, + * so it must compare the bytes it actually parsed. Two reads can observe two + * different files — the generator itself writes this path — and then the diff + * reported would not be the diff that was measured. Absence is a normal state + * (first capture), so ENOENT is handled and every other error is raised. + */ +function readCommittedSnapshot() { + try { + return fs.readFileSync(OUT, 'utf8'); + } catch (err) { + if (err.code === 'ENOENT') return undefined; + throw err; + } +} + +const onDisk = readCommittedSnapshot(); +const previous = onDisk ? JSON.parse(onDisk) : undefined; +const triage = runTriage(previous?.capturedOn ?? today()); +const changed = classificationChanged(previous, triage); +const json = documentFrom(triage, previous, changed); + +if (CHECK) { + if (onDisk !== json) { + console.error( + 'GT-598: native-evaluability-snapshot.json no longer matches a fresh capture of the Core triage.\n' + + ' Re-run: node src/rulesets/standards/capture-native-evaluability-snapshot.mjs\n' + + ' Then: node src/rulesets/standards/build-iso-5055-mapping.mjs', + ); + if (previous) { + for (const c of KNOWN_CLASSES) { + const was = previous.counts?.[c]; + const now = triage.counts[c]; + if (was !== now) console.error(` ${c}: snapshot says ${was ?? '(absent)'}, Core says ${now ?? '(absent)'}`); + } + } + process.exit(1); + } + console.log(`GT-598: native-evaluability-snapshot.json is a faithful capture — ${triage.corpusSize} rules.`); +} else { + fs.writeFileSync(OUT, json); + console.log( + `${changed ? 'Recaptured' : 'Unchanged'} ${path.relative(process.cwd(), OUT)} — ` + + `${triage.corpusSize} rules: ${KNOWN_CLASSES.filter((c) => triage.counts[c] !== undefined).map((c) => `${c} ${triage.counts[c]}`).join(', ')}`, + ); + if (changed) console.log('Now rebuild the mapping: node src/rulesets/standards/build-iso-5055-mapping.mjs'); +} diff --git a/src/rulesets/standards/iso-5055-mapping.json b/src/rulesets/standards/iso-5055-mapping.json index c6c8f5d7..94113ed2 100644 --- a/src/rulesets/standards/iso-5055-mapping.json +++ b/src/rulesets/standards/iso-5055-mapping.json @@ -86,7 +86,7 @@ "src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)", "src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)", "src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)", - "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (corpus loader, renderer and byte-for-byte pin)" + "src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, classification and this renderer)" ], "note": "The handler backlog is the `unimplemented-native` class only. `documentation-only` and `underspecified` rules are not handler work at all, and the two adapter classes are closed by the enforcer seam rather than by a rule handler.", "realBacklogSize": 48, diff --git a/src/rulesets/standards/iso-5055-mapping.test.mjs b/src/rulesets/standards/iso-5055-mapping.test.mjs index eabce76c..7edc334b 100644 --- a/src/rulesets/standards/iso-5055-mapping.test.mjs +++ b/src/rulesets/standards/iso-5055-mapping.test.mjs @@ -7,15 +7,21 @@ * * These assertions are the reason the mapping is worth anything. A mapping table * that silently stops covering the corpus is worse than no table: it reports a - * shrinking denominator as progress. Five things are pinned here — + * shrinking denominator as progress. Six things are pinned here — * * 1. the weakness index really is the 138 of ISO/IEC 5055, per measure; * 2. every CWE the mapping cites is one of those 138; * 3. every rule in the corpus has a row (add a rule, this fails); - * 4. the evaluability capture still agrees with the counts Core pins, read - * out of Core's suite rather than copied into this file; - * 5. nothing in this directory quotes the retired 2011 eight-characteristic + * 4. the evaluability snapshot still agrees with the counts Core pins, read + * OUT OF CORE rather than retyped here (see `pinnedByCore`); + * 5. the snapshot's counts, its per-rule classes and the classes stamped into + * the mapping are three views of one capture, and they agree; + * 6. nothing in this directory quotes the retired 2011 eight-characteristic * ISO/IEC 25010 model. + * + * This job runs without node_modules, so nothing here recomputes the triage. The + * recomputing guard is `rule-corpus-triage.spec.ts` in core-domain, which + * compares the committed snapshot against a fresh classification of the corpus. */ import test from 'node:test'; @@ -106,14 +112,24 @@ test('the handler backlog is scoped to the unimplemented-native class, not to th }); /** - * The class counts Core pins, read out of Core's own suite. + * Read the class counts Core pins, from Core. + * + * This used to be six numbers typed into this file — the same six that are + * literals inside the snapshot it was checking. It compared the snapshot to + * itself, so it passed while `documentation-only` sat at 129 and Core had long + * since moved to 136, and it passed again when twelve rules got handlers and the + * snapshot went on calling them `unimplemented-native`. A guard whose expected + * value is a copy of its actual value is not a guard. + * + * This job has no node_modules, so the counts cannot be recomputed here — they + * are read out of the spec that pins them. Parsing a source file is a real + * coupling and it is stated rather than hidden: if the const is renamed or + * reshaped, this THROWS. Failing to find the source of truth must never read as + * agreement with it. * - * This used to be six literals typed into this file, which made the test below - * compare the snapshot against a second copy of the snapshot's own numbers: it - * passed while Core moved 139 -> 151 and the capture stayed at 139. A guard - * that cannot see the thing it guards is worse than no guard, because it - * reports agreement it never checked. So the numbers come from the source that - * owns them, and a pin this file cannot find is a failure, never a skip. + * The recomputation-based check lives where the dependencies do: the same spec + * asserts the committed snapshot against a freshly computed triage, and + * `capture-native-evaluability-snapshot.mjs --check` re-derives it end to end. */ function pinnedByCore() { const spec = path.resolve( @@ -121,34 +137,89 @@ function pinnedByCore() { '..', 'packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts', ); - assert.ok(fs.existsSync(spec), `Core's triage suite is missing at ${spec} — this guard has no authority to read`); + if (!fs.existsSync(spec)) { + throw new Error(`Core's pinned counts are unreadable: ${spec} does not exist. The snapshot cannot be verified.`); + } - const text = fs.readFileSync(spec, 'utf8'); - const open = text.indexOf('expect(SUMMARY.byClass).toEqual({'); - assert.notEqual(open, -1, `no \`expect(SUMMARY.byClass).toEqual({...})\` pin found in ${path.basename(spec)}`); - const close = text.indexOf('});', open); - assert.notEqual(close, -1, 'the pinned byClass object is not closed'); + const source = fs.readFileSync(spec, 'utf8'); + const literal = /const\s+PINNED_CLASS_COUNTS\s*:[^=]*=\s*\{([^}]*)\}\s*;/.exec(source); + if (!literal) { + throw new Error( + `Could not find the PINNED_CLASS_COUNTS literal in ${path.basename(spec)}. ` + + 'It was renamed or reshaped — update this parser rather than deleting the check.', + ); + } const counts = {}; - const body = text.slice(text.indexOf('{', open) + 1, close); - for (const [, klass, n] of body.matchAll(/'?([a-z-]+)'?\s*:\s*(\d+)/g)) { - counts[klass] = Number(n); + for (const line of literal[1].split('\n')) { + const entry = /^\s*'?([a-zA-Z-]+)'?\s*:\s*(\d+)\s*,?\s*$/.exec(line); + if (entry) counts[entry[1]] = Number(entry[2]); + } + if (Object.keys(counts).length === 0) { + throw new Error(`Parsed the PINNED_CLASS_COUNTS block in ${path.basename(spec)} and read zero counts out of it.`); } - assert.equal(Object.keys(counts).length, 6, `expected six evaluability classes, parsed ${JSON.stringify(counts)}`); return counts; } test('the evaluability snapshot still matches the class counts pinned by Core', () => { - // The capture is rendered by rule-corpus-triage.spec.ts and pinned there - // byte-for-byte against the live triage; this end re-checks the numbers so a - // divergence is red from src/rulesets too, without importing a package this - // directory does not own. Recapture with: - // UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts - const pinned = pinnedByCore(); - assert.deepEqual(evaluability.counts, pinned, 'native-evaluability-snapshot.json has fallen behind Core — recapture it'); - - const total = Object.values(pinned).reduce((a, b) => a + b, 0); - assert.equal(Object.keys(evaluability.classes).length, total, 'the snapshot classifies a different corpus than Core counts'); + assert.deepEqual( + evaluability.counts, + pinnedByCore(), + 'native-evaluability-snapshot.json disagrees with the counts pinned in rule-corpus-triage.spec.ts. ' + + 'Recapture it (node src/rulesets/standards/capture-native-evaluability-snapshot.mjs), THEN rebuild the ' + + 'mapping (node src/rulesets/standards/build-iso-5055-mapping.mjs) — the generator stamps ' + + 'nativeEvaluability per rule from the snapshot, so rebuilding first launders the stale class into the mapping.', + ); +}); + +test('the snapshot is internally consistent — its header describes its own body', () => { + // The counts and the per-rule classes are two views of one capture, and the + // mapping is stamped from the SECOND one. A header that agrees with Core while + // the body has drifted would put the wrong class on every affected row. + const summed = Object.values(evaluability.counts).reduce((a, b) => a + b, 0); + assert.equal(summed, evaluability.corpusSize, 'the class counts do not sum to the corpus the capture measured'); + assert.equal(Object.keys(evaluability.classes).length, evaluability.distinctRuleIds); + assert.ok( + evaluability.distinctRuleIds <= evaluability.corpusSize, + 'more distinct rule ids than rules classified — the capture is not a capture', + ); + + const tally = {}; + for (const klass of Object.values(evaluability.classes)) tally[klass] = (tally[klass] ?? 0) + 1; + + if (evaluability.distinctRuleIds === evaluability.corpusSize) { + // The usual case: one row per rule id, so the two views must agree exactly. + assert.deepEqual(tally, evaluability.counts, 'the per-rule classes do not add up to the stated counts'); + } else { + // A rule id carried by two ruleset files collapses to one key here, so the + // tally is a lower bound rather than an equality. + for (const [klass, n] of Object.entries(tally)) { + assert.ok(n <= evaluability.counts[klass], `${klass}: ${n} classes recorded but only ${evaluability.counts[klass]} counted`); + } + } +}); + +test('every rule the mapping stamps a class onto is a rule the snapshot actually classified', () => { + // `not-in-snapshot` is a legitimate value for the two single-rule + // infrastructure files whose rule metadata sits at the document root, which + // the Core corpus loader does not read. It is NOT a legitimate value for + // anything else: when the snapshot falls behind the corpus, rows quietly + // acquire it and drop out of the backlog. So the exception is enumerated. + const orphans = mapping.rules.filter((r) => r.nativeEvaluability === 'not-in-snapshot').map((r) => r.ruleId).sort(); + assert.deepEqual( + orphans, + ['INFRA-001', 'INFRA-OPA-001'], + `mapping rows carry no class from the snapshot: ${orphans.join(', ')}. Either the snapshot is stale ` + + '(recapture it) or the corpus grew another rule shape Core does not load (then say so here).', + ); + for (const r of mapping.rules) { + if (r.nativeEvaluability === 'not-in-snapshot') continue; + assert.equal( + r.nativeEvaluability, + evaluability.classes[r.ruleId], + `${r.ruleId} is stamped ${r.nativeEvaluability} in the mapping but ${evaluability.classes[r.ruleId]} in the snapshot`, + ); + } }); test('no artifact here quotes the retired 2011 eight-characteristic ISO/IEC 25010 model', () => { diff --git a/src/rulesets/standards/native-evaluability-snapshot.json b/src/rulesets/standards/native-evaluability-snapshot.json index 06ae1184..942fce71 100644 --- a/src/rulesets/standards/native-evaluability-snapshot.json +++ b/src/rulesets/standards/native-evaluability-snapshot.json @@ -3,15 +3,17 @@ "title": "Native-engine evaluability class per rule (snapshot)", "description": "Per-rule evaluability class as computed by the Core native evaluator triage. This is a GENERATED CAPTURE, not the source of truth: the authority is src/packages/core-domain/src/application/validators/rule-evaluability.ts and the handler set registered in native-evaluator.ts. It is recorded here so the ISO/IEC 5055 mapping can be scoped to the real handler backlog without src/rulesets depending on a package it does not own. Do not hand-edit — regenerate.", "version": "1.1.0", - "capturedOn": "2026-07-29", + "capturedOn": "2026-07-30", "capturedFrom": [ "src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)", "src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)", "src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)", - "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (corpus loader, renderer and byte-for-byte pin)" + "src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, classification and this renderer)" ], - "regenerateWith": "UPDATE_EVALUABILITY_SNAPSHOT=1 npx jest src/application/validators/rule-corpus-triage.spec.ts (from src/packages/core-domain)", - "validation": "Rendered by rule-corpus-triage.spec.ts from the live triage and pinned there byte-for-byte, so a divergence between this file and Core is a failing test rather than silent drift (corpus 386; native-handler 151, documentation-only 136, unimplemented-native 48, needs-external-system 20, needs-runtime 17, underspecified 14).", + "regenerateWith": "node src/rulesets/standards/capture-native-evaluability-snapshot.mjs", + "validation": "Rendered by test/rule-corpus-triage.ts from the live triage, written by capture-native-evaluability-snapshot.mjs and pinned byte-for-byte by rule-corpus-triage.spec.ts, so a divergence between this file and Core is a failing test rather than silent drift (corpus 386; native-handler 151, documentation-only 136, unimplemented-native 48, needs-external-system 20, needs-runtime 17, underspecified 14).", + "corpusSize": 386, + "distinctRuleIds": 386, "counts": { "native-handler": 151, "documentation-only": 136,