From 10b525130bce9f77f35cb55875b1ae51e66ceb68 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 14:41:42 -0500 Subject: [PATCH 01/27] feat(cli): prove the machine channel across the registry, and govern the exit taxonomy (GT-580) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the two open criteria of GT-580. CRITERION 2 — `--format json` writes DATA ONLY to stdout. Output formatting and process exits were ALREADY centralized: `installMachineChannelGuard` (main.ts, armed before the command graph boots) reroutes every non-JSON stdout write to stderr while a machine format is active, and `exit-codes.ts` + `BaseEvolithCommand.handleError` map a thrown error onto the taxonomy in one place. What was missing was proof that the centralization actually holds across the CLI: the existing subprocess spec pipes two hand-picked commands. `machine-channel.registry-sweep.spec.ts` asks commander — the CLI's own registry, built from the real AppModule — which commands declare `--format`, and sweeps all 32 in real subprocesses with stdout and stderr on separate pipes. Each must hand back exactly one parseable JSON document and exit from the taxonomy. The enumeration has a floor, so a sweep over nothing fails instead of passing vacuously. Observed red: commenting out `installMachineChannelGuard()` turns `agents` (a clack banner) and `init-wizard` (a progress line) DIRTY while the other thirty stay clean. CRITERION 3 — a ruleset with Rego parity fails a command that exits outside the taxonomy. - `src/rulesets/cli/exit-code-taxonomy.rules.json` — CLI-EXIT-01 (no exit outside {0,1,2,3}), CLI-EXIT-02 (the scan must be non-vacuous), CLI-EXIT-03 (the taxonomy may not be widened to absorb an offender — otherwise CLI-EXIT-01 has a one-line escape). - `src/rulesets/opa/cli-exit-code-taxonomy.rego` + tests — the same three ids decided over the fact document the CLI's own producer emits. Aggregated into `main.rego`, and scoped to `input.core.cli` so a satellite evaluation stays silent. - `CliExitTaxonomyRuleHandler` — the native half, so `evolith validate` evaluates the rules instead of reporting three blocking rules that never ran. Observed red, end to end: planting `process.exit(7)` in a real CLI command file makes `evolith validate --core .` emit a BLOCKING CLI-EXIT-01 naming the file, and the CLI exits 2 (BLOCKED). The same file drives the committed producer and the committed .rego to the same verdict in `exit-code-taxonomy-ruleset.spec.ts`. Derived artifacts moved with the corpus: native-handler 151 -> 154, ISO/IEC 5055 mapping and the inventory summary regenerated. Co-Authored-By: Claude Opus 5 --- .../maturity-reports/inventory-summary.es.md | 4 +- .../maturity-reports/inventory-summary.md | 4 +- .../cli-exit-taxonomy-rule.handler.spec.ts | 173 +++++++++++++ .../cli-exit-taxonomy-rule.handler.ts | 183 +++++++++++++ .../validators/evaluators/native-evaluator.ts | 5 + .../validators/rule-corpus-triage.spec.ts | 7 +- .../cli-exit-code-taxonomy.fixture.json | 106 ++++++++ src/rulesets/cli/README.es.md | 1 + src/rulesets/cli/README.md | 1 + .../cli/exit-code-taxonomy.rules.json | 51 ++++ src/rulesets/opa/cli-exit-code-taxonomy.rego | 78 ++++++ .../opa/cli-exit-code-taxonomy.test.rego | 78 ++++++ src/rulesets/opa/main.rego | 7 + src/rulesets/standards/iso-5055-mapping.csv | 10 + src/rulesets/standards/iso-5055-mapping.json | 210 ++++++++++++++- .../cli/exit-code-taxonomy-ruleset.spec.ts | 215 +++++++++++++++ .../machine-channel.registry-sweep.spec.ts | 245 ++++++++++++++++++ 17 files changed, 1363 insertions(+), 15 deletions(-) create mode 100644 src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.spec.ts create mode 100644 src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.ts create mode 100644 src/packages/core-domain/test/parity-fixtures/cli-exit-code-taxonomy.fixture.json create mode 100644 src/rulesets/cli/exit-code-taxonomy.rules.json create mode 100644 src/rulesets/opa/cli-exit-code-taxonomy.rego create mode 100644 src/rulesets/opa/cli-exit-code-taxonomy.test.rego create mode 100644 src/sdk/cli/src/infrastructure/cli/exit-code-taxonomy-ruleset.spec.ts create mode 100644 src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts diff --git a/reference/core/control-center/maturity-reports/inventory-summary.es.md b/reference/core/control-center/maturity-reports/inventory-summary.es.md index c16996e6f..d7f8b3dc9 100644 --- a/reference/core/control-center/maturity-reports/inventory-summary.es.md +++ b/reference/core/control-center/maturity-reports/inventory-summary.es.md @@ -8,7 +8,7 @@ Este es el conteo automatizado del inventario de la arquitectura de referencia c | Tipo de Artefacto | Conteo | Ubicación | |---|:---:|---| | **Architecture Decision Records (ADR)** | 139 | `reference/core/architecture/adrs/` | -| **Rulesets Legibles por Máquina** | 167 | `src/rulesets/` (en 20 categorías) | +| **Rulesets Legibles por Máquina** | 175 | `src/rulesets/` (en 20 categorías) | | **Schemas de Phase-Gates** | 45 | `src/rulesets/schema/` | -*Última Actualización: 2026-07-28* +*Última Actualización: 2026-07-29* diff --git a/reference/core/control-center/maturity-reports/inventory-summary.md b/reference/core/control-center/maturity-reports/inventory-summary.md index 06211375d..a39afefd2 100644 --- a/reference/core/control-center/maturity-reports/inventory-summary.md +++ b/reference/core/control-center/maturity-reports/inventory-summary.md @@ -8,7 +8,7 @@ This is the automated inventory tally of the core reference architecture and gov | Artifact Type | Count | Location | |---|:---:|---| | **Architecture Decision Records (ADR)** | 139 | `reference/core/architecture/adrs/` | -| **Machine-Readable Rulesets** | 167 | `src/rulesets/` (across 20 categories) | +| **Machine-Readable Rulesets** | 175 | `src/rulesets/` (across 20 categories) | | **Phase-Gate Schemas** | 45 | `src/rulesets/schema/` | -*Last Updated: 2026-07-28* +*Last Updated: 2026-07-29* diff --git a/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.spec.ts b/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.spec.ts new file mode 100644 index 000000000..ce3163f44 --- /dev/null +++ b/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.spec.ts @@ -0,0 +1,173 @@ +import * as path from 'path'; +import { CliExitTaxonomyRuleHandler } from './cli-exit-taxonomy-rule.handler'; +import { IFileSystem } from '../../../../domain/interfaces'; +import { NormalizedRule } from '../../../../domain/models/normalized-rule'; + +/** + * GT-580 — the rule must be SEEN failing. + * + * Every scenario below plants a scratch CLI tree and asks the handler for a + * verdict. The one that matters is `an out-of-taxonomy exit`: a command file + * carrying `process.exit(7)` has to come back `failed`, because an assertion + * that cannot fail is the defect this backlog is about. + */ + +const CORE = '/core'; +const CTX = { satellitePath: '/sat', corePath: CORE }; +const CLI_SRC = path.join(CORE, 'src', 'sdk', 'cli', 'src'); + +/** The real `exit-codes.ts` shape, so CLI-EXIT-03 parses something authentic. */ +const EXIT_CODES_TS = ` +export const CLI_EXIT_CODES = { + OK: 0, + TOOL_FAILURE: 1, + BLOCKED: 2, + INVALID_INPUT: 3, +} as const; +`; + +/** + * An in-memory tree: `files` maps an absolute path to its contents, and every + * directory on the way is derived rather than declared, so a scenario cannot + * forget one and accidentally scan nothing. + */ +function treeFs(files: Record): IFileSystem { + const dirs = new Map>(); + for (const file of Object.keys(files)) { + let current = file; + for (;;) { + const parent = path.dirname(current); + if (parent === current) break; + if (!dirs.has(parent)) dirs.set(parent, new Set()); + dirs.get(parent)!.add(path.basename(current)); + current = parent; + } + } + + return { + exists: jest.fn(async (p: string) => p in files || dirs.has(p)), + readFile: jest.fn(async (p: string) => files[p] ?? ''), + readdirNames: jest.fn(async (p: string) => [...(dirs.get(p) ?? [])]), + stat: jest.fn(async (p: string) => ({ + isDirectory: () => dirs.has(p) && !(p in files), + isFile: () => p in files, + })), + } as unknown as IFileSystem; +} + +function rule(id: string): NormalizedRule { + return { + id, + severity: 'MUST', + category: 'cli-exit-taxonomy', + title: 't', + description: 'd', + blocking: true, + sourceFile: 'cli/exit-code-taxonomy.rules.json', + }; +} + +/** A CLI tree that obeys the taxonomy. */ +function compliantTree(extra: Record = {}): Record { + return { + [path.join(CLI_SRC, 'infrastructure', 'cli', 'exit-codes.ts')]: EXIT_CODES_TS, + [path.join(CLI_SRC, 'commands', 'gate', 'gate.command.ts')]: + 'if (blocked) process.exit(2);\nprocess.exitCode = 0;\n', + [path.join(CLI_SRC, 'main.ts')]: 'process.exit(1);\n', + ...extra, + }; +} + +describe('GT-580 · CliExitTaxonomyRuleHandler', () => { + it('claims the CLI-EXIT- family and nothing else', () => { + const handler = new CliExitTaxonomyRuleHandler(treeFs(compliantTree())); + expect(handler.canHandle(rule('CLI-EXIT-01'))).toBe(true); + expect(handler.canHandle(rule('CLI-EXIT-03'))).toBe(true); + expect(handler.canHandle(rule('CLI-RR-01'))).toBe(false); + expect(handler.canHandle(rule('DEP-01'))).toBe(false); + }); + + it('passes all three rules on a CLI that obeys the taxonomy', async () => { + const handler = new CliExitTaxonomyRuleHandler(treeFs(compliantTree())); + for (const id of ['CLI-EXIT-01', 'CLI-EXIT-02', 'CLI-EXIT-03']) { + const result = await handler.evaluate(rule(id), CTX); + expect({ id, result: result.result }).toEqual({ id, result: 'passed' }); + } + }); + + // ---- the red ----------------------------------------------------------- + + it('FAILS CLI-EXIT-01 on a command that exits outside the taxonomy', async () => { + const rogue = path.join(CLI_SRC, 'commands', 'rogue', 'rogue.command.ts'); + const handler = new CliExitTaxonomyRuleHandler( + treeFs(compliantTree({ [rogue]: 'export function boom(): void {\n process.exit(7);\n}\n' })), + ); + + const result = await handler.evaluate(rule('CLI-EXIT-01'), CTX); + + expect(result.result).toBe('failed'); + // The finding must name the offender: "1 rule failed" sends nobody anywhere. + expect(result.message).toContain('commands/rogue/rogue.command.ts'); + expect(result.message).toContain('process.exit(7)'); + }); + + it('FAILS CLI-EXIT-01 on `process.exitCode = 64` too, not only on process.exit()', async () => { + const rogue = path.join(CLI_SRC, 'commands', 'rogue', 'other.command.ts'); + const handler = new CliExitTaxonomyRuleHandler( + treeFs(compliantTree({ [rogue]: 'process.exitCode = 64;\n' })), + ); + + const result = await handler.evaluate(rule('CLI-EXIT-01'), CTX); + expect(result.result).toBe('failed'); + expect(result.message).toContain('process.exitCode = 64'); + }); + + it('ignores test sources, which legitimately name codes outside the taxonomy', async () => { + // Including the negative fixtures that prove this very rule can fail — if + // they counted, the rule could never be green anywhere. + const handler = new CliExitTaxonomyRuleHandler( + treeFs( + compliantTree({ + [path.join(CLI_SRC, 'commands', 'rogue', 'rogue.spec.ts')]: 'process.exit(9);\n', + [path.join(CLI_SRC, 'commands', 'rogue', 'rogue.test.ts')]: 'process.exit(77);\n', + }), + ), + ); + + expect((await handler.evaluate(rule('CLI-EXIT-01'), CTX)).result).toBe('passed'); + }); + + it('FAILS CLI-EXIT-02 when the scan reads nothing', async () => { + // The source root exists but holds no `.ts` — the shape a moved directory + // produces, and the one that would otherwise report perfect compliance. + const handler = new CliExitTaxonomyRuleHandler( + treeFs({ [path.join(CLI_SRC, 'README.md')]: '# not a source' }), + ); + + const result = await handler.evaluate(rule('CLI-EXIT-02'), CTX); + expect(result.result).toBe('failed'); + expect(result.message).toContain('0 sources'); + }); + + it('FAILS rather than skips when the CLI source root is absent', async () => { + // `skipped` on a blocking rule is the silent pass this rule exists against. + const handler = new CliExitTaxonomyRuleHandler(treeFs({ '/elsewhere/file.ts': '' })); + for (const id of ['CLI-EXIT-01', 'CLI-EXIT-02', 'CLI-EXIT-03']) { + const result = await handler.evaluate(rule(id), CTX); + expect({ id, result: result.result }).toEqual({ id, result: 'failed' }); + } + }); + + it('FAILS CLI-EXIT-03 when the taxonomy is widened to absorb the offender', async () => { + // The escape hatch: declare 7 legal and CLI-EXIT-01 goes quiet. This is the + // rule that stops that from being a fix. + const widened = EXIT_CODES_TS.replace('} as const;', ' ROGUE: 7,\n} as const;'); + const handler = new CliExitTaxonomyRuleHandler( + treeFs(compliantTree({ [path.join(CLI_SRC, 'infrastructure', 'cli', 'exit-codes.ts')]: widened })), + ); + + const result = await handler.evaluate(rule('CLI-EXIT-03'), CTX); + expect(result.result).toBe('failed'); + expect(result.message).toContain('[0, 1, 2, 3, 7]'); + }); +}); diff --git a/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.ts b/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.ts new file mode 100644 index 000000000..3582f2285 --- /dev/null +++ b/src/packages/core-domain/src/application/validators/evaluators/handlers/cli-exit-taxonomy-rule.handler.ts @@ -0,0 +1,183 @@ +import * as path from 'path'; +import { IFileSystem } from '../../../../domain/interfaces'; +import { NormalizedRule } from '../../../../domain/models/normalized-rule'; +import { WorkspaceEvaluationContext, RuleEvaluationResult } from '../evaluator.interface'; +import { INativeRuleHandler } from './rule-handler.interface'; + +/** + * GT-580 — the native half of `src/rulesets/cli/exit-code-taxonomy.rules.json`. + * + * The exit code is the only control primitive an agent harness, a pre-commit + * hook and a CI step have in common, and it only carries meaning while every + * command draws from one published set: `0` pass, `1` tool failure, `2` blocked, + * `3` invalid input. A jest scan inside the CLI package already refuses a source + * that names anything else — but that is a unit test of one package. It cannot + * be evaluated against a repository, it produces no finding, and it has no Rego + * parity, so a governance consumer has no way to ask the question at all. + * + * This handler is that missing half. `evolith validate` on the Core now + * evaluates CLI-EXIT-01/02/03 like any other rule, they are `blocking: true`, so + * a command that exits outside the taxonomy makes the run BLOCK (exit 2) rather + * than merely failing somebody's unit tests. + * + * PARITY. `src/rulesets/opa/cli-exit-code-taxonomy.rego` decides the same three + * ids from the fact document that `src/sdk/cli/scripts/exit-code-taxonomy-facts.mjs` + * emits. The scan below is the same scan: same file filter, same literal regex, + * same taxonomy. Rego receives facts and this handler produces them, which is + * why the duplication is a boundary rather than a fork — core-domain cannot + * import a script out of the CLI package without inverting the dependency. + */ +export class CliExitTaxonomyRuleHandler implements INativeRuleHandler { + /** The published taxonomy. Mirrored from `sdk/cli/src/infrastructure/cli/exit-codes.ts`. */ + private static readonly TAXONOMY: readonly number[] = [0, 1, 2, 3]; + + /** `process.exit(N)` / `process.exitCode = N` with a numeric literal. */ + private static readonly EXIT_LITERAL = /process\.exit\(\s*(-?\d+)\s*\)|process\.exitCode\s*=\s*(-?\d+)/g; + + constructor(private readonly fs: IFileSystem) {} + + canHandle(rule: NormalizedRule): boolean { + return rule.id.startsWith('CLI-EXIT-'); + } + + async evaluate(rule: NormalizedRule, ctx: WorkspaceEvaluationContext): Promise { + if (rule.id === 'CLI-EXIT-01') return this.evalNoOffenders(rule, ctx); + if (rule.id === 'CLI-EXIT-02') return this.evalNonVacuousScan(rule, ctx); + if (rule.id === 'CLI-EXIT-03') return this.evalTaxonomyNotWidened(rule, ctx); + + return { rule, result: 'skipped', message: 'Unhandled CLI-EXIT rule' }; + } + + /** CLI-EXIT-01 — every exit literal in the CLI is a member of the taxonomy. */ + private async evalNoOffenders(rule: NormalizedRule, ctx: WorkspaceEvaluationContext): Promise { + const root = this.cliSourceRoot(ctx); + if (!(await this.fs.exists(root))) { + // A missing tree is not compliance. Reported as failed rather than + // skipped, because `skipped` on a blocking rule is precisely the silent + // pass this rule exists to prevent. + return { rule, result: 'failed', message: `CLI source root not found: ${root}` }; + } + + const scan = await this.scan(root); + if (scan.offenders.length === 0) { + return { rule, result: 'passed' }; + } + + const named = scan.offenders + .map((o) => `${o.file}: ${o.snippet}`) + .join('; '); + return { + rule, + result: 'failed', + message: + `${scan.offenders.length} CLI source(s) exit outside the published taxonomy ` + + `{${CliExitTaxonomyRuleHandler.TAXONOMY.join(', ')}} — ${named}`, + }; + } + + /** CLI-EXIT-02 — a scan that read nothing found no offenders, which is not compliance. */ + private async evalNonVacuousScan(rule: NormalizedRule, ctx: WorkspaceEvaluationContext): Promise { + const root = this.cliSourceRoot(ctx); + if (!(await this.fs.exists(root))) { + return { rule, result: 'failed', message: `CLI source root not found: ${root}` }; + } + + const scan = await this.scan(root); + if (scan.scanned === 0) { + return { + rule, + result: 'failed', + message: `the exit-code taxonomy scan covered 0 sources under ${root} — an empty scan is not a pass`, + }; + } + return { rule, result: 'passed' }; + } + + /** + * CLI-EXIT-03 — the taxonomy has not been widened to absorb an offender. + * + * Read out of the CLI's own `exit-codes.ts` rather than trusted: the cheap way + * to silence CLI-EXIT-01 is to add the offending code to `CLI_EXIT_CODES`, and + * that makes the consumer's problem worse rather than better. + */ + private async evalTaxonomyNotWidened(rule: NormalizedRule, ctx: WorkspaceEvaluationContext): Promise { + const file = path.join(this.cliSourceRoot(ctx), 'infrastructure', 'cli', 'exit-codes.ts'); + if (!(await this.fs.exists(file))) { + return { rule, result: 'failed', message: `exit-code taxonomy not declared: ${file} not found` }; + } + + const source = await this.fs.readFile(file); + const block = /CLI_EXIT_CODES\s*=\s*\{([\s\S]*?)\}\s*as const/.exec(source); + if (!block) { + return { rule, result: 'failed', message: `could not read CLI_EXIT_CODES out of ${file}` }; + } + + const declared = [...block[1].matchAll(/:\s*(-?\d+)\s*,/g)] + .map((m) => Number(m[1])) + .sort((a, b) => a - b); + + const expected = [...CliExitTaxonomyRuleHandler.TAXONOMY]; + if (declared.length === expected.length && declared.every((v, i) => v === expected[i])) { + return { rule, result: 'passed' }; + } + return { + rule, + result: 'failed', + message: + `the CLI declares exit codes [${declared.join(', ')}]; the published taxonomy is exactly ` + + `[${expected.join(', ')}] and widening it is a governance decision, not a fix`, + }; + } + + private cliSourceRoot(ctx: WorkspaceEvaluationContext): string { + return path.join(ctx.corePath, 'src', 'sdk', 'cli', 'src'); + } + + private async scan(root: string): Promise<{ scanned: number; offenders: Array<{ file: string; code: number; snippet: string }> }> { + const files = await this.sourceFiles(root); + const offenders: Array<{ file: string; code: number; snippet: string }> = []; + + for (const file of files) { + const source = await this.fs.readFile(file); + // `matchAll` on a `/g` regex consumes it; a fresh instance per file keeps + // `lastIndex` from carrying between reads. + const pattern = new RegExp(CliExitTaxonomyRuleHandler.EXIT_LITERAL.source, 'g'); + for (const match of source.matchAll(pattern)) { + const code = Number(match[1] ?? match[2]); + if (!CliExitTaxonomyRuleHandler.TAXONOMY.includes(code)) { + offenders.push({ file: path.relative(root, file), code, snippet: match[0] }); + } + } + } + + return { scanned: files.length, offenders }; + } + + /** + * Every non-test `.ts` source under the CLI. + * + * Tests are excluded because they legitimately stub and assert on exit codes + * outside the taxonomy — including the negative fixtures that prove this very + * rule can fail. + */ + private async sourceFiles(dir: string, depth = 0): Promise { + if (depth > 10) return []; + const out: string[] = []; + const entries = await this.fs.readdirNames(dir); + + for (const entry of entries) { + if (entry === 'node_modules' || entry === '__mocks__') continue; + const full = path.join(dir, entry); + const stat = await this.fs.stat(full); + if (stat.isDirectory()) { + out.push(...(await this.sourceFiles(full, depth + 1))); + continue; + } + if (!entry.endsWith('.ts')) continue; + if (entry.endsWith('.spec.ts') || entry.endsWith('.test.ts')) continue; + out.push(full); + } + + return out; + } +} diff --git a/src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts b/src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts index 8f5bf74cb..955bdf3c7 100644 --- a/src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts +++ b/src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts @@ -4,6 +4,7 @@ import { IRuleEvaluatorStrategy, WorkspaceEvaluationContext, RuleEvaluationResul import { INativeRuleHandler } from './handlers/rule-handler.interface'; import { EvidenceRuleHandler } from './handlers/evidence-rule.handler'; import { CliReleaseRuleHandler } from './handlers/cli-release-rule.handler'; +import { CliExitTaxonomyRuleHandler } from './handlers/cli-exit-taxonomy-rule.handler'; import { McpRuleHandler } from './handlers/mcp-rule.handler'; import { DependencyRuleHandler } from './handlers/dependency-rule.handler'; import { TaxonomyRuleHandler } from './handlers/taxonomy-rule.handler'; @@ -29,6 +30,10 @@ export class NativeEvaluator implements IRuleEvaluatorStrategy { this.handlers = [ new EvidenceRuleHandler(fs), new CliReleaseRuleHandler(fs), + // GT-580: CLI-EXIT-01/02/03 — the published exit-code taxonomy, evaluated + // against the CLI source tree so a command that exits outside it BLOCKS a + // run instead of only failing the CLI package's own unit tests. + new CliExitTaxonomyRuleHandler(fs), new McpRuleHandler(fs), new DependencyRuleHandler(fs), new TaxonomyRuleHandler(fs), diff --git a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts index ec71e39e1..e5ff64e73 100644 --- a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts +++ b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts @@ -198,8 +198,13 @@ describe('GT-595 · the published breakdown, with its denominator', () => { // `enforce` was dropped at normalization. It is now carried, and // `ModuleBoundaryRuleHandler` evaluates it. // `unimplemented-native` drops by the same twelve; nothing else moved. + // + // 151 -> 154 on 2026-07-29: GT-580 added `cli/exit-code-taxonomy.rules.json` + // (CLI-EXIT-01/02/03) with `CliExitTaxonomyRuleHandler` claiming all three, + // so the corpus grows by three and every one of them lands directly in + // `native-handler`. No other class moves: these ids did not exist before. expect(SUMMARY.byClass).toEqual({ - 'native-handler': 151, + 'native-handler': 154, 'documentation-only': 136, 'unimplemented-native': 48, 'needs-external-system': 20, diff --git a/src/packages/core-domain/test/parity-fixtures/cli-exit-code-taxonomy.fixture.json b/src/packages/core-domain/test/parity-fixtures/cli-exit-code-taxonomy.fixture.json new file mode 100644 index 000000000..dec7e6a1e --- /dev/null +++ b/src/packages/core-domain/test/parity-fixtures/cli-exit-code-taxonomy.fixture.json @@ -0,0 +1,106 @@ +{ + "domain": "cli-exit-code-taxonomy", + "description": "GT-580 — CLI exit-code taxonomy parity. The same three rule ids decided by src/rulesets/opa/cli-exit-code-taxonomy.rego over the fact document, and by CliExitTaxonomyRuleHandler over the source tree that produces it. The `rogue-exit-code` scenario is the criterion itself: a command exiting 7 must fail the gate.", + "rules": [ + { "id": "CLI-EXIT-01", "severity": "MUST", "category": "cli-exit-taxonomy", "title": "No CLI Command May Exit Outside The Published Taxonomy", "description": "Every exit literal must be 0, 1, 2 or 3", "blocking": true, "sourceFile": "cli/exit-code-taxonomy.rules.json" }, + { "id": "CLI-EXIT-02", "severity": "MUST", "category": "cli-exit-taxonomy", "title": "The Taxonomy Scan Must Be Non-Vacuous", "description": "A scan that reads nothing is not compliance", "blocking": true, "sourceFile": "cli/exit-code-taxonomy.rules.json" }, + { "id": "CLI-EXIT-03", "severity": "MUST", "category": "cli-exit-taxonomy", "title": "The Published Taxonomy Must Not Be Widened", "description": "CLI_EXIT_CODES must declare exactly 0, 1, 2, 3", "blocking": true, "sourceFile": "cli/exit-code-taxonomy.rules.json" } + ], + "scenarios": { + "compliant": { + "filesystem": { + "existing": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/main.ts", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts" + ], + "directories": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli" + ], + "dirs": { + "/core/src/sdk/cli/src": ["main.ts", "infrastructure"], + "/core/src/sdk/cli/src/infrastructure": ["cli"], + "/core/src/sdk/cli/src/infrastructure/cli": ["exit-codes.ts"] + }, + "files": { + "/core/src/sdk/cli/src/main.ts": "process.exit(1);\n", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts": "export const CLI_EXIT_CODES = {\n OK: 0,\n TOOL_FAILURE: 1,\n BLOCKED: 2,\n INVALID_INPUT: 3,\n} as const;\n" + } + }, + "context": { "satellitePath": "/sat", "corePath": "/core" }, + "expectedResults": [ + { "ruleId": "CLI-EXIT-01", "result": "passed" }, + { "ruleId": "CLI-EXIT-02", "result": "passed" }, + { "ruleId": "CLI-EXIT-03", "result": "passed" } + ] + }, + "rogue-exit-code": { + "filesystem": { + "existing": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/main.ts", + "/core/src/sdk/cli/src/rogue.command.ts", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts" + ], + "directories": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli" + ], + "dirs": { + "/core/src/sdk/cli/src": ["main.ts", "rogue.command.ts", "infrastructure"], + "/core/src/sdk/cli/src/infrastructure": ["cli"], + "/core/src/sdk/cli/src/infrastructure/cli": ["exit-codes.ts"] + }, + "files": { + "/core/src/sdk/cli/src/main.ts": "process.exit(1);\n", + "/core/src/sdk/cli/src/rogue.command.ts": "export function boom(): void {\n process.exit(7);\n}\n", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts": "export const CLI_EXIT_CODES = {\n OK: 0,\n TOOL_FAILURE: 1,\n BLOCKED: 2,\n INVALID_INPUT: 3,\n} as const;\n" + } + }, + "context": { "satellitePath": "/sat", "corePath": "/core" }, + "expectedResults": [ + { "ruleId": "CLI-EXIT-01", "result": "failed" }, + { "ruleId": "CLI-EXIT-02", "result": "passed" }, + { "ruleId": "CLI-EXIT-03", "result": "passed" } + ] + }, + "widened-taxonomy": { + "filesystem": { + "existing": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/main.ts", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts" + ], + "directories": [ + "/core/src/sdk/cli/src", + "/core/src/sdk/cli/src/infrastructure", + "/core/src/sdk/cli/src/infrastructure/cli" + ], + "dirs": { + "/core/src/sdk/cli/src": ["main.ts", "infrastructure"], + "/core/src/sdk/cli/src/infrastructure": ["cli"], + "/core/src/sdk/cli/src/infrastructure/cli": ["exit-codes.ts"] + }, + "files": { + "/core/src/sdk/cli/src/main.ts": "process.exit(7);\n", + "/core/src/sdk/cli/src/infrastructure/cli/exit-codes.ts": "export const CLI_EXIT_CODES = {\n OK: 0,\n TOOL_FAILURE: 1,\n BLOCKED: 2,\n INVALID_INPUT: 3,\n ROGUE: 7,\n} as const;\n" + } + }, + "context": { "satellitePath": "/sat", "corePath": "/core" }, + "expectedResults": [ + { "ruleId": "CLI-EXIT-01", "result": "failed" }, + { "ruleId": "CLI-EXIT-02", "result": "passed" }, + { "ruleId": "CLI-EXIT-03", "result": "failed" } + ] + } + } +} diff --git a/src/rulesets/cli/README.es.md b/src/rulesets/cli/README.es.md index d24a1c9c7..b8f0bdec5 100644 --- a/src/rulesets/cli/README.es.md +++ b/src/rulesets/cli/README.es.md @@ -10,6 +10,7 @@ Reglas legibles por máquina para preparación de release del Evolith Evolith CL |---|---| | [Preparación de Release del CLI](./release-readiness.rules.json) | Define evidencia mínima de build, pruebas, paquete y smoke MCP antes de liberar el CLI. | | [Paridad CLI/Core](./core-parity.rules.json) | Exige que cada capacidad de regla Core sea trazable a CLI, MCP, pruebas y estado de evidencia. | +| [Taxonomía de Exit Codes del CLI](./exit-code-taxonomy.rules.json) | Exige que todo comando del CLI salga con un código de la taxonomía publicada (`0` pass, `1` fallo de herramienta, `2` bloqueado, `3` entrada inválida), sobre un escaneo no vacío y sin ensanchar la taxonomía para absorber al infractor. Paridad Rego en [`opa/cli-exit-code-taxonomy.rego`](../opa/cli-exit-code-taxonomy.rego). | ## Intención de Validación diff --git a/src/rulesets/cli/README.md b/src/rulesets/cli/README.md index bcc0591da..4c67665ac 100644 --- a/src/rulesets/cli/README.md +++ b/src/rulesets/cli/README.md @@ -10,6 +10,7 @@ Machine-readable rules for Evolith Evolith CLI release readiness and parity with |---|---| | [CLI Release Readiness](./release-readiness.rules.json) | Defines minimum build, test, package, and MCP smoke evidence before CLI release. | | [CLI/Core Parity](./core-parity.rules.json) | Requires every Core rule capability to be traced to CLI, MCP, tests, and evidence status. | +| [CLI Exit-Code Taxonomy](./exit-code-taxonomy.rules.json) | Requires every CLI command to exit with a code drawn from the published taxonomy (`0` pass, `1` tool failure, `2` blocked, `3` invalid input), over a non-vacuous scan, without widening the taxonomy to absorb an offender. Rego parity in [`opa/cli-exit-code-taxonomy.rego`](../opa/cli-exit-code-taxonomy.rego). | ## Validation Intent diff --git a/src/rulesets/cli/exit-code-taxonomy.rules.json b/src/rulesets/cli/exit-code-taxonomy.rules.json new file mode 100644 index 000000000..6d12b3ceb --- /dev/null +++ b/src/rulesets/cli/exit-code-taxonomy.rules.json @@ -0,0 +1,51 @@ +{ + "$schema": "../schema/ruleset-standard.schema.json", + "$id": "https://evolith.dev/rulesets/cli/exit-code-taxonomy.rules.json", + "title": "CLI Exit-Code Taxonomy Rules", + "description": "GT-580 — the exit code is the only control primitive an agent harness, a pre-commit hook and a CI step have in common. This ruleset governs it: every command must exit with a code drawn from the published taxonomy (0 pass, 1 tool failure, 2 blocked, 3 invalid input), the scan that proves it must be non-vacuous, and the taxonomy itself must not be widened to make an offender disappear.", + "version": "1.0.0", + "effectiveDate": "2026-07-29", + "scope": "core-cli", + "category": "exit-code-taxonomy", + "audience": "core", + "rules": [ + { + "id": "CLI-EXIT-01", + "severity": "MUST", + "category": "cli-exit-taxonomy", + "title": "No CLI Command May Exit Outside The Published Taxonomy", + "description": "Every exit code named by a CLI source MUST be a member of the published taxonomy: 0 pass, 1 tool failure, 2 blocked, 3 invalid input. A command that exits with anything else is unreadable to the consumer the code exists for — an agent cannot tell a blocked gate from a crashed tool, which is the difference between a blocked merge and a retry.", + "validationQuery": "Scan every non-test .ts source under src/sdk/cli/src for `process.exit(N)` and `process.exitCode = N` with a numeric literal, and require every N to be one of 0, 1, 2, 3.", + "evidenceRequired": [ + "offending file and exit literal, per offender", + "number of sources scanned" + ], + "blocking": true + }, + { + "id": "CLI-EXIT-02", + "severity": "MUST", + "category": "cli-exit-taxonomy", + "title": "The Taxonomy Scan Must Be Non-Vacuous", + "description": "The scan behind CLI-EXIT-01 MUST cover a non-empty set of CLI sources. A scan that reads nothing also finds no offenders, so a moved or mistyped source root would report perfect compliance while checking an empty directory. The denominator is part of the verdict, not a diagnostic.", + "validationQuery": "Require the number of CLI sources scanned by the exit-code taxonomy scan to be greater than zero.", + "evidenceRequired": [ + "number of sources scanned", + "source root the scan resolved to" + ], + "blocking": true + }, + { + "id": "CLI-EXIT-03", + "severity": "MUST", + "category": "cli-exit-taxonomy", + "title": "The Published Taxonomy Must Not Be Widened", + "description": "The declared taxonomy MUST be exactly {0, 1, 2, 3}. Without this, CLI-EXIT-01 has a trivial escape hatch: adding the offending code to the taxonomy makes the violation disappear while the consumer's problem — an exit code carrying no agreed meaning — gets worse. Widening the taxonomy is a governance decision, so it must fail the gate rather than pass it silently.", + "validationQuery": "Require the taxonomy declared by src/sdk/cli/src/infrastructure/cli/exit-codes.ts to be exactly the four codes 0, 1, 2 and 3.", + "evidenceRequired": [ + "declared taxonomy values" + ], + "blocking": true + } + ] +} diff --git a/src/rulesets/opa/cli-exit-code-taxonomy.rego b/src/rulesets/opa/cli-exit-code-taxonomy.rego new file mode 100644 index 000000000..78361f1c8 --- /dev/null +++ b/src/rulesets/opa/cli-exit-code-taxonomy.rego @@ -0,0 +1,78 @@ +# GT-580 — the CLI exit-code taxonomy, as policy. +# +# Rego parity for `src/rulesets/cli/exit-code-taxonomy.rules.json`: the same three +# rule ids, the same verdicts, evaluated over the fact document produced by +# `src/sdk/cli/scripts/exit-code-taxonomy-facts.mjs --json` and mounted at +# `input.core.cli.exitCodes`. +# +# The policy deliberately does NOT re-implement the file scan. A policy that +# walked a source tree would be a second scanner drifting away from the first; +# facts in, verdict out is the whole point of keeping the producer separate. +# +# Published taxonomy: 0 pass · 1 tool failure · 2 blocked · 3 invalid input. +package evolith.cli_exit_code_taxonomy + +import rego.v1 + +facts := input.core.cli.exitCodes + +# CLI-EXIT-01 — a command that exits outside the taxonomy fails the gate. +# +# One violation per offender, so the message names the file and the literal +# rather than reporting an anonymous count somebody then has to go hunting for. +violations contains { + "id": "CLI-EXIT-01", + "message": sprintf( + "%s exits with %d, which is outside the published CLI exit-code taxonomy {0,1,2,3} (%s)", + [offender.file, offender.code, offender.snippet], + ), +} if { + some offender in facts.offenders +} + +# CLI-EXIT-02 — a scan that read nothing is not compliance. +# +# Without this, a moved source root turns CLI-EXIT-01 into a rule that can never +# fire: no files scanned means no offenders found, and the gate reports green +# over an empty directory. +violations contains { + "id": "CLI-EXIT-02", + "message": sprintf( + "the exit-code taxonomy scan covered %d CLI sources — a scan that reads nothing finds no offenders, which is not compliance", + [facts.scanned], + ), +} if { + facts.scanned == 0 +} + +# CLI-EXIT-03 — the taxonomy itself may not be widened. +# +# The cheap way to make CLI-EXIT-01 green is to declare the offending code part +# of the taxonomy. That makes the consumer's problem worse, not better, so it +# fails here instead of passing silently. +violations contains { + "id": "CLI-EXIT-03", + "message": sprintf( + "the CLI declares exit codes %v; the published taxonomy is exactly [0, 1, 2, 3] and widening it is a governance decision, not a fix", + [facts.declared], + ), +} if { + facts.declared != [0, 1, 2, 3] +} + +# Absence of facts is not compliance either: a consumer that forgot to attach the +# fact document must not read as a pass. +# +# Scoped to `input.core.cli` on purpose. This ruleset is `audience: core` — it +# addresses the Evolith monorepo's own CLI — and `main.rego` aggregates it, so an +# unscoped rule would fire on every satellite evaluation, where the CLI source +# tree does not exist and the fact document is meaningless. Declaring +# `input.core.cli` is the caller saying "I am evaluating the Core CLI"; having +# said it, omitting the facts is a defect rather than a non-applicable rule. +violations contains { + "id": "CLI-EXIT-02", + "message": "no exit-code taxonomy facts were supplied at input.core.cli.exitCodes — the rule cannot be evaluated, and an unevaluated blocking rule is not a pass", +} if { + input.core.cli + not input.core.cli.exitCodes +} diff --git a/src/rulesets/opa/cli-exit-code-taxonomy.test.rego b/src/rulesets/opa/cli-exit-code-taxonomy.test.rego new file mode 100644 index 000000000..0b5da48b3 --- /dev/null +++ b/src/rulesets/opa/cli-exit-code-taxonomy.test.rego @@ -0,0 +1,78 @@ +# GT-580 — executable tests for the CLI exit-code taxonomy policy. +# +# The negative cases are the point. A guard nobody has ever seen fail is not a +# guard, and this board has repeatedly caught exactly that. Each test below moves +# ONE field of a known-compliant fact document and names the rule it expects. +package evolith.cli_exit_code_taxonomy_test + +import rego.v1 + +import data.evolith.cli_exit_code_taxonomy + +# The real shape emitted by `exit-code-taxonomy-facts.mjs --json`, with the +# numbers measured against the CLI on 2026-07-29. +compliant_input := {"core": {"cli": {"exitCodes": { + "schemaVersion": "1.0.0", + "declared": [0, 1, 2, 3], + "observed": [0, 1, 2, 3], + "scanned": 152, + "offenders": [], + "compliant": true, +}}}} + +test_compliant_cli_has_no_violations if { + violations := cli_exit_code_taxonomy.violations with input as compliant_input + count(violations) == 0 +} + +# The criterion, stated as a test: a command that exits outside {0,1,2,3} fails. +test_command_exiting_outside_the_taxonomy_is_rejected if { + i := json.patch(compliant_input, [{ + "op": "replace", + "path": "/core/cli/exitCodes/offenders", + "value": [{"file": "commands/rogue/rogue.command.ts", "code": 7, "snippet": "process.exit(7)"}], + }]) + violations := cli_exit_code_taxonomy.violations with input as i + violations[_].id == "CLI-EXIT-01" +} + +test_every_offender_is_named_individually if { + i := json.patch(compliant_input, [{ + "op": "replace", + "path": "/core/cli/exitCodes/offenders", + "value": [ + {"file": "commands/rogue/a.command.ts", "code": 7, "snippet": "process.exit(7)"}, + {"file": "commands/rogue/b.command.ts", "code": 64, "snippet": "process.exitCode = 64"}, + ], + }]) + violations := cli_exit_code_taxonomy.violations with input as i + count(violations) == 2 +} + +test_a_vacuous_scan_is_rejected if { + i := json.patch(compliant_input, [{"op": "replace", "path": "/core/cli/exitCodes/scanned", "value": 0}]) + violations := cli_exit_code_taxonomy.violations with input as i + violations[_].id == "CLI-EXIT-02" +} + +test_missing_facts_are_rejected_rather_than_read_as_a_pass if { + violations := cli_exit_code_taxonomy.violations with input as {"core": {"cli": {"mcpServerSource": "x"}}} + violations[_].id == "CLI-EXIT-02" +} + +# The mirror image, and the reason the rule above is scoped: `main.rego` +# aggregates this policy, so a satellite evaluation — which has no Core CLI tree +# and no fact document — must produce nothing at all rather than three blocking +# violations about a source tree it does not contain. +test_a_satellite_evaluation_is_silent if { + violations := cli_exit_code_taxonomy.violations with input as {"satellite": {"contracts": {"phase": 2}}} + count(violations) == 0 +} + +# The escape hatch: declaring 7 part of the taxonomy would silence CLI-EXIT-01. +# It must not silence the gate. +test_widening_the_taxonomy_is_rejected if { + i := json.patch(compliant_input, [{"op": "replace", "path": "/core/cli/exitCodes/declared", "value": [0, 1, 2, 3, 7]}]) + violations := cli_exit_code_taxonomy.violations with input as i + violations[_].id == "CLI-EXIT-03" +} diff --git a/src/rulesets/opa/main.rego b/src/rulesets/opa/main.rego index 98372778d..aedf947a3 100644 --- a/src/rulesets/opa/main.rego +++ b/src/rulesets/opa/main.rego @@ -8,6 +8,7 @@ import data.evolith.capability_source_interface.violations as csi_violations import data.evolith.ci_cd.violations as ci_cd_violations import data.evolith.cicd_quality_gates.violations as cicd_qg_violations import data.evolith.cli_core_parity.violations as cli_cp_violations +import data.evolith.cli_exit_code_taxonomy.violations as cli_exit_violations import data.evolith.cli_readiness.violations as cli_violations import data.evolith.cli_release_readiness.violations as cli_rr_violations import data.evolith.compliance_baseline.violations as cb_violations @@ -157,3 +158,9 @@ violations contains v if { violations contains v if { v := pg_violations[_] } + +# GT-580 — the exit-code taxonomy. Silent unless the caller declares +# `input.core.cli`, so a satellite evaluation is unaffected. +violations contains v if { + v := cli_exit_violations[_] +} diff --git a/src/rulesets/standards/iso-5055-mapping.csv b/src/rulesets/standards/iso-5055-mapping.csv index 720cf8c8f..8f3395975 100644 --- a/src/rulesets/standards/iso-5055-mapping.csv +++ b/src/rulesets/standards/iso-5055-mapping.csv @@ -146,6 +146,13 @@ CORE-0113-01,adr/generated/adr-0113-node-js-platform-lighthouse-apache-2-0-as-th CORE-0115-01,adr/generated/adr-0115-emergent-knowledge-axis-knowledge-originated-by-applying-the.rules.json,architecture-decision,,,none,no,documentation-only CORE-0116-01,adr/generated/adr-0116-canonical-finding-contract-and-an-executable-advisory-author.rules.json,architecture-decision,,,none,no,documentation-only CORE-0117-01,adr/generated/adr-0117-bilingual-parity-applies-to-authored-sources-not-generated-p.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0118-01,adr/generated/adr-0118-the-core-hub-has-its-own-root-taxonomy-distinct-from-satelli.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0119-01,adr/generated/adr-0119-api-security-configuration-hardening.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0120-01,adr/generated/adr-0120-ssrf-prevention-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0121-01,adr/generated/adr-0121-input-validation-and-sanitization-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0122-01,adr/generated/adr-0122-shell-execution-safety-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0123-01,adr/generated/adr-0123-timing-safe-comparison-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0124-01,adr/generated/adr-0124-credential-and-secret-management-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot AI-0001-01,adr/generated/adr-ai-augmented-0001-harness-engineering-for-ai-augmented-development.rules.json,architecture-decision,,,none,no,documentation-only AI-0002-01,adr/generated/adr-ai-augmented-0002-mcp-integration-protocol-for-agent-tool-invocation.rules.json,architecture-decision,,,none,no,documentation-only AI-0003-01,adr/generated/adr-ai-augmented-0003-model-selection-governance-for-ai-augmented-workflows.rules.json,architecture-decision,,,none,no,documentation-only @@ -189,6 +196,9 @@ CLI-PAR-01,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-n CLI-PAR-02,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-native CLI-PAR-03,cli/core-parity.rules.json,platform-surface,,,none,no,needs-runtime CLI-PAR-04,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-native +CLI-EXIT-01,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot +CLI-EXIT-02,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot +CLI-EXIT-03,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot CLI-RR-01,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler CLI-RR-02,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler CLI-RR-03,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler diff --git a/src/rulesets/standards/iso-5055-mapping.json b/src/rulesets/standards/iso-5055-mapping.json index 3efd2456a..e2ea3623e 100644 --- a/src/rulesets/standards/iso-5055-mapping.json +++ b/src/rulesets/standards/iso-5055-mapping.json @@ -12,24 +12,24 @@ "weaknessCount": 138 }, "corpus": { - "rulesetFiles": 167, - "rules": 381, + "rulesetFiles": 175, + "rules": 391, "note": "Files that carry gate definitions or topology recommendations rather than conformance rules contribute no rows: architecture/topology-recommendation.rules.json and sdlc/phase-gates.rules.json." }, "summary": { - "rules": 381, + "rules": 391, "mappedToIso5055": 37, "mappedDirect": 8, "mappedPartial": 29, - "noInternationalEquivalent": 344, - "adoptedFraction": 0.0971, + "noInternationalEquivalent": 354, + "adoptedFraction": 0.0946, "analyserAdoptable": 42, "analyserAdoptablePartial": 23, - "analyserAdoptableFraction": 0.1102, - "analyserAdoptableFractionIncludingPartial": 0.1706, + "analyserAdoptableFraction": 0.1074, + "analyserAdoptableFractionIncludingPartial": 0.1662, "byClass": { "architecture-decision": { - "rules": 155, + "rules": 162, "mapped": 12, "adoptable": 6 }, @@ -59,7 +59,7 @@ "adoptable": 0 }, "platform-surface": { - "rules": 14, + "rules": 17, "mapped": 0, "adoptable": 0 }, @@ -217,7 +217,7 @@ ] }, "not-in-snapshot": { - "rules": 2, + "rules": 12, "mappedToIso5055": 0, "analyserAdoptable": 0, "analyserAdoptablePartial": 0, @@ -3185,6 +3185,139 @@ "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, + { + "ruleId": "CORE-0118-01", + "sourceFile": "adr/generated/adr-0118-the-core-hub-has-its-own-root-taxonomy-distinct-from-satelli.rules.json", + "title": "Conform to ADR-0118: The Core Hub Has Its Own Root Taxonomy, Distinct From Satellites", + "severity": "MUST", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0119-01", + "sourceFile": "adr/generated/adr-0119-api-security-configuration-hardening.rules.json", + "title": "Conform to ADR-0119: API Security Configuration Hardening", + "severity": "MUST", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0120-01", + "sourceFile": "adr/generated/adr-0120-ssrf-prevention-standard.rules.json", + "title": "Honor design decision in ADR-0120: SSRF Prevention Standard", + "severity": "SHOULD", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0121-01", + "sourceFile": "adr/generated/adr-0121-input-validation-and-sanitization-standard.rules.json", + "title": "Honor design decision in ADR-0121: Input Validation and Sanitization Standard", + "severity": "SHOULD", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0122-01", + "sourceFile": "adr/generated/adr-0122-shell-execution-safety-standard.rules.json", + "title": "Conform to ADR-0122: Shell Execution Safety Standard", + "severity": "MUST", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0123-01", + "sourceFile": "adr/generated/adr-0123-timing-safe-comparison-standard.rules.json", + "title": "Conform to ADR-0123: Timing-Safe Comparison Standard", + "severity": "MUST", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, + { + "ruleId": "CORE-0124-01", + "sourceFile": "adr/generated/adr-0124-credential-and-secret-management-standard.rules.json", + "title": "Conform to ADR-0124: Credential and Secret Management Standard", + "severity": "MUST", + "ruleClass": "architecture-decision", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." + }, { "ruleId": "AI-0001-01", "sourceFile": "adr/generated/adr-ai-augmented-0001-harness-engineering-for-ai-augmented-development.rules.json", @@ -4017,6 +4150,63 @@ "nativeEvaluability": "unimplemented-native", "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." }, + { + "ruleId": "CLI-EXIT-01", + "sourceFile": "cli/exit-code-taxonomy.rules.json", + "title": "No CLI Command May Exit Outside The Published Taxonomy", + "severity": "MUST", + "ruleClass": "platform-surface", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." + }, + { + "ruleId": "CLI-EXIT-02", + "sourceFile": "cli/exit-code-taxonomy.rules.json", + "title": "The Taxonomy Scan Must Be Non-Vacuous", + "severity": "MUST", + "ruleClass": "platform-surface", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." + }, + { + "ruleId": "CLI-EXIT-03", + "sourceFile": "cli/exit-code-taxonomy.rules.json", + "title": "The Published Taxonomy Must Not Be Widened", + "severity": "MUST", + "ruleClass": "platform-surface", + "iso5055": { + "cwes": [], + "weaknesses": [], + "measures": [], + "strength": "none" + }, + "analyser": { + "adoptable": "no", + "examples": [] + }, + "nativeEvaluability": "not-in-snapshot", + "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." + }, { "ruleId": "CLI-RR-01", "sourceFile": "cli/release-readiness.rules.json", diff --git a/src/sdk/cli/src/infrastructure/cli/exit-code-taxonomy-ruleset.spec.ts b/src/sdk/cli/src/infrastructure/cli/exit-code-taxonomy-ruleset.spec.ts new file mode 100644 index 000000000..058f46975 --- /dev/null +++ b/src/sdk/cli/src/infrastructure/cli/exit-code-taxonomy-ruleset.spec.ts @@ -0,0 +1,215 @@ +/** + * GT-580 criterion 3, end to end — the RULESET fails a command that exits + * outside the taxonomy. + * + * Three artifacts have to agree for the criterion to be met, and each one alone + * proves nothing: + * + * src/rulesets/cli/exit-code-taxonomy.rules.json the authored rule + * src/rulesets/opa/cli-exit-code-taxonomy.rego its Rego parity + * src/sdk/cli/scripts/exit-code-taxonomy-facts.mjs the facts it decides over + * + * `opa test` already exercises the policy against hand-written fact documents, + * and `CliExitTaxonomyRuleHandler` already scans a tree. Neither answers the + * question this spec asks: does a REAL out-of-taxonomy exit, in a REAL file on + * disk, travel through the REAL producer and come out of the REAL policy as a + * blocking violation? A hand-written fixture cannot answer that — it asserts + * that the policy handles the input somebody imagined the producer emits. + * + * So each case below plants a scratch CLI tree, runs the committed producer over + * it, and feeds the result to the committed `.rego` through the pinned OPA + * binary. The `rogue command` case is the criterion stated literally. + * + * WHY A MISSING OPA IS RED, NOT SKIPPED. The binary is resolved through the + * repository's own `ensureOpa`, which fetches and pins it — the same path every + * OPA gate in CI takes. A spec that skipped itself when the runtime were absent + * would be green on exactly the machine where nothing was verified, which is the + * failure mode this board keeps finding. + */ +import { spawnSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const CLI_ROOT = path.resolve(__dirname, '..', '..', '..'); +const REPO_ROOT = path.resolve(CLI_ROOT, '..', '..', '..'); +const FACTS_SCRIPT = path.join(CLI_ROOT, 'scripts', 'exit-code-taxonomy-facts.mjs'); +const POLICY = path.join(REPO_ROOT, 'src', 'rulesets', 'opa', 'cli-exit-code-taxonomy.rego'); +const OPA_RUNTIME = path.join(REPO_ROOT, '.harness', 'scripts', 'opa-runtime.mjs'); +const CLI_SRC = path.join(CLI_ROOT, 'src'); + +/** The published taxonomy, as the CLI's own `exit-codes.ts` declares it. */ +const TAXONOMY_TS = [ + 'export const CLI_EXIT_CODES = {', + ' OK: 0,', + ' TOOL_FAILURE: 1,', + ' BLOCKED: 2,', + ' INVALID_INPUT: 3,', + '} as const;', + '', +].join('\n'); + +// A possible one-off OPA download on a cold machine. +jest.setTimeout(300_000); + +interface Violation { + id: string; + message: string; +} + +/** + * Resolve the pinned OPA binary through the repository's own runtime helper. + * + * Done in a subprocess because `opa-runtime.mjs` is ESM and this suite is + * transpiled to CJS; shelling out keeps the helper as the single source of the + * pinned version instead of hard-coding a path that would drift from it. + */ +function resolveOpaBinary(): string { + const probe = spawnSync( + process.execPath, + [ + '--input-type=module', + '-e', + `import { ensureOpa } from ${JSON.stringify(OPA_RUNTIME)};` + + `const opa = await ensureOpa(${JSON.stringify(REPO_ROOT)});` + + `process.stdout.write(opa.binary);`, + ], + { encoding: 'utf8', cwd: REPO_ROOT }, + ); + if (probe.status !== 0) { + throw new Error(`could not resolve the pinned OPA binary:\n${probe.stderr}`); + } + return probe.stdout.trim(); +} + +/** Run the committed fact producer over a source tree. */ +function collectFacts(root: string): Record { + const probe = spawnSync( + process.execPath, + [ + '--input-type=module', + '-e', + `import { collectExitCodeFacts } from ${JSON.stringify(FACTS_SCRIPT)};` + + `process.stdout.write(JSON.stringify(collectExitCodeFacts(${JSON.stringify(root)})));`, + ], + { encoding: 'utf8' }, + ); + if (probe.status !== 0) throw new Error(`fact producer failed:\n${probe.stderr}`); + return JSON.parse(probe.stdout) as Record; +} + +/** Ask the committed policy for its verdict on those facts. */ +function evaluatePolicy(opa: string, facts: Record, workdir: string): Violation[] { + const inputFile = path.join(workdir, 'input.json'); + fs.writeFileSync(inputFile, JSON.stringify({ core: { cli: { exitCodes: facts } } })); + + const run = spawnSync( + opa, + [ + 'eval', + '--format', 'json', + '--data', POLICY, + '--input', inputFile, + 'data.evolith.cli_exit_code_taxonomy.violations', + ], + { encoding: 'utf8' }, + ); + if (run.status !== 0) throw new Error(`opa refused the policy:\n${run.stderr}`); + + const parsed = JSON.parse(run.stdout) as { + result?: Array<{ expressions?: Array<{ value?: Violation[] }> }>; + }; + return parsed.result?.[0]?.expressions?.[0]?.value ?? []; +} + +describe('GT-580 · the exit-code ruleset, from a real source tree to a Rego verdict', () => { + let opa: string; + let workdir: string; + + beforeAll(() => { + opa = resolveOpaBinary(); + expect(fs.existsSync(POLICY)).toBe(true); + expect(fs.existsSync(FACTS_SCRIPT)).toBe(true); + workdir = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gt580-ruleset-'))); + }); + + afterAll(() => { + fs.rmSync(workdir, { recursive: true, force: true }); + }); + + /** A scratch CLI tree with a declared taxonomy and whatever else is asked for. */ + function plant(name: string, sources: Record): string { + const root = path.join(workdir, name); + const all = { + 'infrastructure/cli/exit-codes.ts': TAXONOMY_TS, + ...sources, + }; + for (const [rel, contents] of Object.entries(all)) { + const full = path.join(root, rel); + fs.mkdirSync(path.dirname(full), { recursive: true }); + fs.writeFileSync(full, contents); + } + return root; + } + + it('clears the REAL CLI source tree, over a non-empty scan', () => { + const facts = collectFacts(CLI_SRC); + + // The denominator first: a scan that read nothing also finds no offenders. + expect(facts.scanned as number).toBeGreaterThan(80); + expect(evaluatePolicy(opa, facts, workdir)).toEqual([]); + }); + + it('FAILS on a command that exits outside the taxonomy', () => { + // The criterion, stated as an experiment: force an out-of-taxonomy exit in a + // scratch fixture and require the ruleset to go red. + const root = plant('rogue', { + 'commands/rogue/rogue.command.ts': 'export function boom(): void {\n process.exit(7);\n}\n', + }); + + const violations = evaluatePolicy(opa, collectFacts(root), workdir); + + expect(violations.map((v) => v.id)).toEqual(['CLI-EXIT-01']); + expect(violations[0].message).toContain('commands/rogue/rogue.command.ts'); + expect(violations[0].message).toContain('process.exit(7)'); + }); + + it('FAILS on `process.exitCode = 64` as well — the other spelling', () => { + const root = plant('rogue-assignment', { + 'commands/rogue/other.command.ts': 'process.exitCode = 64;\n', + }); + + const violations = evaluatePolicy(opa, collectFacts(root), workdir); + expect(violations.map((v) => v.id)).toEqual(['CLI-EXIT-01']); + expect(violations[0].message).toContain('process.exitCode = 64'); + }); + + it('does NOT let the taxonomy be widened to silence the offender', () => { + // The cheapest way to make the violation above disappear is to declare 7 + // legal. CLI-EXIT-03 is the rule that stops that from counting as a fix, so + // the tree that tries it fails on BOTH counts. + const root = plant('widened', { + 'commands/rogue/rogue.command.ts': 'process.exit(7);\n', + 'infrastructure/cli/exit-codes.ts': TAXONOMY_TS.replace('} as const;', ' ROGUE: 7,\n} as const;'), + }); + + const facts = collectFacts(root); + // The producer reads the taxonomy from its own module, so the widening is + // injected here to exercise the policy rule rather than the producer's + // hard-coded mirror of the CLI constant. + (facts as { declared: number[] }).declared = [0, 1, 2, 3, 7]; + + const ids = evaluatePolicy(opa, facts, workdir).map((v) => v.id).sort(); + expect(ids).toEqual(['CLI-EXIT-01', 'CLI-EXIT-03']); + }); + + it('refuses a vacuous scan rather than reporting it as compliance', () => { + const root = path.join(workdir, 'empty'); + fs.mkdirSync(root, { recursive: true }); + fs.writeFileSync(path.join(root, 'README.md'), '# no sources here'); + + const facts = collectFacts(root); + expect(facts.scanned).toBe(0); + expect(evaluatePolicy(opa, facts, workdir).map((v) => v.id)).toEqual(['CLI-EXIT-02']); + }); +}); diff --git a/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts new file mode 100644 index 000000000..43191329c --- /dev/null +++ b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts @@ -0,0 +1,245 @@ +/** + * GT-580 criterion 2 — `--format json` writes DATA ONLY to stdout, for EVERY + * command that offers the flag. + * + * WHY A SWEEP AND NOT A SAMPLE. `machine-channel.subprocess.spec.ts` already + * pipes two hand-picked commands (`scaffold`, `init`) and parses their stdout. + * That proves the guard works where it was pointed; it says nothing about the + * thirty other commands that accept `--format json`, and nothing at all about + * the command somebody adds next week. A machine consumer does not pipe the two + * commands we happened to test — it pipes whichever one it needs, and one stray + * progress line makes `JSON.parse` throw on the first byte while the command + * still exits 0. + * + * WHY THE ENUMERATION COMES FROM THE REGISTRY. A hand-written list of commands + * is a second source of truth that silently stops covering the CLI the moment a + * command is added. This spec asks commander — the CLI's own registry, built + * from the real `AppModule` — which commands declare a `--format` option, and + * sweeps exactly those. A new `--format json` command is therefore in the sweep + * on the day it is registered, without anybody remembering to add it. + * + * WHY THE DENOMINATOR IS ASSERTED. A sweep over an empty enumeration passes and + * proves nothing — this board has repeatedly caught guards that had never once + * been observed to fail. If the registry walk ever returns nothing (a changed + * commander internal, a broken module graph), the floor assertion turns this + * spec RED instead of letting it report a vacuous green. + * + * WHAT COUNTS AS CONFORMANCE. For each command: stdout must be exactly one + * parseable JSON document — not "mostly JSON", not "JSON after a banner" — and + * the exit status must be a member of the published taxonomy. Most invocations + * here land on a FAILURE path (no satellite, no TTY, no required flag), which is + * deliberate: the failure path is where prose leaks, and it is the path an agent + * harness most needs to parse. + * + * OBSERVED RED. Commenting out `installMachineChannelGuard()` in `main.ts` and + * re-running this sweep on 2026-07-29 turned `agents` (`┌ Evolith …`, a + * `@clack/prompts` banner) and `init-wizard` (`Starting Evolith …`) DIRTY while + * the other thirty stayed clean. The guard is what makes this spec green, and + * removing it is caught. + */ +import { spawn } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CommandFactory } from 'nest-commander'; +import { Commander } from 'nest-commander/src/constants'; +import { CLI_EXIT_CODE_VALUES } from './exit-codes'; + +const CLI_ROOT = path.resolve(__dirname, '..', '..', '..'); +const REPO_ROOT = path.resolve(CLI_ROOT, '..', '..', '..'); +const TS_NODE_BIN = path.join(REPO_ROOT, 'node_modules', 'ts-node', 'dist', 'bin.js'); +const CLI_ENTRY = path.join(CLI_ROOT, 'src', 'main.ts'); +const TS_CONFIG = path.join(CLI_ROOT, 'tsconfig.test.json'); + +/** + * The floor the enumeration may not fall through. + * + * 32 commands declared `--format` when this spec was written. The floor sits + * just below so that adding commands never breaks the spec, while losing the + * enumeration entirely — the failure mode that would make the sweep vacuous — + * fails it immediately. + */ +const MIN_FORMAT_COMMANDS = 28; + +/** Six cold ts-node boots at a time; the whole sweep is ~30 process starts. */ +const CONCURRENCY = 6; +jest.setTimeout(900_000); + +/** + * Positional arguments for the commands whose `` is required. + * + * These exist so the invocation reaches the command's own code instead of + * stopping at commander's arity check. The values are read-only actions on + * purpose: the sweep must not mutate anything, and a command that refuses the + * argument is still a valid observation — its refusal must be a JSON document. + */ +const REQUIRED_ARGS: Readonly> = { + 'patterns get': ['no-such-pattern'], + 'patterns for-topology': ['modular-monolith'], + gate: ['status'], + phase: ['status'], + profile: ['show'], + enforce: ['status'], +}; + +interface RegisteredCommand { + /** Space-joined path as typed, e.g. `topology recommend`. */ + readonly path: string; + readonly formatFlags: string; +} + +/** + * Ask the CLI's own commander registry which commands accept `--format`. + * + * Built from the real `AppModule`, so this is the same graph `main.ts` boots — + * not a directory listing of `commands/`, which would also count files that + * register nothing. + */ +async function enumerateFormatCommands(): Promise { + const { AppModule } = await import('../../app.module'); + const app = await CommandFactory.createWithoutRunning(AppModule, { logger: false }); + try { + // Structural access: the root workspace hoists a different major of + // `commander` than nest-commander runs on, so its typings are not the + // contract here (the same reasoning as `applyProgramName` in main.ts). + const program = app.get(Commander, { strict: false }); + const found: RegisteredCommand[] = []; + + const walk = (command: CommanderLike, prefix: readonly string[]): void => { + const trail = [...prefix, command.name()]; + const format = (command.options ?? []).find((option) => option.long === '--format'); + if (format) found.push({ path: trail.join(' '), formatFlags: format.flags }); + for (const child of command.commands ?? []) walk(child, trail); + }; + + for (const child of program.commands ?? []) walk(child, []); + return found; + } finally { + await app.close(); + } +} + +interface CommanderLike { + name(): string; + readonly options?: ReadonlyArray<{ long?: string; flags: string }>; + readonly commands?: readonly CommanderLike[]; +} + +interface CliRun { + readonly command: string; + readonly stdout: string; + readonly stderr: string; + readonly status: number | null; +} + +function runCli(command: RegisteredCommand): Promise { + return new Promise((resolve, reject) => { + // A fresh empty cwd per command: none of these may depend on, or touch, the + // repository they are being tested from. + const cwd = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gt580-sweep-'))); + // The jest setup sets EVOLITH_FORCE_INTERACTIVE for in-process specs. A + // piped subprocess is the machine case by definition and must not inherit it. + const env: NodeJS.ProcessEnv = { ...process.env, NO_COLOR: '1' }; + delete env.EVOLITH_FORCE_INTERACTIVE; + + const argv = [ + TS_NODE_BIN, + '--transpile-only', + '--project', + TS_CONFIG, + CLI_ENTRY, + ...command.path.split(' '), + ...(REQUIRED_ARGS[command.path] ?? []), + '--format', + 'json', + ]; + + // `pipe` on both is the whole point: the two channels must be observed apart. + const child = spawn(process.execPath, argv, { cwd, stdio: ['ignore', 'pipe', 'pipe'], env }); + let stdout = ''; + let stderr = ''; + child.stdout.setEncoding('utf8'); + child.stderr.setEncoding('utf8'); + child.stdout.on('data', (chunk: string) => (stdout += chunk)); + child.stderr.on('data', (chunk: string) => (stderr += chunk)); + child.on('error', reject); + child.on('close', (status) => { + fs.rmSync(cwd, { recursive: true, force: true }); + resolve({ command: command.path, stdout, stderr, status }); + }); + }); +} + +async function runAll(commands: readonly RegisteredCommand[]): Promise { + const pending = [...commands]; + const runs: CliRun[] = []; + const worker = async (): Promise => { + for (let next = pending.shift(); next; next = pending.shift()) { + runs.push(await runCli(next)); + } + }; + await Promise.all(Array.from({ length: CONCURRENCY }, worker)); + return runs; +} + +/** The classification, stated once so both assertions read the same way. */ +function classify(run: CliRun): string { + const text = run.stdout.trim(); + if (text.length === 0) return 'EMPTY (no document for a machine consumer to read)'; + try { + const parsed: unknown = JSON.parse(text); + if (parsed === null || typeof parsed !== 'object') return 'NOT-A-DOCUMENT (bare scalar)'; + return 'JSON'; + } catch (error) { + const preview = text.slice(0, 60).replace(/\n/g, '\\n'); + return `DIRTY (${(error as Error).message}) first bytes: ${preview}`; + } +} + +describe('GT-580 · every `--format json` command in the registry pipes cleanly', () => { + let commands: RegisteredCommand[]; + let runs: CliRun[]; + + beforeAll(async () => { + // A missing runner must be a RED test, never a skipped one. + expect(fs.existsSync(TS_NODE_BIN)).toBe(true); + commands = await enumerateFormatCommands(); + runs = await runAll(commands); + }); + + it('enumerates the `--format` commands from the CLI registry, and finds many', () => { + // Guards the denominator: everything below is vacuous if this is empty. + expect(commands.length).toBeGreaterThanOrEqual(MIN_FORMAT_COMMANDS); + // Spot-check the walk actually descends into sub-commands rather than + // listing only top-level ones. + const paths = commands.map((c) => c.path); + expect(paths).toEqual(expect.arrayContaining(['validate', 'evaluate', 'topology recommend', 'patterns list'])); + expect(runs).toHaveLength(commands.length); + }); + + it('writes exactly one JSON document to stdout, for every one of them', () => { + const offenders = runs + .map((run) => ({ command: run.command, verdict: classify(run) })) + .filter((entry) => entry.verdict !== 'JSON'); + + // Reported as a list so a regression names every command it broke, not just + // the first one jest happened to reach. + expect(offenders).toEqual([]); + }); + + it('exits from the published taxonomy, for every one of them', () => { + const offenders = runs + .filter((run) => !CLI_EXIT_CODE_VALUES.includes(run.status ?? -1)) + .map((run) => ({ command: run.command, status: run.status })); + + expect(offenders).toEqual([]); + }); + + it('keeps diagnostics on stderr rather than deleting them', () => { + // The guard must be a channel rule, not a blunt silencer: the cheap "fix" + // for a dirty stdout is to stop printing, and that would pass the two + // assertions above while destroying every diagnostic a human needs. + const withDiagnostics = runs.filter((run) => run.stderr.trim().length > 0); + expect(withDiagnostics.length).toBeGreaterThan(0); + }); +}); From b319a6bac7c2a5b2a59258b97c10b991c2416255 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 15:44:51 -0500 Subject: [PATCH 02/27] fix(cli): make exit code agree with envelope.success across the taxonomy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GT-580's exit-code taxonomy guards catch a status outside {0,1,2,3}, but three commands stayed inside the taxonomy while contradicting their own --format json envelope: gate-status exited 0 on an INTERNAL_ERROR envelope, alias exited 1 (tool failure) instead of 3 (invalid input) on a usage error, and init-wizard's catch-all collapsed a NonInteractiveError to INTERNAL_ERROR/1 instead of preserving its VALIDATION_FAILED/3 classification. The registry sweep spec now asserts envelope.success against status === 0 for every --format json command, which also caught sdlc generate exiting 0 with success:false — its JSON branch returned before reaching the process.exitCode assignment below it. --- .../cli/src/commands/alias/alias.command.ts | 5 ++-- src/sdk/cli/src/commands/init/init.wizard.ts | 14 ++++++++-- .../src/commands/sdlc/gate-status.command.ts | 2 ++ .../commands/sdlc/generate-domain.command.ts | 11 ++++---- .../machine-channel.registry-sweep.spec.ts | 28 ++++++++++++++++++- 5 files changed, 49 insertions(+), 11 deletions(-) diff --git a/src/sdk/cli/src/commands/alias/alias.command.ts b/src/sdk/cli/src/commands/alias/alias.command.ts index 6ca978087..31f566028 100644 --- a/src/sdk/cli/src/commands/alias/alias.command.ts +++ b/src/sdk/cli/src/commands/alias/alias.command.ts @@ -5,6 +5,7 @@ import { randomUUID } from 'node:crypto'; import { AliasService } from '../../config/alias.service'; import { createSuccessEnvelope, createErrorEnvelope, OUTPUT_ENVELOPE_SCHEMA_VERSION } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; import { BaseEvolithCommand } from '../../infrastructure/cli/base-command'; +import { exitCodeForErrorCode } from '../../infrastructure/cli/exit-codes'; interface AliasCommandOptions { add?: string; @@ -38,7 +39,7 @@ export class AliasCommand extends BaseEvolithCommand { if (!alias || !cmd) { const message = 'Usage: evolith alias --add ='; if (json) { - process.exitCode = 1; + process.exitCode = exitCodeForErrorCode('VALIDATION_FAILED'); console.log(JSON.stringify(createErrorEnvelope('VALIDATION_FAILED', message, { ...meta, durationMs: Date.now() - startedAt }), null, 2)); } else { this.promptService.showError(message); @@ -94,7 +95,7 @@ export class AliasCommand extends BaseEvolithCommand { } } else { if (json) { - process.exitCode = 1; + process.exitCode = exitCodeForErrorCode('VALIDATION_FAILED'); console.log(JSON.stringify(createErrorEnvelope('VALIDATION_FAILED', 'Use --add, --remove, or --list', { ...meta, durationMs: Date.now() - startedAt }), null, 2)); } else { this.promptService.showInfo('Use --add, --remove, or --list.'); diff --git a/src/sdk/cli/src/commands/init/init.wizard.ts b/src/sdk/cli/src/commands/init/init.wizard.ts index a9639659b..231880286 100644 --- a/src/sdk/cli/src/commands/init/init.wizard.ts +++ b/src/sdk/cli/src/commands/init/init.wizard.ts @@ -6,8 +6,9 @@ import { PromptService } from '../../infrastructure/prompts/prompt.service'; import { CatalogLoader } from '../../infrastructure/catalog/catalog-loader'; import { Inject } from '@nestjs/common'; import { IFileSystem } from '@beyondnet/evolith-core-domain/domain/interfaces'; -import { createSuccessEnvelope, createErrorEnvelope, OUTPUT_ENVELOPE_SCHEMA_VERSION } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; +import { createSuccessEnvelope, createErrorEnvelope, OUTPUT_ENVELOPE_SCHEMA_VERSION, type ErrorCode } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; import { InitializeProjectUseCase } from '@beyondnet/evolith-core-domain/application/services'; +import { carriesCliExitCode, resolveExitCode } from '../../infrastructure/cli/exit-codes'; interface WizardInitOptions { wizard?: boolean; @@ -192,8 +193,15 @@ export class InitWizardCommand extends BaseEvolithCommand { } const message = error instanceof Error ? error.message : String(error); if (json) { - process.exitCode = 1; - console.log(JSON.stringify(createErrorEnvelope('INTERNAL_ERROR', message, { ...meta, durationMs: Date.now() - startedAt }), null, 2)); + // GT-580: a carrier error (e.g. `NonInteractiveError` on a non-TTY) + // already knows its taxonomy classification — collapsing it to + // INTERNAL_ERROR here would lose that and make the envelope disagree + // with the exit code. + const code: ErrorCode = carriesCliExitCode(error) && error.envelopeErrorCode + ? (error.envelopeErrorCode as ErrorCode) + : 'INTERNAL_ERROR'; + process.exitCode = resolveExitCode(error); + console.log(JSON.stringify(createErrorEnvelope(code, message, { ...meta, durationMs: Date.now() - startedAt }), null, 2)); } else { throw error; } diff --git a/src/sdk/cli/src/commands/sdlc/gate-status.command.ts b/src/sdk/cli/src/commands/sdlc/gate-status.command.ts index 55c696370..a115c6ada 100644 --- a/src/sdk/cli/src/commands/sdlc/gate-status.command.ts +++ b/src/sdk/cli/src/commands/sdlc/gate-status.command.ts @@ -13,6 +13,7 @@ import { } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; import { BaseEvolithCommand } from '../../infrastructure/cli/base-command'; import { resolveCoreOverride } from '../../infrastructure/paths/core-resolver'; +import { exitCodeForErrorCode } from '../../infrastructure/cli/exit-codes'; function ratingBadge(rating: DoraRating): string { switch (rating) { @@ -76,6 +77,7 @@ export class GateStatusCommand extends BaseEvolithCommand { } if (json) { const msg = error instanceof Error ? error.message : String(error); + process.exitCode = exitCodeForErrorCode('INTERNAL_ERROR'); console.log(JSON.stringify(createErrorEnvelope('INTERNAL_ERROR', msg, { ...meta, durationMs: Date.now() - startedAt }), null, 2)); return; } diff --git a/src/sdk/cli/src/commands/sdlc/generate-domain.command.ts b/src/sdk/cli/src/commands/sdlc/generate-domain.command.ts index 913857513..05632f5f7 100644 --- a/src/sdk/cli/src/commands/sdlc/generate-domain.command.ts +++ b/src/sdk/cli/src/commands/sdlc/generate-domain.command.ts @@ -11,6 +11,7 @@ import { OUTPUT_ENVELOPE_SCHEMA_VERSION, } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; import { BaseEvolithCommand } from '../../infrastructure/cli/base-command'; +import { exitCodeForErrorCode } from '../../infrastructure/cli/exit-codes'; @SubCommand({ name: 'generate', @@ -38,6 +39,11 @@ export class GenerateDomainCommand extends BaseEvolithCommand { }; if (!target || !fromFile) { + // A missing required arg is a user-input error, not an exceptional + // condition: render the guidance, mark the run failed, and return + // gracefully. Throwing here would re-render the error and emit a stack + // trace via BaseEvolithCommand.handleError, and reject command.run(). + process.exitCode = exitCodeForErrorCode('VALIDATION_FAILED'); if (json) { const msg = 'Both a generation target and a source file must be specified.'; console.log(JSON.stringify( @@ -49,11 +55,6 @@ export class GenerateDomainCommand extends BaseEvolithCommand { } this.promptService.showError('Both a generation target and a source file must be specified.'); this.promptService.showInfo(chalk.yellow('Example: evolith sdlc generate domain --from ddd-model.md')); - // A missing required arg is a user-input error, not an exceptional - // condition: render the guidance, mark the run failed, and return - // gracefully. Throwing here would re-render the error and emit a stack - // trace via BaseEvolithCommand.handleError, and reject command.run(). - process.exitCode = 1; return; } diff --git a/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts index 43191329c..216d594e8 100644 --- a/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts +++ b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts @@ -43,7 +43,7 @@ import * as os from 'node:os'; import * as path from 'node:path'; import { CommandFactory } from 'nest-commander'; import { Commander } from 'nest-commander/src/constants'; -import { CLI_EXIT_CODE_VALUES } from './exit-codes'; +import { CLI_EXIT_CODES, CLI_EXIT_CODE_VALUES } from './exit-codes'; const CLI_ROOT = path.resolve(__dirname, '..', '..', '..'); const REPO_ROOT = path.resolve(CLI_ROOT, '..', '..', '..'); @@ -235,6 +235,32 @@ describe('GT-580 · every `--format json` command in the registry pipes cleanly' expect(offenders).toEqual([]); }); + it('exits `0` iff the envelope says `success: true` — never a taxonomy member merely', () => { + // Membership in the taxonomy is necessary but not sufficient: a command + // that exits 0 with `success: false` (or a non-zero code with `success: + // true`) still hands a consumer branching on the exit code the wrong + // verdict, and the "member of the taxonomy" assertion above cannot see it + // because 0 and 1 and 2 and 3 are all valid members. This asserts the two + // facts the envelope and the exit code each claim actually agree. + const offenders: Array<{ command: string; status: number | null; success: unknown }> = []; + for (const run of runs) { + let envelope: unknown; + try { + envelope = JSON.parse(run.stdout.trim()); + } catch { + continue; // already reported by the "writes exactly one JSON document" assertion + } + if (envelope === null || typeof envelope !== 'object' || !('success' in envelope)) continue; + const success = (envelope as { success: unknown }).success; + const exitsOk = run.status === CLI_EXIT_CODES.OK; + if (success !== exitsOk) { + offenders.push({ command: run.command, status: run.status, success }); + } + } + + expect(offenders).toEqual([]); + }); + it('keeps diagnostics on stderr rather than deleting them', () => { // The guard must be a channel rule, not a blunt silencer: the cheap "fix" // for a dirty stdout is to stop printing, and that would pass the two From ad51f240436832b888e6ff7e072be763fb9817e9 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 15:51:10 -0500 Subject: [PATCH 03/27] docs(board): GT-580 criteria 2+3 closed for the JSON channel MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The registry sweep and Rego-parity ruleset landed in PR #278 close the stdout/stderr-purity half of criterion 2 (machine-channel.registry-sweep.spec.ts, 32/32 --format json commands clean) and criterion 3 (exit-code-taxonomy.rules.json + cli-exit-code-taxonomy.rego, opa test 7/7) — the taxonomy is now governed rather than merely scanned. Records the four envelope/exit-code mismatches the strengthened sweep found and fixed along the way. --format ndjson streaming and the Tracker cross-repo follow-up remain open, so status stays IN-PROGRESS. --- reference/core/control-center/gaps/gap-tracking.es.md | 2 +- reference/core/control-center/gaps/gap-tracking.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/core/control-center/gaps/gap-tracking.es.md b/reference/core/control-center/gaps/gap-tracking.es.md index 630ff9921..efd98d3b9 100644 --- a/reference/core/control-center/gaps/gap-tracking.es.md +++ b/reference/core/control-center/gaps/gap-tracking.es.md @@ -61,7 +61,7 @@ Este tablero es la única fuente de verdad para deuda técnica, gaps, oportunida | [`GT-576`](./gap-reference-catalog.es.md#gt-576) | **La autoevaluación que un comprador lee primero es falsificable en diez minutos, y el documento se autoincrimina.** `maturity-assessment.md` define *Validated (Weight 1.0) — Passing all quality gates, tests, and active in CI/CD*, y acto seguido marca **Pillar 1 Security "Level 4 (Managed) / Validated"** citando Row-Level Security multi-tenant (ADR-0010) y audit trails inmutables vía CDC (ADR-0016): `grep -rniE 'row.level.security\|current_setting\(\|debezium\|change data capture'` sobre `src` devuelve **CERO ficheros**, y core-api no declara driver de base de datos ni ORM. El **Pillar 4** está marcado Level 4 / Validated citando "builds deterministas de monorepo vía Nx" — no existe `nx.json` ni dependencia `nx`. También afirma paridad dual-engine 8/8 mientras el gate cubre 3 topologías y el paquete publicado trae políticas para 5. (La cita obsoleta de `opossum` en Pillar 3 es otro asunto — ese pilar está honestamente marcado `Designed`.) Fix: degradar Pillar 1 a `Designed` con los ADRs listados como intención, borrar la cita de Nx, reportar paridad contra el artefacto publicado, y añadir una regla mecánica a `09-reconcile-maturity.mjs`: una capacidad solo puede marcarse `Validated` si su lista de evidencia contiene al menos una referencia `file:line` o un job de CI, nunca un ADR solo. **COMPLETADO (`44fe8dd3`):** cada afirmación del enunciado se re-verificó contra el árbol y todas se sostuvieron. Pillar 1 (Security) y Pillar 4 bajaron de `Validated` a `Designed` con sus ADRs listados como intención, la cita de Nx eliminada, y la paridad reportada contra el artefacto publicado — en AMBOS idiomas, estructuralmente idénticos. La clase de error queda mecánicamente bloqueada: `09-reconcile-maturity.mjs` rechaza cualquier capacidad `Validated` cuya evidencia no contenga `file:line` ni job de CI, y esa regla lleva un auto-test negativo que prueba que se pone roja ante una entrada deliberadamente mala. Verificado: `09-reconcile-maturity.mjs --check` ✅, `04-check-bilingual-parity.mjs` ✅, gobernanza 17/17. | | | `Governance` | Cross | P1 | S | `COMPLETADO` | | [`GT-577`](./gap-reference-catalog.es.md#gt-577) | **El artefacto que un cliente cablearía a su CI se lee como una herramienta rota aunque el gate funcione.** `.github/actions/evolith-validate/action.yml` lee `jq -r '.summary.violations // 0'`, pero el envelope real del CLI tiene claves de primer nivel `[success, data, meta]` con `data = {status, rulesChecked, issues, coreRef, timestamp}` — no existe `.summary`. Lo mismo aplica al fichero `--output` (`validate.command.ts:353-361` escribe el mismo `createSuccessEnvelope`). **Matiz importante: la action sí bloquea correctamente** — captura `EXIT_CODE`, pone `compliance-status=non-compliant` y sale 1 cuando `fail-on-violation=true`; lo roto es el contador y el texto del resumen del PR, que renderiza literalmente "Non-compliant -- 0 violation(s) found". Y `grep -rn 'evolith-validate' .github/` solo coincide dentro del propio README de la action: cero consumidores en los 12 workflows, así que no hay regresión posible. Fix: cambiar el path jq a `.data.issues \| map(select(.blocking)) \| length` y añadir un workflow en este repositorio que ejecute la action contra un satélite fixture no conforme, de modo que quede dogfooded y con test de regresión. **EN-PROGRESO (`44fe8dd3`):** el path `jq` ahora cuenta issues blocking del envelope ADR-0073 real en vez de leer una clave `.summary` que nunca existió, y `.github/workflows/evolith-validate-dogfood.yml` ejecuta la action contra un satélite fixture deliberadamente no conforme, de modo que la action queda por fin dogfooded. Validado por parseo YAML y ejecutando los scripts de sus steps localmente contra el CLI construido. **NO cerrado:** el workflow nunca se ha ejecutado en un runner de GitHub, así que por el estándar de esta propia auditoría el gate aún no está probado — un gate que nadie ha visto fallar es el defecto del que trata toda esta ola. Cerrarlo tras su primera corrida verde. Aparte, no está en el conjunto de checks requeridos, que es la acción de dueño de `GT-574`. **COMPLETADO (verificado contra la corrida de CI 30228607519, `develop`, 2026-07-27T00:56:37Z):** ambos jobs verdes — `Action contract (recorded envelopes)` y `Action vs non-conforming satellite`. El cierre NO es "el workflow está verde": el log lleva las dos aserciones que antes eran imposibles, `OK: 34 blocking violation(s) of 274 issue(s), agreeing with the report.` y `OK: the gate blocked as expected with 34 blocking violation(s).` Es decir, el contador es distinto de cero, concuerda con el fichero de reporte que escribe la action, `compliance-status` es `non-compliant`, y la mitad negativa — `fail-on-violation: true` DEBE hacer fallar el step — se ejercita en cada corrida. Cinco corridas verdes consecutivas. La action queda dogfooded y bajo regresión, que era la mitad más importante de este gap. Pendiente y registrado en otro sitio: el workflow no está en el conjunto de checks requeridos, que es configuración de branch protection. | | | `Infra` | Cross | P2 | XS | `COMPLETADO` | | [`GT-578`](./gap-reference-catalog.es.md#gt-578) | **Las dos causas raíz sistémicas detrás de la mayoría de hallazgos de la auditoría 2026-07-26, atacadas en el mecanismo y no instancia por instancia.** (a) *Literales de ruta*: la migración a `src/` movió código e imports pero no los cientos de cadenas de ruta en scripts de CI, pasos `run:` de workflows, constantes de evaluadores y values de Helm — el compilador detecta un módulo movido, nada detecta un fichero movido referenciado por una cadena, y una ruta que no resuelve produce silencio. Instancias vivas: `OpaEvaluator` hardcodeando `/rulesets/opa/policy.wasm` (el motor OPA entero es no funcional contra el layout del Core); `upgrade` diffeando contra un `/rulesets` inexistente; el config de fronteras del CLI guardando `src/domain`, `src/application`, `src/core`, ninguno de los cuales ha existido nunca; `sdk-cli-ci.yml:467` invocando un script inexistente cuyo homólogo real apunta a una ruta pre-refactor, así que "Winston Agentic Review" está muerto dos veces y reporta `success`. (b) *Pase vacuo*: el patrón ya existe en `34-boundary-guard-repository.mjs:57-73` ("A zero-file scan must never be reported as boundary guard passed") y como auto-test negativo en el gate de contrato del Tracker — está aplicado a 2 guards de ~46. Fix: un guard de ~40 líneas que resuelva contra disco todo literal de ruta; que cada guard publique su denominador y salga 1 ante un escaneo de cero elementos; que cada guard lleve una fixture deliberadamente mala que DEBE ponerlo rojo en CI; y que CI ejecute todos los `validationCommand` de `gap-closure-evidence.json`. **COMPLETADO (`44fe8dd3`):** `40-validate-path-literals.mjs` escanea scripts del harness, pasos `run:` de workflows y values de Helm, resuelve cada literal contra disco, publica su denominador (109 literales en 197 ficheros) y sale 1 ante un escaneo vacuo — obedece la disciplina que impone, con una fixture negativa en `node --test` (15 tests). Los dos defectos vivos que encontró están reparados: `sdk-cli-ci.yml:467` invocaba un `.harness/scripts/ci/13-agentic-code-review.mjs` inexistente (la ruta real está bajo `ci/agentic/`) y el argv de ese script apuntaba a un `packages/mcp-server/dist/main.js` pre-refactor — que es la razón por la que "Winston Agentic Review" reportaba `success` durante meses sin llegar nunca al servidor. Con ambos arreglados, el guard pasó de `--report` a su modo ESTRICTO por defecto en `ci-runner.mjs`, así que un literal muerto nuevo ahora FALLA la corrida de gobernanza. La mitad anti-vacua queda impuesta para este guard; extender el patrón a los otros ~45 guards queda como trabajo posterior. **NO cerrado:** dos de sus tres criterios siguen sin cumplirse — el patrón anti-pase-vacuo está impuesto solo para ESTE guard (faltan ~45), y los `validationCommands` del tablero todavía no se ejecutan en CI. El guard de literales de ruta sí está estricto, cableado y verde. **+2026-07-26 (ola 3) — aterrizaron dos guards nuevos; los criterios 2 y 3 avanzaron, ninguno cerró.** `42-validate-guard-denominators.mjs` clasifica los **54** guards de CI y asserta que ninguno puede pasar en silencio un escaneo de cero elementos (8/8 tests, exit 0). `41-validate-evidence-commands.mjs` por fin EJECUTA los `validationCommands` del tablero en vez de sólo comprobar que sean cadenas no vacías (27/27 tests), y reporta su denominador con honestidad: sobre **937 comandos registrados en 544 registros de cierre encuentra 290 referencias muertas**, e imprime `THIS IS NOT A PASS` en vez de salir verde con una cifra así. Está cableado en modo reporte con un trinquete `--max-dead 290` para que el conteo sólo pueda bajar. **Sigue abierto:** el criterio 3 exige muertas == 0, que es una migración de datos del fichero de evidencia (propiedad del tablero), no un cambio de guard; el criterio 2 exige una fixture negativa por guard, y clasificar no sustituye a eso. Las 290 referencias muertas son en sí un hallazgo: son el tamaño medido de la causa raíz C5 — un tablero que registra intención en vez de resultado. Ola 2: el criterio 2 pasa a exigirse por EJECUCIÓN — `43-validate-guard-negative-fixtures.mjs` apunta cada guard de barrido a un único árbol deliberadamente vacío y 32/32 se pusieron en rojo, 0 reportaron pase. Una fixture compartida en vez de 34 a medida, derivada de la clasificación de 42, así que el siguiente guard se ejercita el día que aterriza. Dos clases de falso-MUERTO en 41 reparadas y el ratchet vuelto independiente de la máquina (ahora imprime su propia base: 303, coincidente entre un portátil y el runner pese a diferir 15 en el conteo bruto). 43, 44 y 34 cableados en ci-cd.yml; ratchet bajado de 305 a 303. **AC3 intacto:** 288 referencias muertas, 198 nombrando workspaces previos al refactor — una migración de datos del fichero de evidencia. | | | `Governance` | Cross | P1 | M | `EN-PROGRESO` | -| [`GT-580`](./gap-reference-catalog.es.md#gt-580) | `grep -rno "process.exit([0-9]*)" src/sdk/cli/src` devuelve **20 × `process.exit(1)` más 2 `process.exit()` sin argumento** — un único valor de fallo para un veredicto FAIL, un flag mal escrito, un fichero ausente o una caída de infraestructura, así que ningún consumidor puede ramificar por causa. Los diagnósticos comparten el canal máquina: **341 `console.log` frente a 115 `console.error`**. Y no hay salida incremental — no existe `--format ndjson` — así que un `validate` largo es opaco hasta que termina. Un harness de agente, un hook de pre-commit y un paso de CI tienen exactamente un primitivo en común (el código de salida del proceso) y el CLI hoy renuncia a usarlo. Fix: taxonomía publicada (`0` PASS · `2` error de uso · `3` **veredicto FAIL** · `1` fallo de infraestructura · `4` HITL requerido), todo diagnóstico a stderr, un stream de eventos NDJSON versionado, y la taxonomía gobernada por un ruleset propio con paridad Rego para que un comando nuevo no pueda romperla. **+ola 2026-07-28 — la taxonomía existe y su rotura de contrato está propagada.** `0` pasa · `1` fallo de herramienta · `2` bloqueado · `3` entrada inválida, impuesto centralmente en `BaseEvolithCommand.handleError` más un guard jest que rompe la build ante cualquier literal de salida fuera del conjunto. **La propagación se auditó en vez de suponerse:** un barrido de todo el repo no encontró NADA que interpretara `1` como "gate falló", así que no había rotura viva aquí — pero la composite action colapsaba todo lo no-cero en `non-compliant`, lo que **afirmaba que el repositorio de un cliente era no conforme cuando el CLI simplemente había reventado o se le había invocado mal**. Eso es el inverso de un gate que reporta verde sobre un repo sin evaluar, y es igual de falso. La action ahora reporta `error` / `invalid-input`, expone `exit-code`, falla el paso ante un no-veredicto ignorando `fail-on-violation`, y su resumen de PR dice "Not evaluated — esto no es un hallazgo sobre este repositorio". Sus tests pasaron de 11 a 13, reclasificados y no aflojados: cuatro casos eran veredictos bloqueados (ahora exit 2) y dos eran fallos de herramienta genuinos (siguen en exit 1, ahora `error`). Ambas guías publican la tabla de cuatro valores. **NO cerrado:** la disciplina stdout/stderr está intacta (341 `console.log` frente a 115 `console.error`), no hay flujo de eventos `--format ndjson`, y el tercer criterio pide un ruleset de paridad Rego mientras lo que aterrizó es un guard jest que sólo cubre este paquete. **Follow-up fuera de este repositorio:** el Tracker invoca el CLI y puede ramificar por su código de salida — no verificable desde aquí, hay que revisarlo en `evolith_tracker`. **Criterio 1 marcado el 2026-07-28 tras verificarlo:** `exit-code-taxonomy.spec.ts` publica exactamente cuatro códigos, separa un veredicto bloqueante de un Core inalcanzable y — la parte que sostiene el criterio — BARRE las fuentes de los comandos y comprueba que ninguna nombra un exit code fuera de `0`, `1`, `2` o `3`, con protección anti-vacua para que un verde sobre un barrido vacío no cuente como verde. 21/21. Quedan los criterios 2 y 3: no hay test que afirme que `--format json`/`ndjson` escribe datos solo en stdout con todo diagnóstico en stderr, ni un par ruleset+Rego que exija la taxonomía como gobierno en vez de como barrido de jest. | | | `Evolith CLI` | Cross | P1 | M | `EN-PROGRESO` | +| [`GT-580`](./gap-reference-catalog.es.md#gt-580) | `grep -rno "process.exit([0-9]*)" src/sdk/cli/src` devuelve **20 × `process.exit(1)` más 2 `process.exit()` sin argumento** — un único valor de fallo para un veredicto FAIL, un flag mal escrito, un fichero ausente o una caída de infraestructura, así que ningún consumidor puede ramificar por causa. Los diagnósticos comparten el canal máquina: **341 `console.log` frente a 115 `console.error`**. Y no hay salida incremental — no existe `--format ndjson` — así que un `validate` largo es opaco hasta que termina. Un harness de agente, un hook de pre-commit y un paso de CI tienen exactamente un primitivo en común (el código de salida del proceso) y el CLI hoy renuncia a usarlo. Fix: taxonomía publicada (`0` PASS · `2` error de uso · `3` **veredicto FAIL** · `1` fallo de infraestructura · `4` HITL requerido), todo diagnóstico a stderr, un stream de eventos NDJSON versionado, y la taxonomía gobernada por un ruleset propio con paridad Rego para que un comando nuevo no pueda romperla. **+ola 2026-07-28 — la taxonomía existe y su rotura de contrato está propagada.** `0` pasa · `1` fallo de herramienta · `2` bloqueado · `3` entrada inválida, impuesto centralmente en `BaseEvolithCommand.handleError` más un guard jest que rompe la build ante cualquier literal de salida fuera del conjunto. **La propagación se auditó en vez de suponerse:** un barrido de todo el repo no encontró NADA que interpretara `1` como "gate falló", así que no había rotura viva aquí — pero la composite action colapsaba todo lo no-cero en `non-compliant`, lo que **afirmaba que el repositorio de un cliente era no conforme cuando el CLI simplemente había reventado o se le había invocado mal**. Eso es el inverso de un gate que reporta verde sobre un repo sin evaluar, y es igual de falso. La action ahora reporta `error` / `invalid-input`, expone `exit-code`, falla el paso ante un no-veredicto ignorando `fail-on-violation`, y su resumen de PR dice "Not evaluated — esto no es un hallazgo sobre este repositorio". Sus tests pasaron de 11 a 13, reclasificados y no aflojados: cuatro casos eran veredictos bloqueados (ahora exit 2) y dos eran fallos de herramienta genuinos (siguen en exit 1, ahora `error`). Ambas guías publican la tabla de cuatro valores. **NO cerrado:** la disciplina stdout/stderr está intacta (341 `console.log` frente a 115 `console.error`), no hay flujo de eventos `--format ndjson`, y el tercer criterio pide un ruleset de paridad Rego mientras lo que aterrizó es un guard jest que sólo cubre este paquete. **Follow-up fuera de este repositorio:** el Tracker invoca el CLI y puede ramificar por su código de salida — no verificable desde aquí, hay que revisarlo en `evolith_tracker`. **Criterio 1 marcado el 2026-07-28 tras verificarlo:** `exit-code-taxonomy.spec.ts` publica exactamente cuatro códigos, separa un veredicto bloqueante de un Core inalcanzable y — la parte que sostiene el criterio — BARRE las fuentes de los comandos y comprueba que ninguna nombra un exit code fuera de `0`, `1`, `2` o `3`, con protección anti-vacua para que un verde sobre un barrido vacío no cuente como verde. 21/21. Quedan los criterios 2 y 3: no hay test que afirme que `--format json`/`ndjson` escribe datos solo en stdout con todo diagnóstico en stderr, ni un par ruleset+Rego que exija la taxonomía como gobierno en vez de como barrido de jest. **+ola 2026-07-29 — criterios 2 y 3 cerrados para el canal JSON; un barrido reforzado encontró cuatro roturas más, miembros de la taxonomía pero incorrectas.** Criterio 2: `machine-channel.registry-sweep.spec.ts` le pregunta al propio registro de commander del CLI qué comandos declaran `--format` (no una lista a mano) y comprueba que los 32 encontrados escriben exactamente un documento JSON parseable en stdout con los diagnósticos en stderr — 32/32 limpios. Esto cierra la mitad de pureza stdout/stderr del criterio 2; el streaming `--format ndjson`, el cuarto pilar del fix original, sigue sin construirse. Criterio 3: la taxonomía ahora está gobernada por `src/rulesets/cli/exit-code-taxonomy.rules.json` con paridad Rego (`src/rulesets/opa/cli-exit-code-taxonomy.rego`, `opa test` 7/7) y un evaluador nativo (`cli-exit-taxonomy-rule.handler.ts`, 28/28 junto con `rule-corpus-triage.spec.ts`) — un comando nuevo ya no puede romper la taxonomía sin que falle un ruleset. **Reforzar el barrido sacó a la luz justo la clase de bug para la que existe:** al sumar `envelope.success === (status === 0)` junto a la pertenencia a la taxonomía aparecieron CUATRO comandos que eran miembros válidos de la taxonomía mientras contradecían su propio envelope — `sdlc gate-status` salía con 0 sobre un envelope `INTERNAL_ERROR` (un consumidor que ramifica por código lo lee como PASS), `alias` salía con 1 en vez de 3 ante un error de uso, el catch-all de `init-wizard` colapsaba un `NonInteractiveError` a `INTERNAL_ERROR`/1 antes de que llegara a `resolveExitCode`, y la rama JSON de `sdlc generate` hacía `return` antes de llegar a la asignación de `process.exitCode` que tenía debajo. Los cuatro se corrigieron enrutando por los helpers ya existentes `exitCodeForErrorCode`/`resolveExitCode`/`carriesCliExitCode` en vez de un literal fijo, y el barrido ahora exige esa coherencia para todo comando de aquí en más. Verificado: suite del CLI 101 suites/1446 tests, handler+triage de core-domain 28/28, `opa test` 7/7, `tsc --noEmit` limpio. [PR #278](https://github.com/beyondnetcode/evolith_arch32/pull/278). **Sigue abierto:** el flujo de eventos `--format ndjson`, y el seguimiento cross-repo del Tracker señalado arriba (no verificable desde este repo). | | | `Evolith CLI` | Cross | P1 | M | `EN-PROGRESO` | | [`GT-581`](./gap-reference-catalog.es.md#gt-581) | `grep -rn "outputSchema\\|structuredContent" src/packages/mcp-server/src` devuelve **0**, e igual `annotations` — sobre **50 tools anunciadas**, con el SDK `1.29.0`, que soporta las tres cosas. Todo llamante recibe por tanto un bloque de texto y debe reconstruir su forma por ingeniería inversa, y ningún cliente puede distinguir una tool de solo lectura (`evolith-adr-list`) de una destructiva (`evolith-satellite-create`) antes de invocarla. El draft de la especificación además relajó `inputSchema`/`outputSchema` para aceptar cualquier keyword de JSON Schema 2020-12 (SEP-2106) y pide a los servidores devolver `tools/list` en orden determinista para mejorar el acierto de caché de cliente y de prompt — dos ganancias gratis en la misma pasada. Fix: derivar `outputSchema` por tool desde `@beyondnet/evolith-contracts`, emitir `structuredContent`, añadir annotations `readOnlyHint`/`destructiveHint`/`idempotentHint`, y hacer determinista el orden de `tools/list`. Ola 2: las 50 herramientas declaran `outputSchema` y devuelven `structuredContent` que valida con el propio validador del SDK — incluso en las denegaciones FORBIDDEN, que es el resultado que un agente más necesita leer mecánicamente. Se genera de forma central desde la versión del sobre, así que no se puede registrar una herramienta sin contrato. **Salvedad que se arrastra, no se oculta:** la rama `data` queda declarada-pero-abierta en las 50; estrecharla exige esquemas por operación, que es GT-583. | | | `MCP Server` | Cross | P1 | M | `COMPLETADO` | | [`GT-582`](./gap-reference-catalog.es.md#gt-582) | Verificado directamente contra la especificación viva el 2026-07-26, no tomado de una fuente secundaria. La revisión **actual** del protocolo es `2025-11-25`; el **draft** elimina las sesiones a nivel de protocolo y la cabecera `Mcp-Session-Id` (SEP-2567), elimina el handshake `initialize`/`notifications/initialized` en favor de `_meta` por petición (SEP-2575), hace obligatorio `server/discover` (SEP-2575), sustituye las peticiones iniciadas por el servidor con el patrón **MRTR** — `InputRequiredResult`, un `resultType` obligatorio, e `inputResponses` en un reintento de la petición original (SEP-2322) —, deprecia Roots/Sampling/Logging (SEP-2577) y deprecia el Dynamic Client Registration en favor de Client ID Metadata Documents. En este repositorio: SDK `1.29.0`, **7 sitios con `sessionId`** bajo `src/packages/mcp-server/src`, y `grep -rn "well-known\\|oauth-protected-resource" src` devuelve 2 coincidencias no relacionadas, así que tampoco se sirve documento de metadatos de recurso protegido. MRTR importa más allá de la conformidad: es *la aprobación como protocolo*, y es la única forma en que el gate HITL sobrevive a la desaparición de las sesiones. **Corrección al documento origen, que esta fila registra en vez de repetir:** el documento etiqueta esta revisión como `2026-07-28` y la plantea como una emergencia a 3 días. Esa fecha no pudo confirmarse — la propia página de versionado de la especificación sigue nombrando `2025-11-25` como actual, describe la negociación como parte de `initialize`, y no publica fecha de release para el draft. El contenido técnico es real y está confirmado; la urgencia no. Esto es trabajo preparatorio contra un draft, a revisar en cada revisión de la especificación. | | | `MCP Server` | Cross | P1 | L | `PENDIENTE` | | [`GT-583`](./gap-reference-catalog.es.md#gt-583) | `TOOL_SCHEMAS` es un mapa escrito a mano en `src/sdk/cli/src/commands/api/api.catalog.ts:81`, las tools MCP declaran sus propios input schemas inline en código, y `buildCapabilityManifest` (GT-513) publica solo `evaluationKinds`, `engines`, `surfaces`, `supportedConsumers` y un `sha256` — **ningún esquema de entrada o salida por operación**. Así que "un registro genera las tres superficies" se afirma en prosa y se mantiene a mano. Aparte, **los 154 ficheros `*.schema.json` declaran `http://json-schema.org/draft-07/schema#`**, mientras el draft de MCP espera keywords de 2020-12 en los esquemas de tool — lo que convierte el pin en un bloqueo para GT-581 y no en una elección neutral. Fix: extender el manifiesto con `inputSchema`/`outputSchema` por operación, generar `TOOL_SCHEMAS` y los registros MCP desde él, y migrar el meta-esquema a 2020-12 compilando con el entry point 2020 de ajv. | | | `Evolith Core` | Cross | P1 | L | `PENDIENTE` | diff --git a/reference/core/control-center/gaps/gap-tracking.md b/reference/core/control-center/gaps/gap-tracking.md index 52c7b9a4c..1126ac57d 100644 --- a/reference/core/control-center/gaps/gap-tracking.md +++ b/reference/core/control-center/gaps/gap-tracking.md @@ -61,7 +61,7 @@ This board is the single source of truth for technical debt, gaps, opportunities | [`GT-576`](./gap-reference-catalog.md#gt-576) | **The self-assessment a buyer reads first is falsifiable in ten minutes, and the document incriminates itself.** `maturity-assessment.md` defines *Validated (Weight 1.0) — Passing all quality gates, tests, and active in CI/CD*, then marks **Pillar 1 Security "Level 4 (Managed) / Validated"** citing multi-tenant Row-Level Security (ADR-0010) and immutable audit trails via CDC (ADR-0016): `grep -rniE 'row.level.security\|current_setting\(\|debezium\|change data capture'` over `src` returns **ZERO files**, and core-api declares no database driver or ORM. **Pillar 4** is marked Level 4 / Validated citing "deterministic monorepo builds via Nx" — no `nx.json` and no `nx` dependency exist. It also claims dual-engine 8/8 parity while the gate covers 3 topologies and the published package ships policies for 5. (Pillar 3's stale `opossum` citation is a different matter — that pillar is honestly marked `Designed`.) Fix: downgrade Pillar 1 to `Designed` with the ADRs listed as intent, delete the Nx citation, report parity against the published artifact, and add a mechanical rule to `09-reconcile-maturity.mjs`: a capability may only be marked `Validated` if its evidence list contains at least one `file:line` reference or CI job, never an ADR alone. **DONE (`44fe8dd3`):** every claim in the statement was re-verified against the tree and all held. Pillar 1 (Security) and Pillar 4 dropped from `Validated` to `Designed` with their ADRs listed as intent, the Nx citation deleted, and parity reported against the published artifact — in BOTH languages, structurally identical. The class of error is now mechanically blocked: `09-reconcile-maturity.mjs` rejects any `Validated` capability whose evidence contains no `file:line` and no CI job, and that rule ships with a negative self-test that proves it goes red on a deliberately bad input. Verified: `09-reconcile-maturity.mjs --check` ✅, `04-check-bilingual-parity.mjs` ✅, governance 17/17. | | | `Governance` | Cross | P1 | S | `DONE` | | [`GT-577`](./gap-reference-catalog.md#gt-577) | **The artifact a customer would wire into their CI reads as a broken tool even though the gate works.** `.github/actions/evolith-validate/action.yml` reads `jq -r '.summary.violations // 0'`, but the real CLI envelope has top-level keys `[success, data, meta]` with `data = {status, rulesChecked, issues, coreRef, timestamp}` — there is no `.summary`. The same applies to the `--output` file (`validate.command.ts:353-361` writes the same `createSuccessEnvelope`). **Important nuance: the action does block correctly** — it captures `EXIT_CODE`, sets `compliance-status=non-compliant` and exits 1 when `fail-on-violation=true`; what is broken is the counter and the PR summary text, which renders literally "Non-compliant -- 0 violation(s) found". And `grep -rn 'evolith-validate' .github/` matches only inside the action's own README: zero consumers across the 12 workflows, so no regression is possible. Fix: change the jq path to `.data.issues \| map(select(.blocking)) \| length` and add a workflow in this repository that runs the action against a non-conforming satellite fixture, so it is dogfooded and regression-tested. **IN-PROGRESS (`44fe8dd3`):** the `jq` path now counts blocking issues from the real ADR-0073 envelope instead of reading a `.summary` key that never existed, and `.github/workflows/evolith-validate-dogfood.yml` runs the action against a deliberately non-conforming satellite fixture so the action is finally dogfooded. Validated by YAML parse and by executing the action's step scripts locally against the built CLI. **NOT closed:** the workflow has never executed on a GitHub runner, so by this audit's own standard the gate is not yet proven — a gate that has never been seen failing is the defect this whole wave is about. Close it after its first green run. Separately, it is not in the required-checks set, which is the `GT-574` owner action. **DONE (verified against CI run 30228607519, `develop`, 2026-07-27T00:56:37Z):** both jobs green — `Action contract (recorded envelopes)` and `Action vs non-conforming satellite`. The closure is NOT "the workflow is green": the log carries the two assertions that were impossible before, `OK: 34 blocking violation(s) of 274 issue(s), agreeing with the report.` and `OK: the gate blocked as expected with 34 blocking violation(s).` So the counter is non-zero, it agrees with the report file the action writes, `compliance-status` is `non-compliant`, and the negative half — `fail-on-violation: true` MUST fail the step — is exercised on every run. Five consecutive green runs. The action is now dogfooded and under regression, which was the more important half of this gap. Remaining and tracked elsewhere: the workflow is not in the required-checks set, which is branch-protection configuration. | | | `Infra` | Cross | P2 | XS | `DONE` | | [`GT-578`](./gap-reference-catalog.md#gt-578) | **The two systemic root causes behind most findings of the 2026-07-26 audit, addressed at the mechanism rather than instance by instance.** (a) *Path literals*: the move to `src/` migrated code and imports but not the hundreds of path strings in CI scripts, workflow `run:` steps, evaluator constants and Helm values — the compiler catches a moved module, nothing catches a moved file referenced by a string, and a path that does not resolve produces silence. Live instances: `OpaEvaluator` hardcoding `/rulesets/opa/policy.wasm` (the whole OPA engine is non-functional against the Core layout); `upgrade` diffing against a nonexistent `/rulesets`; the CLI boundary config guarding `src/domain`, `src/application`, `src/core`, none of which has ever existed; `sdk-cli-ci.yml:467` invoking a nonexistent script whose real counterpart points at a pre-refactor path, so "Winston Agentic Review" is dead twice and reports `success`. (b) *Anti-vacuous pass*: the pattern already exists at `34-boundary-guard-repository.mjs:57-73` ("A zero-file scan must never be reported as boundary guard passed") and as a negative self-test in the Tracker contract gate — it is applied to 2 guards of ~46. Fix: a ~40-line guard resolving every path literal against disk; every guard publishes its denominator and exits 1 on a zero-element scan; every guard ships a deliberately bad fixture that MUST turn it red in CI; and CI executes every `validationCommand` in `gap-closure-evidence.json`. **DONE (`44fe8dd3`):** `40-validate-path-literals.mjs` scans harness scripts, workflow `run:` steps and Helm values, resolves each literal against disk, publishes its denominator (109 literals across 197 files) and exits 1 on a vacuous scan — it obeys the discipline it enforces, with a negative fixture in `node --test` (15 tests). Both live defects it found are repaired: `sdk-cli-ci.yml:467` invoked a nonexistent `.harness/scripts/ci/13-agentic-code-review.mjs` (real path under `ci/agentic/`) and that script's argv targeted a pre-refactor `packages/mcp-server/dist/main.js` — which is why "Winston Agentic Review" reported `success` for months while never reaching the server. With both fixed, the guard was flipped from `--report` to its default STRICT mode in `ci-runner.mjs`, so a new dead literal now FAILS the governance run. The anti-vacuous half is enforced for this guard; extending the pattern to the other ~45 guards remains follow-on work. **NOT closed:** two of its three criteria remain unmet — the anti-vacuous-pass pattern is enforced for THIS guard only (~45 others to go), and the board's `validationCommands` are still not executed in CI. The path-literal guard itself is strict, wired and green. **+2026-07-26 (wave 3) — two new guards landed; criteria 2 and 3 advanced, neither closed.** `42-validate-guard-denominators.mjs` classifies all **54** CI guards and asserts none can pass a zero-element scan silently (8/8 unit tests, exit 0). `41-validate-evidence-commands.mjs` finally EXECUTES the board's `validationCommands` instead of only checking they are non-empty strings (27/27 unit tests), and it reports its denominator honestly: over **937 recorded commands across 544 closure records it finds 290 dead references**, and it prints `THIS IS NOT A PASS` rather than exiting green on a number that large. It is wired in report mode with a `--max-dead 290` ratchet so the count can only go down. **Still open:** criterion 3 needs dead == 0, which is a data migration of the board-owned evidence file, not a guard change; criterion 2 needs a negative fixture per guard, which classification does not substitute for. The 290 dead references are themselves a finding: they are the measured size of root cause C5 — a board that records intent rather than result. Wave 2: criterion 2 is now enforced by EXECUTION — `43-validate-guard-negative-fixtures.mjs` points every scanning guard at one deliberately empty tree and 32/32 turned red, 0 reported a pass. One shared fixture rather than 34 bespoke ones, derived from 42’s classification, so the next guard is exercised the day it lands. Two false-DEAD classes in 41 repaired and the ratchet made machine-independent (it now prints its own basis: 303, agreed between a laptop and the runner despite raw counts differing by 15). 43, 44 and 34 wired into ci-cd.yml, ratchet lowered 305→303. **AC3 untouched:** 288 dead references, 198 naming pre-refactor workspaces — a data migration of the evidence file. | | | `Governance` | Cross | P1 | M | `IN-PROGRESS` | -| [`GT-580`](./gap-reference-catalog.md#gt-580) | `grep -rno "process.exit([0-9]*)" src/sdk/cli/src` returns **20 × `process.exit(1)` plus 2 bare `process.exit()`** — one single failure value for a FAIL verdict, a bad flag, a missing file and an infra crash alike, so no consumer can branch on the cause. Diagnostics share the machine channel: **341 `console.log` against 115 `console.error`**. And there is no incremental output — no `--format ndjson` — so a long `validate` is opaque until it terminates. An agent harness, a pre-commit hook and a CI step all have exactly one primitive in common (the process exit code) and the CLI currently declines to use it. Fix: a published taxonomy (`0` PASS · `2` usage error · `3` **verdict FAIL** · `1` infra failure · `4` HITL required), every diagnostic to stderr, a versioned NDJSON event stream, and the taxonomy governed by its own ruleset with Rego parity so a new command cannot regress it. **+wave 2026-07-28 — the taxonomy exists and its contract break is propagated.** `0` pass · `1` tool failure · `2` blocked · `3` invalid input, enforced centrally in `BaseEvolithCommand.handleError` plus a jest guard that fails the build on any exit literal outside the set. **The propagation was audited rather than assumed:** a repo-wide sweep found NOTHING branching on `1` as "gate failed", so there was no live break here — but the composite action collapsed every non-zero onto `non-compliant`, which **asserted a customer's repository was non-compliant when the CLI had merely crashed or been invoked wrongly**. That is the inverse of a gate reporting green over an unevaluated repo, and it is just as false. The action now reports `error` / `invalid-input`, exposes `exit-code`, fails the step for a non-verdict regardless of `fail-on-violation`, and its PR summary says "Not evaluated — this is not a finding about this repository". Its tests went 11 → 13, reclassified rather than loosened: four cases were blocked verdicts (now exit 2) and two were genuine tool failures (still exit 1, now `error`). Both language guides publish the four-value table. **NOT closed:** stdout/stderr discipline is untouched (341 `console.log` vs 115 `console.error`), there is no `--format ndjson` event stream, and the third criterion asks for a Rego-parity ruleset while what shipped is a jest guard covering only this package. **Follow-up outside this repository:** the Tracker invokes the CLI and may branch on its exit code — unverifiable from here and must be checked in `evolith_tracker`. **Criterion 1 ticked on 2026-07-28 after verification:** `exit-code-taxonomy.spec.ts` publishes exactly four codes, separates a blocked verdict from an unreachable Core, and — the load-bearing part — SCANS the command sources and asserts none names an exit code outside `0`, `1`, `2` or `3`, with an anti-vacuous guard so a green from an empty scan is not a green. 21/21. Criteria 2 and 3 remain: there is no test asserting that `--format json`/`ndjson` writes data only to stdout with every diagnostic on stderr, and no ruleset+Rego pair enforcing the taxonomy as governance rather than as a jest scan. | | | `Evolith CLI` | Cross | P1 | M | `IN-PROGRESS` | +| [`GT-580`](./gap-reference-catalog.md#gt-580) | `grep -rno "process.exit([0-9]*)" src/sdk/cli/src` returns **20 × `process.exit(1)` plus 2 bare `process.exit()`** — one single failure value for a FAIL verdict, a bad flag, a missing file and an infra crash alike, so no consumer can branch on the cause. Diagnostics share the machine channel: **341 `console.log` against 115 `console.error`**. And there is no incremental output — no `--format ndjson` — so a long `validate` is opaque until it terminates. An agent harness, a pre-commit hook and a CI step all have exactly one primitive in common (the process exit code) and the CLI currently declines to use it. Fix: a published taxonomy (`0` PASS · `2` usage error · `3` **verdict FAIL** · `1` infra failure · `4` HITL required), every diagnostic to stderr, a versioned NDJSON event stream, and the taxonomy governed by its own ruleset with Rego parity so a new command cannot regress it. **+wave 2026-07-28 — the taxonomy exists and its contract break is propagated.** `0` pass · `1` tool failure · `2` blocked · `3` invalid input, enforced centrally in `BaseEvolithCommand.handleError` plus a jest guard that fails the build on any exit literal outside the set. **The propagation was audited rather than assumed:** a repo-wide sweep found NOTHING branching on `1` as "gate failed", so there was no live break here — but the composite action collapsed every non-zero onto `non-compliant`, which **asserted a customer's repository was non-compliant when the CLI had merely crashed or been invoked wrongly**. That is the inverse of a gate reporting green over an unevaluated repo, and it is just as false. The action now reports `error` / `invalid-input`, exposes `exit-code`, fails the step for a non-verdict regardless of `fail-on-violation`, and its PR summary says "Not evaluated — this is not a finding about this repository". Its tests went 11 → 13, reclassified rather than loosened: four cases were blocked verdicts (now exit 2) and two were genuine tool failures (still exit 1, now `error`). Both language guides publish the four-value table. **NOT closed:** stdout/stderr discipline is untouched (341 `console.log` vs 115 `console.error`), there is no `--format ndjson` event stream, and the third criterion asks for a Rego-parity ruleset while what shipped is a jest guard covering only this package. **Follow-up outside this repository:** the Tracker invokes the CLI and may branch on its exit code — unverifiable from here and must be checked in `evolith_tracker`. **Criterion 1 ticked on 2026-07-28 after verification:** `exit-code-taxonomy.spec.ts` publishes exactly four codes, separates a blocked verdict from an unreachable Core, and — the load-bearing part — SCANS the command sources and asserts none names an exit code outside `0`, `1`, `2` or `3`, with an anti-vacuous guard so a green from an empty scan is not a green. 21/21. Criteria 2 and 3 remain: there is no test asserting that `--format json`/`ndjson` writes data only to stdout with every diagnostic on stderr, and no ruleset+Rego pair enforcing the taxonomy as governance rather than as a jest scan. **+wave 2026-07-29 — criteria 2 and 3 closed for the JSON channel; a strengthened sweep then found four more taxonomy-membership-but-wrong breaks.** Criterion 2: `machine-channel.registry-sweep.spec.ts` asks the CLI's own commander registry which of its commands declare `--format` (not a hand-picked list) and asserts every one of the 32 found writes exactly one parseable JSON document to stdout with diagnostics on stderr — 32/32 clean. This closes the stdout/stderr purity half of criterion 2; `--format ndjson` streaming, the fourth pillar of the original fix, is still unbuilt. Criterion 3: the taxonomy is now governed by `src/rulesets/cli/exit-code-taxonomy.rules.json` with Rego parity (`src/rulesets/opa/cli-exit-code-taxonomy.rego`, `opa test` 7/7) and a native evaluator (`cli-exit-taxonomy-rule.handler.ts`, 28/28 with `rule-corpus-triage.spec.ts`) — a new command can no longer regress the taxonomy without a ruleset failure. **Strengthening the sweep surfaced exactly the class of bug it exists to catch:** adding `envelope.success === (status === 0)` alongside taxonomy-membership found FOUR commands that were valid taxonomy members while contradicting their own envelope — `sdlc gate-status` exited 0 on an `INTERNAL_ERROR` envelope (a branching consumer reads that as a PASS), `alias` exited 1 instead of 3 on a usage error, `init-wizard`'s own catch-all collapsed a `NonInteractiveError` to `INTERNAL_ERROR`/1 before it ever reached `resolveExitCode`, and `sdlc generate`'s JSON branch `return`ed before reaching the `process.exitCode` assignment below it. All four fixed by routing through the existing `exitCodeForErrorCode`/`resolveExitCode`/`carriesCliExitCode` helpers rather than a hardcoded literal, and the sweep now asserts this agreement for every command going forward. Verified: CLI suite 101 suites/1446 tests, core-domain handler+triage 28/28, `opa test` 7/7, `tsc --noEmit` clean. [PR #278](https://github.com/beyondnetcode/evolith_arch32/pull/278). **Still open:** the `--format ndjson` event stream, and the Tracker cross-repo follow-up noted above (unverifiable from this repo). | | | `Evolith CLI` | Cross | P1 | M | `IN-PROGRESS` | | [`GT-581`](./gap-reference-catalog.md#gt-581) | `grep -rn "outputSchema\\|structuredContent" src/packages/mcp-server/src` returns **0**, and so does `annotations` — across **50 announced tools**, on SDK `1.29.0`, which supports all three. Every caller therefore receives a text block and must reverse-engineer its shape, and no client can tell a read-only tool (`evolith-adr-list`) from a destructive one (`evolith-satellite-create`) before invoking it. The specification draft additionally loosened `inputSchema`/`outputSchema` to accept any JSON Schema 2020-12 keyword (SEP-2106) and asks servers to return `tools/list` in a deterministic order to improve client and prompt-cache hit rates — both free wins on the same pass. Fix: derive `outputSchema` per tool from `@beyondnet/evolith-contracts`, emit `structuredContent`, add `readOnlyHint`/`destructiveHint`/`idempotentHint` annotations, and make `tools/list` ordering deterministic. Wave 2: all 50 tools declare an outputSchema and return structuredContent that validates under the SDK’s own validator — including on FORBIDDEN denials, the result an agent most needs to read mechanically. Generated centrally from the envelope version, so a new tool cannot be registered without a contract. **Caveat carried, not hidden:** the `data` branch is declared-but-open for all 50; narrowing it needs per-operation schemas, which is GT-583. | | | `MCP Server` | Cross | P1 | M | `DONE` | | [`GT-582`](./gap-reference-catalog.md#gt-582) | Verified directly against the live specification on 2026-07-26, not taken from a secondary source. The **current** protocol revision is `2025-11-25`; the **draft** removes protocol-level sessions and the `Mcp-Session-Id` header (SEP-2567), removes the `initialize`/`notifications/initialized` handshake in favour of per-request `_meta` (SEP-2575), makes `server/discover` mandatory (SEP-2575), replaces server-initiated requests with the **MRTR** pattern — `InputRequiredResult`, a required `resultType`, and `inputResponses` on a retry of the original request (SEP-2322) — deprecates Roots/Sampling/Logging (SEP-2577) and deprecates Dynamic Client Registration in favour of Client ID Metadata Documents. In this repository: SDK `1.29.0`, **7 `sessionId` sites** under `src/packages/mcp-server/src`, and `grep -rn "well-known\\|oauth-protected-resource" src` returns 2 unrelated matches, so no protected-resource metadata document is served either. MRTR matters beyond conformance: it is *approval as a protocol*, and it is the only way the HITL gate survives the removal of sessions. **Correction to the source document, which this row records rather than repeats:** the document labels this revision `2026-07-28` and frames it as a 3-day emergency. That date could not be confirmed — the specification's own versioning page still names `2025-11-25` as current, describes negotiation as happening during `initialize`, and publishes no release date for the draft. The technical content is real and confirmed; the urgency is not. This is tracked preparatory work against a draft, to be re-checked at each spec revision. | | | `MCP Server` | Cross | P1 | L | `PENDING` | | [`GT-583`](./gap-reference-catalog.md#gt-583) | `TOOL_SCHEMAS` is a hand-written map at `src/sdk/cli/src/commands/api/api.catalog.ts:81`, the MCP tools declare their own input schemas inline in code, and `buildCapabilityManifest` (GT-513) publishes only `evaluationKinds`, `engines`, `surfaces`, `supportedConsumers` and a `sha256` — **no per-operation input or output schema at all**. So "one registry generates the three surfaces" is asserted in prose and maintained by hand. Separately, **all 154 `*.schema.json` files declare `http://json-schema.org/draft-07/schema#`**, while the MCP draft expects 2020-12 keywords in tool schemas — which makes the pin a blocker for GT-581 rather than a neutral choice. Fix: extend the manifest with per-operation `inputSchema`/`outputSchema`, generate `TOOL_SCHEMAS` and the MCP registrations from it, and migrate the meta-schema to 2020-12 compiling under ajv's 2020 entry point. | | | `Evolith Core` | Cross | P1 | L | `PENDING` | From 7f3fd0933480927ab778480d5fca274e2bdfde6e Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 15:55:41 -0500 Subject: [PATCH 04/27] chore(maturity): reconcile after GT-580's new ruleset 09-reconcile-maturity.mjs --check failed CI's Validate documentation job: rulesetCount was stale at 174, one behind the exit-code-taxonomy.rules.json that PR #278 added. Regenerated via 09-reconcile-maturity.mjs. --- .../maturity-reports/maturity-reconciliation.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/core/control-center/maturity-reports/maturity-reconciliation.json b/reference/core/control-center/maturity-reports/maturity-reconciliation.json index 722d157da..2f7efefbc 100644 --- a/reference/core/control-center/maturity-reports/maturity-reconciliation.json +++ b/reference/core/control-center/maturity-reports/maturity-reconciliation.json @@ -13,7 +13,7 @@ "closureRecords": 574, "cliPackage": "@beyondnet/evolith-cli@1.2.1", "adrCount": 139, - "rulesetCount": 174, + "rulesetCount": 175, "schemaCount": 45 }, "readiness": [ From e21e162924373e900533c7f1c54bd8205c01ec45 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 16:09:19 -0500 Subject: [PATCH 05/27] fix(cli): correct the sweep's own invariant, and a fifth taxonomy mismatch it then found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI's required Test job turned up two things the strengthened sweep assertion missed or got wrong: 1. The assertion itself was too strict. `validate`/`evaluate` legitimately pair `createSuccessEnvelope` with exit 2 (BLOCKED) — the tool succeeded and produced a verdict, and the verdict happens to block. success:true does not imply exit 0; it only rules out TOOL_FAILURE/INVALID_INPUT, which success itself denies happened. Rewrote the assertion: success:true allows OK or BLOCKED, success:false must never be OK and, where the envelope names an ADR-0073 error.code, must match exitCodeForErrorCode's mapping for that code exactly — the same function BaseEvolithCommand.handleError already uses, so the sweep checks commands against the taxonomy's own source of truth instead of a second copy of it. 2. With that corrected, the sweep found a fifth real mismatch: `topology phase-artifacts` with no --phase emits an INVALID_PHASE envelope but hardcoded exit 1 (tool failure) instead of the 3 the taxonomy assigns that code. Fixed the same way as the other four: exitCodeForErrorCode instead of a literal. Updated the command's own unit spec, which had asserted the wrong exit code. Verified: full CLI suite 101 suites/1446 tests, tsc --noEmit clean. --- .../topology/phase-artifacts.command.spec.ts | 5 ++- .../topology/phase-artifacts.command.ts | 3 +- .../machine-channel.registry-sweep.spec.ts | 43 +++++++++++++------ 3 files changed, 36 insertions(+), 15 deletions(-) diff --git a/src/sdk/cli/src/commands/topology/phase-artifacts.command.spec.ts b/src/sdk/cli/src/commands/topology/phase-artifacts.command.spec.ts index 9e16b6e2d..958ea5699 100644 --- a/src/sdk/cli/src/commands/topology/phase-artifacts.command.spec.ts +++ b/src/sdk/cli/src/commands/topology/phase-artifacts.command.spec.ts @@ -201,14 +201,15 @@ describe('PhaseArtifactsCommand', () => { }, ); - it('reports an invalid phase as an INVALID_PHASE envelope with exit code 1 in JSON mode', async () => { + it('reports an invalid phase as an INVALID_PHASE envelope with exit code 3 in JSON mode', async () => { // JSON mode must not throw — but it must not look successful either. await command.executeCommand([], { phase: 'nope' as never, core: corpusRoot, format: 'json' }); const envelope = JSON.parse(stdout()); expect(envelope.error.code).toBe('INVALID_PHASE'); expect(envelope.success).not.toBe(true); - expect(process.exitCode).toBe(1); + // GT-580: INVALID_PHASE is an invalid-input error, not a tool failure. + expect(process.exitCode).toBe(3); }); it.each(['construction', 'quality', 'deployment'] as const)('accepts the downstream phase %s', async (phase) => { diff --git a/src/sdk/cli/src/commands/topology/phase-artifacts.command.ts b/src/sdk/cli/src/commands/topology/phase-artifacts.command.ts index 87d3af368..b9cb1c74e 100644 --- a/src/sdk/cli/src/commands/topology/phase-artifacts.command.ts +++ b/src/sdk/cli/src/commands/topology/phase-artifacts.command.ts @@ -17,6 +17,7 @@ import { } from '@beyondnet/evolith-core-domain/domain/gate-evidence'; import { BaseEvolithCommand } from '../../infrastructure/cli/base-command'; import { resolveRulesets } from '../../infrastructure/paths/rulesets-resolver'; +import { exitCodeForErrorCode } from '../../infrastructure/cli/exit-codes'; const DOWNSTREAM_PHASES: readonly DownstreamPhase[] = ['construction', 'quality', 'deployment']; @@ -72,7 +73,7 @@ export class PhaseArtifactsCommand extends BaseEvolithCommand { if (!phase || !DOWNSTREAM_PHASES.includes(phase)) { const message = `--phase must be one of: ${DOWNSTREAM_PHASES.join(', ')}`; if (json) { - process.exitCode = 1; + process.exitCode = exitCodeForErrorCode('INVALID_PHASE'); console.log(JSON.stringify(createErrorEnvelope('INVALID_PHASE', message, { ...meta, durationMs: Date.now() - startedAt }), null, 2)); return; } diff --git a/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts index 216d594e8..50d91d776 100644 --- a/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts +++ b/src/sdk/cli/src/infrastructure/cli/machine-channel.registry-sweep.spec.ts @@ -43,7 +43,7 @@ import * as os from 'node:os'; import * as path from 'node:path'; import { CommandFactory } from 'nest-commander'; import { Commander } from 'nest-commander/src/constants'; -import { CLI_EXIT_CODES, CLI_EXIT_CODE_VALUES } from './exit-codes'; +import { CLI_EXIT_CODES, CLI_EXIT_CODE_VALUES, exitCodeForErrorCode } from './exit-codes'; const CLI_ROOT = path.resolve(__dirname, '..', '..', '..'); const REPO_ROOT = path.resolve(CLI_ROOT, '..', '..', '..'); @@ -235,14 +235,24 @@ describe('GT-580 · every `--format json` command in the registry pipes cleanly' expect(offenders).toEqual([]); }); - it('exits `0` iff the envelope says `success: true` — never a taxonomy member merely', () => { + it('the envelope and the exit code agree, for every one of them', () => { // Membership in the taxonomy is necessary but not sufficient: a command - // that exits 0 with `success: false` (or a non-zero code with `success: - // true`) still hands a consumer branching on the exit code the wrong - // verdict, and the "member of the taxonomy" assertion above cannot see it - // because 0 and 1 and 2 and 3 are all valid members. This asserts the two - // facts the envelope and the exit code each claim actually agree. - const offenders: Array<{ command: string; status: number | null; success: unknown }> = []; + // that exits 0 with `success: false` still hands a consumer branching on + // the exit code the wrong verdict, and the "member of the taxonomy" + // assertion above cannot see it because 0 is a valid member. But + // `success: true` does NOT imply exit 0 — `validate`/`evaluate` legitimately + // pair a successful, well-formed result with exit 2 when the VERDICT + // itself blocks (`validate.command.ts` emits `createSuccessEnvelope` then + // separately `exitWith(CLI_EXIT_CODES.BLOCKED)`): the tool did not fail, + // it did its job and reported a blocking verdict. So the rule is: + // success: true → exit is OK or BLOCKED (never a TOOL_FAILURE/INVALID_INPUT + // that success itself denies happened) + // success: false → exit is never OK, and where the envelope names an + // ADR-0073 `error.code`, the exit must match what + // `exitCodeForErrorCode` — the taxonomy's own mapping, + // also used by `BaseEvolithCommand.handleError` — says + // that code means. + const offenders: Array<{ command: string; status: number | null; success: unknown; errorCode?: unknown }> = []; for (const run of runs) { let envelope: unknown; try { @@ -251,10 +261,19 @@ describe('GT-580 · every `--format json` command in the registry pipes cleanly' continue; // already reported by the "writes exactly one JSON document" assertion } if (envelope === null || typeof envelope !== 'object' || !('success' in envelope)) continue; - const success = (envelope as { success: unknown }).success; - const exitsOk = run.status === CLI_EXIT_CODES.OK; - if (success !== exitsOk) { - offenders.push({ command: run.command, status: run.status, success }); + const { success } = envelope as { success: unknown }; + const status = run.status; + + if (success === true) { + if (status !== CLI_EXIT_CODES.OK && status !== CLI_EXIT_CODES.BLOCKED) { + offenders.push({ command: run.command, status, success }); + } + } else if (success === false) { + const errorCode = (envelope as { error?: { code?: unknown } }).error?.code; + const expected = typeof errorCode === 'string' ? exitCodeForErrorCode(errorCode) : undefined; + if (status === CLI_EXIT_CODES.OK || (expected !== undefined && status !== expected)) { + offenders.push({ command: run.command, status, success, errorCode }); + } } } From 74fd2f9f52ea41e6178f3c59b8192829cd92863e Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 17:45:35 -0500 Subject: [PATCH 06/27] fix(cli): use real --arch flag in E2E architecture validation test The test passed --architecture and --arch-level, neither of which exist on the validate command (real flags are -a/--arch and -t/--topology). Commander rejected the unknown option before any command code ran, exiting 1 instead of the [0,2] pass/blocked verdict the test expected. --- src/sdk/cli/test/e2e/cli-e2e.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/cli/test/e2e/cli-e2e.test.ts b/src/sdk/cli/test/e2e/cli-e2e.test.ts index cb80b91e8..7e7a5ad54 100644 --- a/src/sdk/cli/test/e2e/cli-e2e.test.ts +++ b/src/sdk/cli/test/e2e/cli-e2e.test.ts @@ -367,7 +367,7 @@ export class UserRepository { describe('validate with architecture analysis', () => { it('should validate architecture with --arch flag', async () => { - const result = await runCli(['validate', '--satellite', testRepoPath, '--architecture', '--arch-level', 'F1']); + const result = await runCli(['validate', '--satellite', testRepoPath, '--arch']); // GT-580: 0 pass, 2 blocked verdict. expect([0, 2]).toContain(result.exitCode); From b0824a6459cdc4ac86f903cacabc386ffe67f903 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 17:50:38 -0500 Subject: [PATCH 07/27] fix(standards): make the evaluability guard read Core, and capture the snapshot it guards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `native-evaluability-snapshot.json` declared in its own header that it was a capture. Nothing captured it — there was no capture script in the repository. It was hand-maintained, and it drifted: it said `documentation-only: 129` while Core pinned 136, and it kept calling rules `unimplemented-native` after they got handlers. The guard was blind to that by construction. `iso-5055-mapping.test.mjs` compared the snapshot's counts against six numbers typed into the test — the same six that are literals inside the snapshot. Expected value and actual value were copies of each other, so it passed for as long as the file went untouched, whatever Core actually pinned. Its comment claimed it read `rule-corpus-triage.spec.ts`; it never opened that file. Drift does not stay contained. `build-iso-5055-mapping.mjs` stamps `nativeEvaluability` onto every row of the mapping FROM the snapshot, so a stale class is laundered into a 391-row artifact and overstates the handler backlog. Three checks replace the six literals, each in the job that can afford it: - `rule-corpus-triage.spec.ts` (core-domain jest, required on main) asserts the committed snapshot against a FRESHLY COMPUTED triage — counts and the class of every rule, in both directions. The real guard: the only place with the dependencies to recompute the truth. - `iso-5055-mapping.test.mjs` runs in the documentation job, which has no node_modules, so it reads Core's counts OUT OF the spec that pins them. Parsing a source file is a real coupling, stated rather than hidden: a missing or reshaped literal THROWS, because failing to find the source of truth must never read as agreement with it. Plus: the snapshot's header must agree with its own body, and every class stamped into the mapping must match the snapshot. - `capture-native-evaluability-snapshot.mjs --check` is the end-to-end re-derivation. The capture script is the missing half. It runs Core's REAL triage through ts-node, resolved via `createRequire` from a package that DECLARES ts-node rather than relying on workspace hoisting — otherwise a required check would be green locally and red on a clean runner. Nothing is reimplemented: a second classifier would be a second source of truth, which is the defect being removed. That required moving the measurement out of the spec, where only jest could reach it, into `test/rule-corpus-triage.ts` — that unreachability is why the snapshot was maintained by hand. It reads the committed file ONCE and keeps both views of that read. `--check` decides whether what is on disk equals a fresh capture, so it must compare the bytes it parsed; the first draft did `existsSync` then two separate reads, which CodeQL correctly flagged, and which could report a diff it had not measured. Order is written into both READMEs and into CI: recapture, THEN rebuild. Rebased onto main, which moved three times while this was open — each time moving the very numbers this branch pins (GT-595 and GT-632 closed twelve rules, GT-580 added three). Re-extracted against main's current spec, which now carries `enforce` in the loader, then recaptured and rebuilt in order: 389 rules triaged, `native-handler` 154, backlog 48; mapping 391 rows. Also: the mapping guard ran in NO workflow. It does now. And `.gitignore` matched `node_modules/` as a directory only, so the symlink a fresh worktree needs was left untracked-but-visible and `git add -A` staged a link to an absolute path. Verified: rule-corpus-triage 23/23; mapping guard 9/9; both `--check` modes current; docs 01 and bilingual 04 green. Negative-tested both failure modes — reverting a count goes red, and building the mapping from a stale snapshot goes red on the stamp test. Co-Authored-By: Claude Opus 5 --- .github/workflows/docs.yml | 17 + .gitignore | 7 +- .../validators/rule-corpus-triage.spec.ts | 199 +++++------ .../core-domain/test/rule-corpus-triage.ts | 163 +++++++++ src/rulesets/standards/README.es.md | 87 +++-- src/rulesets/standards/README.md | 79 +++-- .../capture-native-evaluability-snapshot.mjs | 314 ++++++++++++++++++ src/rulesets/standards/iso-5055-mapping.csv | 44 +-- src/rulesets/standards/iso-5055-mapping.json | 93 +++--- .../standards/iso-5055-mapping.test.mjs | 131 +++++++- .../native-evaluability-snapshot.json | 52 +-- 11 files changed, 916 insertions(+), 270 deletions(-) create mode 100644 src/packages/core-domain/test/rule-corpus-triage.ts create mode 100644 src/rulesets/standards/capture-native-evaluability-snapshot.mjs diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 1caea1de6..4d49b6cdf 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -99,6 +99,23 @@ jobs: - name: Self-tests for the ADR ruleset generator run: node --test .harness/scripts/generate-adr-rulesets.test.mjs + # GT-598: `native-evaluability-snapshot.json` says of itself that it is a + # capture, and until now nothing captured it — it was written by hand, and + # it drifted. It still declared `documentation-only: 129` after Core moved + # to 136 (the same seven ADR rulesets the step above exists for), and its + # guard could not see that, because it compared the snapshot against six + # numbers hardcoded in the test: the same six the snapshot contained. + # + # Drift here is not contained. `build-iso-5055-mapping.mjs` stamps + # nativeEvaluability onto all 388 rows of the mapping FROM this file, so a + # stale class is laundered into the larger artifact and overstates the + # handler backlog. The order of these two steps is the order of the chain. + - name: Native evaluability snapshot is a faithful capture of Core's triage + run: node src/rulesets/standards/capture-native-evaluability-snapshot.mjs --check + + - name: ISO/IEC 5055 mapping guard + run: node --test src/rulesets/standards/iso-5055-mapping.test.mjs + # GT-630: the derived artifacts have a dependency ORDER — the executive # summary is built FROM the maturity reconciliation, which is built from # the board. Generate the summary first and it captures a value the diff --git a/.gitignore b/.gitignore index 453187c08..2030016fb 100644 --- a/.gitignore +++ b/.gitignore @@ -38,7 +38,12 @@ bower_components build/Release # Dependency directories -node_modules/ +# No trailing slash on purpose: `node_modules/` matches directories ONLY, and a +# fresh worktree has no node_modules of its own — the documented workaround is to +# symlink the main checkout's. A symlink is not a directory, so the slashed form +# left it untracked-but-visible, and `git add -A` staged a link to an absolute +# path on one machine. +node_modules .harness/bin/ jspm_packages/ diff --git a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts index e5ff64e73..9081829a0 100644 --- a/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts +++ b/src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts @@ -22,128 +22,37 @@ import * as fs from 'fs'; import * as path from 'path'; -import { NativeEvaluator } from './evaluators/native-evaluator'; -import { INativeRuleHandler } from './evaluators/handlers/rule-handler.interface'; -import { NormalizedRule } from '../../domain/models/normalized-rule'; import { - ClassifiedRule, RULE_TRIAGE, RuleEvaluability, - classifyRule, isNonExecutable, - summarizeEvaluability, } from './rule-evaluability'; +import { RULESETS_ROOT, triageCorpus } from '../../../test/rule-corpus-triage'; -const REPO_ROOT = path.resolve(__dirname, '../../../../../..'); -const RULESETS_ROOT = path.join(REPO_ROOT, 'src', 'rulesets'); - -// --------------------------------------------------------------------------- -// Load the corpus exactly as DiskRulesetRepository does, minus the schema pass -// (this suite measures classification, not schema conformance). -// --------------------------------------------------------------------------- - -function rulesetFiles(dir: string, depth = 0): string[] { - if (depth > 4) return []; - const out: string[] = []; - for (const entry of fs.readdirSync(dir).sort()) { - const full = path.join(dir, entry); - if (entry.endsWith('.rules.json')) { out.push(full); continue; } - if (!entry.includes('.') && fs.statSync(full).isDirectory()) out.push(...rulesetFiles(full, depth + 1)); - } - return out; -} - -const CATEGORY_BY_PREFIX: Record = { - inh: 'inheritance', acl: 'anti-corruption', ocb: 'open-core', gov: 'governance', - evd: 'identity', 'obs-evd': 'tracing', dep: 'version-pinning', tax: 'naming-conventions', - hxa: 'layer-structure', git: 'branch-naming', cicd: 'ci-cd', tpy: 'testing-pyramid', - mtn: 'multi-tenancy', prot: 'protocol', runt: 'multi-runtime', dora: 'metrics', - space: 'metrics', drift: 'governance', 'cli-rr': 'build', 'cli-par': 'shared-logic', - mcp: 'protocol', 'modular-monolith': 'topology', 'distributed-modules': 'module-autonomy', - microservices: 'autonomous-deployment', -}; - -function deriveCategory(raw: Record): string { - if (raw['category']) return String(raw['category']); - const prefix = String(raw['id'] ?? '') - .replace(/-(?:EVD|RR|PAR)-?\d*$/, '') - .replace(/-\d+$/, '') - .toLowerCase(); - return CATEGORY_BY_PREFIX[prefix] ?? 'general'; -} - -function deriveSeverity(raw: Record): NormalizedRule['severity'] { - const declared = String(raw['severity'] ?? '').toUpperCase().trim(); - if (declared === 'MUST NOT') return 'MUST NOT'; - if (declared === 'MUST') return 'MUST'; - if (declared === 'SHOULD') return 'SHOULD'; - if (declared === 'COULD' || declared === 'MAY') return 'COULD'; - return raw['blocking'] === true || raw['enforcement'] ? 'MUST' : 'SHOULD'; -} - -function loadCorpus(): NormalizedRule[] { - const rules: NormalizedRule[] = []; - for (const file of rulesetFiles(RULESETS_ROOT)) { - const parsed = JSON.parse(fs.readFileSync(file, 'utf8')) as Record; - const list = (parsed['rules'] ?? parsed['principles']) as Array> | undefined; - if (!Array.isArray(list)) continue; - if (list.length > 0 && !list[0]['id'] && list[0]['rules']) continue; - - for (const raw of list.filter(r => Boolean(r['id']))) { - // `blocking` defaults from the RAW severity string, exactly as - // DiskRulesetRepository.defaultBlocking does — NOT from the normalized - // severity, which promotes `enforcement`-bearing rules to MUST. - const rawSeverity = String(raw['severity'] ?? '').toUpperCase(); - rules.push({ - id: String(raw['id']), - severity: deriveSeverity(raw), - category: deriveCategory(raw), - title: String(raw['title'] ?? raw['principle'] ?? raw['id']), - description: String(raw['description'] ?? raw['statement'] ?? ''), - blocking: Boolean(raw['blocking'] ?? (rawSeverity === 'MUST' || rawSeverity === 'MUST NOT')), - validationQuery: raw['validationQuery'] ? String(raw['validationQuery']) : undefined, - // GT-632: `enforce` was missing here for the same reason it was missing - // from DiskRulesetRepository — and while it was missing, this suite could - // not see the four rules that carry a machine-readable check, so it - // measured them as handler backlog. A loader that drops a field the - // handlers dispatch on does not "load the corpus exactly as - // DiskRulesetRepository does"; it measures a different corpus. - enforce: raw['enforce'] as NormalizedRule['enforce'], - sourceFile: path.relative(REPO_ROOT, file), - }); - } - } - return rules; -} - -/** The real handler set, asked only which rules it CLAIMS (no I/O performed). */ -function handlerSet(): INativeRuleHandler[] { - const evaluator = new NativeEvaluator({} as never, {} as never, {} as never); - return (evaluator as unknown as { handlers: INativeRuleHandler[] }).handlers; -} - -const CORPUS = loadCorpus(); -const HANDLERS = handlerSet(); - -/** A handler claims the rule — it is routed somewhere instead of falling through. */ -const claims = (rule: NormalizedRule) => HANDLERS.some(h => h.canHandle(rule)); +// The corpus loader, the real handler set and the classification live in +// `test/rule-corpus-triage.ts`. They used to live HERE, which meant jest was the +// only thing that could run them — and that is why +// `src/rulesets/standards/native-evaluability-snapshot.json`, which needs exactly +// this computation, was maintained by hand and drifted. The last describe block +// below is the other half of the repair. +const { corpus: CORPUS, classified: CLASSIFIED, summary: SUMMARY, claims } = triageCorpus(); /** - * A handler claims the rule AND can return a verdict for it. + * The class counts this repository is pinned to. * - * {@link AdrConformanceRuleHandler} is deliberately excluded: it claims the 126 - * generated rules only to CLASSIFY them, and never returns `passed`. Counting - * "claimed" as "evaluated" would reproduce, one layer up, exactly the false - * green GT-569 removed — so this predicate, not `claims`, drives the breakdown. + * Also the counts `native-evaluability-snapshot.json` must reproduce, and the + * ones its capture script writes. Read out of this file by + * `src/rulesets/standards/iso-5055-mapping.test.mjs`, which runs in a job with no + * node_modules and so cannot recompute them — keep it a plain literal. */ -const evaluates = (rule: NormalizedRule) => claims(rule) && rule.category !== 'adr-conformance'; - -const CLASSIFIED: ClassifiedRule[] = CORPUS.map(rule => { - const { evaluability, why } = classifyRule(rule, evaluates(rule)); - return { ruleId: rule.id, sourceFile: rule.sourceFile, blocking: rule.blocking, evaluability, why }; -}); - -const SUMMARY = summarizeEvaluability(CLASSIFIED); +const PINNED_CLASS_COUNTS: Readonly> = { + 'native-handler': 154, + 'documentation-only': 136, + 'unimplemented-native': 48, + 'needs-external-system': 20, + 'needs-runtime': 17, + underspecified: 14, +}; describe('GT-595 · the corpus is fully classified', () => { it('loads a corpus of the expected size (guards the measurement itself)', () => { @@ -203,14 +112,7 @@ describe('GT-595 · the published breakdown, with its denominator', () => { // (CLI-EXIT-01/02/03) with `CliExitTaxonomyRuleHandler` claiming all three, // so the corpus grows by three and every one of them lands directly in // `native-handler`. No other class moves: these ids did not exist before. - expect(SUMMARY.byClass).toEqual({ - 'native-handler': 154, - 'documentation-only': 136, - 'unimplemented-native': 48, - 'needs-external-system': 20, - 'needs-runtime': 17, - underspecified: 14, - }); + expect(SUMMARY.byClass).toEqual(PINNED_CLASS_COUNTS); }); it('publishes the honest denominator: 150 rules nothing can ever run', () => { @@ -468,3 +370,60 @@ describe('GT-595 · OCB-02 is vacuous as written — measured, not asserted', () expect(parsed.openCoreMatrix?.enterprise?.length).toBeGreaterThan(0); }); }); + +/** + * GT-598 — the snapshot `src/rulesets/standards` consumes must be a CAPTURE. + * + * `native-evaluability-snapshot.json` is read by `build-iso-5055-mapping.mjs`, + * which stamps `nativeEvaluability` onto every row of the ISO/IEC 5055 mapping. + * A stale snapshot therefore does not stay contained: it launders a wrong + * classification into a larger derived artifact and misstates the handler + * backlog. It said `documentation-only: 129` long after this file pinned 136, + * and its own guard could not see that — it compared the snapshot against six + * numbers typed into the test, the same six the snapshot already contained. + * + * This is where the truth is computed, so this is where the comparison belongs. + * The guard in `src/rulesets/standards` runs in a job with no node_modules and + * cannot recompute anything; it reads {@link PINNED_CLASS_COUNTS} out of this + * file instead. If this block fails, run: + * node src/rulesets/standards/capture-native-evaluability-snapshot.mjs + * node src/rulesets/standards/build-iso-5055-mapping.mjs + * in that order — the second consumes what the first writes. + */ +describe('GT-598 · the captured snapshot still matches a fresh triage', () => { + const SNAPSHOT_PATH = path.join(RULESETS_ROOT, 'standards', 'native-evaluability-snapshot.json'); + const snapshot = JSON.parse(fs.readFileSync(SNAPSHOT_PATH, 'utf8')) as { + counts: Record; + classes: Record; + }; + + it('agrees on the class counts', () => { + expect(snapshot.counts).toEqual(PINNED_CLASS_COUNTS); + }); + + it('agrees on the class of every rule, id by id', () => { + // Duplicate ids across rulesets collapse to one entry in the snapshot, so + // compare against the same collapse rather than against the corpus length. + const fresh: Record = {}; + for (const c of CLASSIFIED) fresh[c.ruleId] = c.evaluability; + + const drifted = Object.keys(fresh) + .filter(id => snapshot.classes[id] !== fresh[id]) + .map(id => `${id}: snapshot says ${snapshot.classes[id] ?? '(absent)'}, triage says ${fresh[id]}`); + const orphaned = Object.keys(snapshot.classes).filter(id => !(id in fresh)); + + expect(drifted).toEqual([]); + expect(orphaned).toEqual([]); + }); + + it('carries counts that agree with its own per-rule classes', () => { + // A snapshot whose header disagreed with its body would let the .mjs guard + // pass on the header while the mapping was stamped from the body. + const tally: Record = {}; + for (const rule of CORPUS) { + const klass = snapshot.classes[rule.id]; + tally[klass] = (tally[klass] ?? 0) + 1; + } + expect(tally).toEqual(snapshot.counts); + }); +}); diff --git a/src/packages/core-domain/test/rule-corpus-triage.ts b/src/packages/core-domain/test/rule-corpus-triage.ts new file mode 100644 index 000000000..d77d44a3c --- /dev/null +++ b/src/packages/core-domain/test/rule-corpus-triage.ts @@ -0,0 +1,163 @@ +/** + * GT-598 — the corpus triage, extracted so it has more than one caller. + * + * WHY THIS FILE EXISTS + * This code used to live inside `rule-corpus-triage.spec.ts`, where it was + * reachable only by jest. That made `src/rulesets/standards/native-evaluability- + * snapshot.json` impossible to CAPTURE: nothing outside the spec could ask the + * real handler set which rules it claims, so the snapshot was maintained by + * hand, and drifted (it still said `documentation-only: 129` after Core moved to + * 136, and its guard could not see it because the guard re-read the snapshot's + * own literals). + * + * So the measurement moved here. It has two callers now, and they are the two + * halves of the same guarantee: + * + * - `rule-corpus-triage.spec.ts` pins the numbers and asserts the committed + * snapshot still agrees with a freshly computed triage; + * - `src/rulesets/standards/capture-native-evaluability-snapshot.mjs` + * regenerates that snapshot from this same computation. + * + * It lives under `test/` rather than `src/` deliberately: it walks the + * repository with `fs`, which is not something the published application layer + * should do, and `files: ["dist"]` keeps it out of the package. + * + * Nothing here is mocked. The handler set is the real {@link NativeEvaluator} + * one, asked only which rules it CLAIMS — no evaluation, no I/O. + */ + +import * as fs from 'fs'; +import * as path from 'path'; +import { NativeEvaluator } from '../src/application/validators/evaluators/native-evaluator'; +import { INativeRuleHandler } from '../src/application/validators/evaluators/handlers/rule-handler.interface'; +import { NormalizedRule } from '../src/domain/models/normalized-rule'; +import { + ADR_CONFORMANCE_CATEGORY, + ClassifiedRule, + EvaluabilitySummary, + classifyRule, + summarizeEvaluability, +} from '../src/application/validators/rule-evaluability'; + +/** Repository root, from this file's location (`/src/packages/core-domain/test`). */ +export const REPO_ROOT = path.resolve(__dirname, '../../../..'); +export const RULESETS_ROOT = path.join(REPO_ROOT, 'src', 'rulesets'); + +// --------------------------------------------------------------------------- +// Load the corpus exactly as DiskRulesetRepository does, minus the schema pass +// (this measures classification, not schema conformance). +// --------------------------------------------------------------------------- + +function rulesetFiles(dir: string, depth = 0): string[] { + if (depth > 4) return []; + const out: string[] = []; + for (const entry of fs.readdirSync(dir).sort()) { + const full = path.join(dir, entry); + if (entry.endsWith('.rules.json')) { out.push(full); continue; } + if (!entry.includes('.') && fs.statSync(full).isDirectory()) out.push(...rulesetFiles(full, depth + 1)); + } + return out; +} + +const CATEGORY_BY_PREFIX: Record = { + inh: 'inheritance', acl: 'anti-corruption', ocb: 'open-core', gov: 'governance', + evd: 'identity', 'obs-evd': 'tracing', dep: 'version-pinning', tax: 'naming-conventions', + hxa: 'layer-structure', git: 'branch-naming', cicd: 'ci-cd', tpy: 'testing-pyramid', + mtn: 'multi-tenancy', prot: 'protocol', runt: 'multi-runtime', dora: 'metrics', + space: 'metrics', drift: 'governance', 'cli-rr': 'build', 'cli-par': 'shared-logic', + mcp: 'protocol', 'modular-monolith': 'topology', 'distributed-modules': 'module-autonomy', + microservices: 'autonomous-deployment', +}; + +function deriveCategory(raw: Record): string { + if (raw['category']) return String(raw['category']); + const prefix = String(raw['id'] ?? '') + .replace(/-(?:EVD|RR|PAR)-?\d*$/, '') + .replace(/-\d+$/, '') + .toLowerCase(); + return CATEGORY_BY_PREFIX[prefix] ?? 'general'; +} + +function deriveSeverity(raw: Record): NormalizedRule['severity'] { + const declared = String(raw['severity'] ?? '').toUpperCase().trim(); + if (declared === 'MUST NOT') return 'MUST NOT'; + if (declared === 'MUST') return 'MUST'; + if (declared === 'SHOULD') return 'SHOULD'; + if (declared === 'COULD' || declared === 'MAY') return 'COULD'; + return raw['blocking'] === true || raw['enforcement'] ? 'MUST' : 'SHOULD'; +} + +export function loadCorpus(rulesetsRoot: string = RULESETS_ROOT): NormalizedRule[] { + const rules: NormalizedRule[] = []; + for (const file of rulesetFiles(rulesetsRoot)) { + const parsed = JSON.parse(fs.readFileSync(file, 'utf8')) as Record; + const list = (parsed['rules'] ?? parsed['principles']) as Array> | undefined; + if (!Array.isArray(list)) continue; + if (list.length > 0 && !list[0]['id'] && list[0]['rules']) continue; + + for (const raw of list.filter(r => Boolean(r['id']))) { + // `blocking` defaults from the RAW severity string, exactly as + // DiskRulesetRepository.defaultBlocking does — NOT from the normalized + // severity, which promotes `enforcement`-bearing rules to MUST. + const rawSeverity = String(raw['severity'] ?? '').toUpperCase(); + rules.push({ + id: String(raw['id']), + severity: deriveSeverity(raw), + category: deriveCategory(raw), + title: String(raw['title'] ?? raw['principle'] ?? raw['id']), + description: String(raw['description'] ?? raw['statement'] ?? ''), + blocking: Boolean(raw['blocking'] ?? (rawSeverity === 'MUST' || rawSeverity === 'MUST NOT')), + validationQuery: raw['validationQuery'] ? String(raw['validationQuery']) : undefined, + // GT-632: `enforce` was missing here for the same reason it was missing + // from DiskRulesetRepository — and while it was missing, this measurement + // could not see the four rules that carry a machine-readable check, so it + // counted them as handler backlog. A loader that drops a field the + // handlers dispatch on does not "load the corpus exactly as + // DiskRulesetRepository does"; it measures a different corpus. + enforce: raw['enforce'] as NormalizedRule['enforce'], + sourceFile: path.relative(REPO_ROOT, file), + }); + } + } + return rules; +} + +/** The real handler set, asked only which rules it CLAIMS (no I/O performed). */ +export function handlerSet(): INativeRuleHandler[] { + const evaluator = new NativeEvaluator({} as never, {} as never, {} as never); + return (evaluator as unknown as { handlers: INativeRuleHandler[] }).handlers; +} + +export interface CorpusTriage { + readonly corpus: readonly NormalizedRule[]; + readonly classified: readonly ClassifiedRule[]; + readonly summary: EvaluabilitySummary; + /** A handler claims the rule — it is routed somewhere instead of falling through. */ + readonly claims: (rule: NormalizedRule) => boolean; + /** A handler claims the rule AND can return a verdict for it. */ + readonly evaluates: (rule: NormalizedRule) => boolean; +} + +/** + * Classify the whole corpus with the real handler set. + * + * {@link import('../src/application/validators/evaluators/handlers/adr-conformance-rule.handler').AdrConformanceRuleHandler} + * is deliberately excluded from `evaluates`: it claims the generated ADR rules + * only to CLASSIFY them, and never returns `passed`. Counting "claimed" as + * "evaluated" would reproduce, one layer up, exactly the false green GT-569 + * removed — so `evaluates`, not `claims`, drives the breakdown. + */ +export function triageCorpus(rulesetsRoot: string = RULESETS_ROOT): CorpusTriage { + const corpus = loadCorpus(rulesetsRoot); + const handlers = handlerSet(); + + const claims = (rule: NormalizedRule) => handlers.some(h => h.canHandle(rule)); + const evaluates = (rule: NormalizedRule) => claims(rule) && rule.category !== ADR_CONFORMANCE_CATEGORY; + + const classified: ClassifiedRule[] = corpus.map(rule => { + const { evaluability, why } = classifyRule(rule, evaluates(rule)); + return { ruleId: rule.id, sourceFile: rule.sourceFile, blocking: rule.blocking, evaluability, why }; + }); + + return { corpus, classified, summary: summarizeEvaluability(classified), claims, evaluates }; +} diff --git a/src/rulesets/standards/README.es.md b/src/rulesets/standards/README.es.md index d2b0adedf..15bca204f 100644 --- a/src/rulesets/standards/README.es.md +++ b/src/rulesets/standards/README.es.md @@ -17,10 +17,23 @@ confiar en nosotros. | `iso-5055-weaknesses.json` | Los 138 identificadores CWE de ISO/IEC 5055, por medida, con procedencia. | | `iso-5055-mapping.json` | Una fila por regla del corpus: mapeo a CWE (o un "sin equivalente" explícito con motivo), adoptabilidad por analizador y clase de evaluabilidad nativa. | | `iso-5055-mapping.csv` | La misma tabla, plana, para hojas de cálculo y auditores. | -| `native-evaluability-snapshot.json` | Captura por regla de la clase de evaluabilidad del triage del Core, para acotar la aritmética de backlog de abajo al backlog real. | +| `native-evaluability-snapshot.json` | Captura por regla de la clase de evaluabilidad del triage del Core, para acotar la aritmética de backlog de abajo al backlog real. Generado — no editar a mano. | +| `capture-native-evaluability-snapshot.mjs` | La captura. Ejecuta el triage real del Core vía `ts-node`; `--check` falla cuando la captura comiteada ya no es lo que el Core calcula. | | `build-iso-5055-mapping.mjs` | El generador. `--check` falla cuando la tabla se quedó atrás del corpus. | | `iso-5055-mapping.test.mjs` | La guarda. `node --test src/rulesets/standards/iso-5055-mapping.test.mjs`. | +### Regenerar, en orden + +``` +node src/rulesets/standards/capture-native-evaluability-snapshot.mjs # 1. captura +node src/rulesets/standards/build-iso-5055-mapping.mjs # 2. mapeo +``` + +El orden no es una preferencia. El generador estampa `nativeEvaluability` en cada fila del mapeo a +partir de la captura, así que reconstruir primero blanquea una clasificación obsoleta dentro de un +artefacto de 391 filas y sobreestima el backlog de handlers en tantas reglas como el Core haya cerrado +desde la última captura. + Aquí no se reproduce texto de ISO/IEC 5055 ni de ISO/IEC 25010. Solo se registran identificadores CWE, nombres CWE de MITRE y pertenencia a cada medida. @@ -34,20 +47,20 @@ hace que la unión sea menor que la suma. La fecha y el método de extracción c ## Resultado -Sobre **381 reglas** en 167 archivos de ruleset: +Sobre **391 reglas** en 175 archivos de ruleset: | Medida | Cantidad | Proporción | |---|---|---| -| Reglas mapeadas a una debilidad ISO/IEC 5055 | 37 | 9,7% | +| Reglas mapeadas a una debilidad ISO/IEC 5055 | 37 | 9,5% | | — de ellas con mapeo directo | 8 | | | — de ellas con mapeo parcial / proxy | 29 | | -| Reglas sin equivalente internacional (cada una con motivo declarado) | 344 | 90,3% | -| Reglas que un analizador existente podría decidir por completo | 42 | 11,0% | -| Reglas que un analizador podría decidir parcialmente | 23 | 6,0% | +| Reglas sin equivalente internacional (cada una con motivo declarado) | 354 | 90,5% | +| Reglas que un analizador existente podría decidir por completo | 42 | 10,7% | +| Reglas que un analizador podría decidir parcialmente | 23 | 5,9% | -**La fracción adoptada es el 9,7% del corpus.** Es un resultado real y es menor de lo que sugería la +**La fracción adoptada es el 9,5% del corpus.** Es un resultado real y es menor de lo que sugería la premisa del gap, por una razón estructural: ISO/IEC 5055 mide la estructura interna del código fuente, -y 300 de nuestras 381 reglas no tratan de estructura de código. Son conformidad con ADR (155), +y 313 de nuestras 391 reglas no tratan de estructura de código. Son conformidad con ADR (162), contratos de topología (66), invariantes de gobierno (51) y proceso de desarrollo (34). Ningún estándar internacional modela "¿honraste el ADR-0092?", y ninguno lo hará. @@ -57,28 +70,37 @@ mapean y 13 son decidibles por analizador. ## Re-dimensionar el backlog de handlers El enunciado del gap dimensionaba el beneficio contra "~240 handlers por escribir". **Esa cifra ya está -retirada.** GT-595 hizo el triage del corpus y el backlog real, decidible desde el repositorio, son **60 -reglas** — la clase `unimplemented-native`. Las otras 180 son 129 placeholders de generador solo -documentales, 14 reglas sin check redactado, 20 que requieren un sistema externo y 17 que requieren uno -en ejecución. +retirada.** GT-595 hizo el triage del corpus y el backlog real, decidible desde el repositorio, son **48 +reglas** — la clase `unimplemented-native`. De las 389 reglas que carga el triage del Core, 154 ya se +ejecutan y las otras 187 son 136 placeholders de generador solo documentales, 14 reglas sin check +redactado, 20 que requieren un sistema externo y 17 que requieren uno en ejecución. -Proyectar este mapeo sobre esa clase es la cifra que importa: +(389, no 391: los dos archivos de regla única llevan sus metadatos en la raíz del documento, y el +cargador de corpus del Core no los lee. Aparecen en el mapeo como `not-in-snapshot`.) -| Del backlog de 60 handlers | Cantidad | -|---|---| -| Decidibles hoy por un analizador estándar | 9 | -| Decidibles parcialmente (señal necesaria pero no suficiente) | 8 | -| Que hay que escribir de verdad | 43 | +60 → 48 el 2026-07-29: ocho reglas con forma de configuración recibieron handler (GT-595) y cuatro +reglas de límites de módulo ya traían una cláusula `enforce` completa que la normalización descartaba +(GT-632). -Las 9 son `HXA-01`…`HXA-04` (estructura de capas — dependency-cruiser o ArchUnit), `GIT-08` -(commitlint), `SEC-INJ-01`, `SEC-PATH-01`, `SEC-PATH-02` (consultas de inyección y path traversal de -CodeQL/Semgrep) y `SEC-TIMING-01` (comparación en tiempo constante). Las 8 parciales están listadas en -`handlerBacklog.byEvaluabilityClass` del JSON de mapeo. +Proyectar este mapeo sobre esa clase es la cifra que importa: -Es decir, adoptar vale **15% del backlog por completo, 28% incluyendo parciales** — 17 de 60 reglas que -no necesitan handlers a medida. No es el orden de magnitud que esperaba el gap, pero es una reducción -concreta y nominada, y apunta las reglas de seguridad hacia analizadores mejores que cualquier cosa que -escribiéramos. +| Del backlog de 48 handlers | Cantidad | +|---|---| +| Decidibles hoy por un analizador estándar | 5 | +| Decidibles parcialmente (señal necesaria pero no suficiente) | 5 | +| Que hay que escribir de verdad | 38 | + +Las 5 son `HXA-03` (estructura de capas — dependency-cruiser o ArchUnit), `SEC-INJ-01`, `SEC-PATH-01`, +`SEC-PATH-02` (consultas de inyección y path traversal de CodeQL/Semgrep) y `SEC-TIMING-01` (comparación +en tiempo constante). Las 5 parciales están listadas en `handlerBacklog.byEvaluabilityClass` del JSON de +mapeo. + +Es decir, adoptar vale **10,4% del backlog por completo, 20,8% incluyendo parciales** — 10 de 48 reglas +que no necesitan handlers a medida. Ambas proporciones bajaron al encogerse el backlog, y esa es la +dirección correcta: entre las doce reglas cerradas el 2026-07-29 están `HXA-01`, `HXA-02`, `HXA-04` y +`GIT-08`, que eran cuatro de las nueve que esta tabla ofrecía a un analizador. Evolith escribió el +handler primero. Lo que queda se inclina más hacia escribir, y las reglas de seguridad siguen apuntando +a analizadores mejores que cualquier cosa que escribiéramos. ## Taxonomía compañera: ISO/IEC 25010:2023 @@ -101,3 +123,16 @@ reintroduce. `nativeEvaluability` se copia del snapshot de triage del Core. La autoridad es `src/packages/core-domain/src/application/validators/rule-evaluability.ts` y su spec fijado; si esas cifras se mueven, el snapshot de aquí está obsoleto y hay que recapturarlo. + +## Cómo se detecta la deriva + +El snapshot se mantenía a mano hasta GT-598, y derivó: declaraba `documentation-only: 129` mucho después +de que el Core pasara a 136, y la guarda que debía notarlo comparaba el snapshot contra seis números +escritos en el propio test — los mismos seis que el snapshot ya contenía. Comparaba el snapshot consigo +mismo y solo podía pasar. La reemplazan tres controles, en los dos jobs que pueden costearlos: + +| Dónde | Qué prueba | +|---|---| +| `rule-corpus-triage.spec.ts` (jest de core-domain) | El snapshot comiteado es igual a un triage **recalculado** — las cuentas y la clase de cada regla. Esta es la guarda real: tiene las dependencias para recalcular la verdad. | +| `iso-5055-mapping.test.mjs` (job de documentación, sin `node_modules`) | Las cuentas del snapshot son iguales a las **leídas desde** ese spec, su cabecera concuerda con su propio cuerpo, y toda clase estampada en el mapeo coincide con el snapshot. | +| `capture-native-evaluability-snapshot.mjs --check` | La re-derivación extremo a extremo, ejecutable donde el workspace esté instalado. | diff --git a/src/rulesets/standards/README.md b/src/rulesets/standards/README.md index f0802b358..504d65661 100644 --- a/src/rulesets/standards/README.md +++ b/src/rulesets/standards/README.md @@ -17,10 +17,22 @@ to trust us. | `iso-5055-weaknesses.json` | The 138 CWE identifiers of ISO/IEC 5055, split per measure, with provenance. | | `iso-5055-mapping.json` | One row per corpus rule: CWE mapping (or an explicit "no equivalent" with a reason), analyser adoptability, and the rule's native evaluability class. | | `iso-5055-mapping.csv` | The same table, flat, for spreadsheets and auditors. | -| `native-evaluability-snapshot.json` | Captured per-rule evaluability class from Core's triage, so the backlog arithmetic below is scoped to the real backlog. | +| `native-evaluability-snapshot.json` | Captured per-rule evaluability class from Core's triage, so the backlog arithmetic below is scoped to the real backlog. Generated — do not edit by hand. | +| `capture-native-evaluability-snapshot.mjs` | The capture. Runs Core's real triage through `ts-node`; `--check` fails when the committed snapshot is no longer what Core computes. | | `build-iso-5055-mapping.mjs` | The generator. `--check` fails when the table has fallen behind the corpus. | | `iso-5055-mapping.test.mjs` | The guard. `node --test src/rulesets/standards/iso-5055-mapping.test.mjs`. | +### Regenerating, in order + +``` +node src/rulesets/standards/capture-native-evaluability-snapshot.mjs # 1. snapshot +node src/rulesets/standards/build-iso-5055-mapping.mjs # 2. mapping +``` + +The order is not a preference. The generator stamps `nativeEvaluability` onto every row of the mapping +from the snapshot, so rebuilding first launders a stale classification into a 391-row artifact and +overstates the handler backlog by however many rules Core has closed since the last capture. + No text of ISO/IEC 5055 or ISO/IEC 25010 is reproduced here. Only CWE identifiers, MITRE CWE names and measure membership are recorded. @@ -34,20 +46,20 @@ makes the union smaller than the sum. The extraction date and method are recorde ## Result -Against **381 rules** in 167 ruleset files: +Against **391 rules** in 175 ruleset files: | Measure | Count | Share | |---|---|---| -| Rules mapped to an ISO/IEC 5055 weakness | 37 | 9.7% | +| Rules mapped to an ISO/IEC 5055 weakness | 37 | 9.5% | | — of which the mapping is direct | 8 | | | — of which the mapping is a partial / proxy | 29 | | -| Rules with no international equivalent (each with a stated reason) | 344 | 90.3% | -| Rules an existing analyser could decide outright | 42 | 11.0% | -| Rules an analyser could decide partially | 23 | 6.0% | +| Rules with no international equivalent (each with a stated reason) | 354 | 90.5% | +| Rules an existing analyser could decide outright | 42 | 10.7% | +| Rules an analyser could decide partially | 23 | 5.9% | -**The adopted fraction is 9.7% of the corpus.** That is a real result and it is smaller than the gap's +**The adopted fraction is 9.5% of the corpus.** That is a real result and it is smaller than the gap's premise implied, for a structural reason: ISO/IEC 5055 measures the internal structure of source code, -and 300 of our 381 rules are not about source structure at all. They are ADR conformance (155), +and 313 of our 391 rules are not about source structure at all. They are ADR conformance (162), topology contracts (66), governance invariants (51) and development process (34). No international standard models "did you honour ADR-0092", and none ever will. @@ -57,27 +69,35 @@ and 13 are analyser-decidable. ## Re-scoping the handler backlog The gap statement sized the payoff against "~240 handlers to write". **That figure is already -retired.** GT-595 triaged the corpus and the real, decidable-from-the-repository backlog is **60 -rules** — the `unimplemented-native` class. The other 180 are 129 documentation-only generator -placeholders, 14 underspecified rules with no authored check, 20 that need an external system and 17 -that need a running one. +retired.** GT-595 triaged the corpus and the real, decidable-from-the-repository backlog is **48 +rules** — the `unimplemented-native` class. Of the 389 rules Core's triage loads, 154 already run and +the other 187 are 136 documentation-only generator placeholders, 14 underspecified rules with no +authored check, 20 that need an external system and 17 that need a running one. + +(389, not 391: the two single-rule infrastructure files carry their rule metadata at the document root, +which Core's corpus loader does not read. They appear in the mapping as `not-in-snapshot`.) + +60 → 48 on 2026-07-29: eight config-shaped rules got handlers (GT-595) and four module-boundary rules +turned out to already carry a complete `enforce` clause that normalization was dropping (GT-632). Folding this mapping onto that class is the number that matters: -| Of the 60-rule handler backlog | Count | +| Of the 48-rule handler backlog | Count | |---|---| -| Decidable today by an off-the-shelf analyser | 9 | -| Decidable partially (analyser gives a necessary-but-not-sufficient signal) | 8 | -| Genuinely has to be authored | 43 | +| Decidable today by an off-the-shelf analyser | 5 | +| Decidable partially (analyser gives a necessary-but-not-sufficient signal) | 5 | +| Genuinely has to be authored | 38 | -The 9 are `HXA-01`…`HXA-04` (layer structure — dependency-cruiser or ArchUnit), `GIT-08` (commitlint), -`SEC-INJ-01`, `SEC-PATH-01`, `SEC-PATH-02` (CodeQL/Semgrep injection and path-traversal queries) and -`SEC-TIMING-01` (timing-safe comparison). The 8 partials are listed in -`handlerBacklog.byEvaluabilityClass` in the mapping JSON. +The 5 are `HXA-03` (layer structure — dependency-cruiser or ArchUnit), `SEC-INJ-01`, `SEC-PATH-01`, +`SEC-PATH-02` (CodeQL/Semgrep injection and path-traversal queries) and `SEC-TIMING-01` (timing-safe +comparison). The 5 partials are listed in `handlerBacklog.byEvaluabilityClass` in the mapping JSON. -So adoption is worth **15% of the backlog outright, 28% including partials** — 17 of 60 rules that do -not need bespoke handlers. Not the order-of-magnitude the gap hoped for, but a concrete, named -reduction, and it points the security rules at analysers that are better than anything we would write. +So adoption is worth **10.4% of the backlog outright, 20.8% including partials** — 10 of 48 rules that +do not need bespoke handlers. Both shares fell when the backlog shrank, and that is the right +direction: the twelve rules closed on 2026-07-29 include `HXA-01`, `HXA-02`, `HXA-04` and `GIT-08`, +which were four of the nine this table used to offer to an analyser. Evolith wrote the handler first. +What is left leans further toward authoring, and the security rules still point at analysers better +than anything we would write. ## Companion taxonomy: ISO/IEC 25010:2023 @@ -99,3 +119,16 @@ stale, and `iso-5055-mapping.test.mjs` fails the build if anything in this direc `nativeEvaluability` is copied from the Core triage snapshot. The authority for it is `src/packages/core-domain/src/application/validators/rule-evaluability.ts` and its pinned spec; if the counts there move, the snapshot here is stale and must be recaptured. + +## How drift is caught + +The snapshot was hand-maintained until GT-598, and it drifted: it declared `documentation-only: 129` +long after Core had moved to 136, and the guard that was meant to notice compared the snapshot against +six numbers typed into the test — the same six the snapshot already contained. It compared the snapshot +to itself and could only ever pass. Three checks replace it, in the two jobs that can afford them: + +| Where | What it proves | +|---|---| +| `rule-corpus-triage.spec.ts` (core-domain jest) | The committed snapshot equals a **freshly computed** triage — counts and every per-rule class. This is the real guard; it has the dependencies to recompute the truth. | +| `iso-5055-mapping.test.mjs` (documentation job, no `node_modules`) | The snapshot's counts equal the counts **read out of** that spec, its header agrees with its own body, and every class stamped into the mapping matches the snapshot. | +| `capture-native-evaluability-snapshot.mjs --check` | The end-to-end re-derivation, runnable anywhere the workspace is installed. | diff --git a/src/rulesets/standards/capture-native-evaluability-snapshot.mjs b/src/rulesets/standards/capture-native-evaluability-snapshot.mjs new file mode 100644 index 000000000..b3ec03790 --- /dev/null +++ b/src/rulesets/standards/capture-native-evaluability-snapshot.mjs @@ -0,0 +1,314 @@ +#!/usr/bin/env node +/** + * @file capture-native-evaluability-snapshot.mjs + * @description GT-598 — capture `native-evaluability-snapshot.json` from Core. + * + * WHY THIS EXISTS + * --------------- + * The snapshot said, in its own header, that it was "a CAPTURED SNAPSHOT, not + * the source of truth". No capture script existed. It was written by hand, and + * it drifted: it still declared `documentation-only: 129` long after Core moved + * to 136, and the guard that was supposed to notice + * (`iso-5055-mapping.test.mjs`) compared the snapshot's counts against six + * numbers hardcoded in the test — the same numbers the snapshot already + * contained. It could only ever pass. This script is the missing half. + * + * Drift here is not contained. `build-iso-5055-mapping.mjs` stamps + * `nativeEvaluability` onto every row of the ISO/IEC 5055 mapping from this + * file, so a stale class silently overstates the handler backlog in a much + * larger derived artifact. ALWAYS recapture before rebuilding the mapping. + * + * WHAT IT RUNS + * ------------ + * The real Core triage — `test/rule-corpus-triage.ts` in @beyondnet/evolith-core-domain, + * which instantiates the real `NativeEvaluator`, asks its handler set which + * rules it claims, and classifies the rest through `classifyRule`. Nothing is + * reimplemented here: a second implementation of the classification would be a + * second source of truth, which is the defect this script exists to remove. + * + * Core is TypeScript, so the triage is executed through `ts-node` (transpile + * only, no emit). That makes this the one script under `src/rulesets` that + * needs `node_modules` — which is exactly why the guard that runs in the + * dependency-free documentation job stays a separate, non-executing check. + * + * USAGE + * node src/rulesets/standards/capture-native-evaluability-snapshot.mjs + * node src/rulesets/standards/capture-native-evaluability-snapshot.mjs --check + * + * EXIT CODES + * 0 snapshot written, or (with --check) already identical to a fresh capture + * 1 the capture disagrees with the committed file, or the triage refused to + * run / returned an implausible corpus + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { createRequire } from 'node:module'; +import { execFileSync } from 'node:child_process'; + +const HERE = path.dirname(fileURLToPath(import.meta.url)); +const REPO_ROOT = path.resolve(HERE, '..', '..', '..'); +const OUT = path.join(HERE, 'native-evaluability-snapshot.json'); + +const CORE_DOMAIN = 'src/packages/core-domain'; +const TRIAGE_MODULE = `${CORE_DOMAIN}/test/rule-corpus-triage.ts`; +const TSCONFIG = `${CORE_DOMAIN}/tsconfig.json`; + +/** The classes Core can emit. A capture holding anything else is not a capture. */ +const KNOWN_CLASSES = [ + 'native-handler', + 'unimplemented-native', + 'needs-runtime', + 'needs-external-system', + 'documentation-only', + 'underspecified', +]; + +/** + * A corpus scan that finds nothing is never a real state — it is a moved path. + * Refuse to overwrite a good snapshot with the result of one. + */ +const MIN_PLAUSIBLE_CORPUS = 300; + +/** + * Packages that DECLARE ts-node, most-likely location first. + * + * `ts-node` is a devDependency of core-api and mcp-server, not of the root and + * not of core-domain. It normally lands in the root `node_modules` because npm + * workspaces hoist it, and resolving it from the repo root works — until a + * version conflict makes npm install it under the declaring package instead, at + * which point a root-relative `-r ts-node/register` fails on a clean runner and + * nowhere else. Resolving through the packages that actually declare it removes + * the dependency on hoisting. + */ +const TS_NODE_DECLARED_BY = [ + 'package.json', + 'src/packages/mcp-server/package.json', + 'src/apps/core-api/package.json', +]; + +/** Absolute path to ts-node's CommonJS register hook, or a stated failure. */ +function resolveTsNodeRegister() { + const tried = []; + for (const manifest of TS_NODE_DECLARED_BY) { + const from = path.join(REPO_ROOT, manifest); + if (!fs.existsSync(from)) continue; + try { + return createRequire(from).resolve('ts-node/register'); + } catch { + tried.push(manifest); + } + } + throw new Error( + 'ts-node is not resolvable, so Core\'s triage cannot be executed and the snapshot cannot be captured.\n' + + ` looked from: ${tried.join(', ') || '(no manifest found)'}\n` + + ' Install the workspace first: npm ci', + ); +} + +// --------------------------------------------------------------------------- +// 1. Run the real triage +// --------------------------------------------------------------------------- + +/** + * Execute Core's triage in a child process and read back its result as JSON. + * + * A child process rather than an import: this file is ESM under `src/rulesets`, + * Core is CommonJS TypeScript compiled by `ts-node`, and the boundary between + * them is a process, not a module resolution problem worth solving. + */ +function runTriage() { + const program = ` + const { triageCorpus } = require(${JSON.stringify(path.join(REPO_ROOT, TRIAGE_MODULE))}); + const { corpus, classified, summary } = triageCorpus(); + const classes = {}; + const conflicts = []; + for (const c of classified) { + if (classes[c.ruleId] && classes[c.ruleId] !== c.evaluability) { + conflicts.push(c.ruleId + ': ' + classes[c.ruleId] + ' vs ' + c.evaluability); + } + classes[c.ruleId] = c.evaluability; + } + process.stdout.write(JSON.stringify({ + corpusSize: corpus.length, + counts: summary.byClass, + classes, + conflicts, + })); + `; + + const register = resolveTsNodeRegister(); + + let raw; + try { + raw = execFileSync(process.execPath, ['-r', register, '-e', program], { + cwd: REPO_ROOT, + encoding: 'utf8', + maxBuffer: 32 * 1024 * 1024, + env: { + ...process.env, + TS_NODE_PROJECT: path.join(REPO_ROOT, TSCONFIG), + TS_NODE_TRANSPILE_ONLY: 'true', + }, + stdio: ['ignore', 'pipe', 'inherit'], + }); + } catch (err) { + throw new Error( + `Could not run the Core triage (${TRIAGE_MODULE}). This script needs the workspace ` + + `installed — ts-node and Core's dependencies must be resolvable from ${REPO_ROOT}.\n${err.message}`, + ); + } + + const result = JSON.parse(raw); + + if (result.conflicts.length > 0) { + throw new Error( + 'The same rule id is classified two different ways by the corpus, so a snapshot keyed by ' + + `rule id cannot represent it:\n - ${result.conflicts.join('\n - ')}`, + ); + } + if (result.corpusSize < MIN_PLAUSIBLE_CORPUS) { + throw new Error( + `The triage classified only ${result.corpusSize} rules (expected at least ${MIN_PLAUSIBLE_CORPUS}). ` + + 'A collapsed corpus is a moved path, not progress — refusing to capture it.', + ); + } + for (const [id, klass] of Object.entries(result.classes)) { + if (!KNOWN_CLASSES.includes(klass)) throw new Error(`${id} was classified as unknown class "${klass}".`); + } + + const summed = Object.values(result.counts).reduce((a, b) => a + b, 0); + if (summed !== result.corpusSize) { + throw new Error(`Core's own class counts sum to ${summed} but it classified ${result.corpusSize} rules.`); + } + + return result; +} + +// --------------------------------------------------------------------------- +// 2. Render the snapshot +// --------------------------------------------------------------------------- + +/** + * Keep the capture date stable when nothing else changed. + * + * `capturedOn` is provenance, not content: bumping it on a run that produced an + * identical classification would turn every recapture into a diff and make real + * drift harder to see in review. + */ +function capturedOn(previous, changed) { + if (!changed && previous?.capturedOn) return previous.capturedOn; + return new Date().toISOString().slice(0, 10); +} + +function render(triage, previous) { + const counts = Object.fromEntries( + KNOWN_CLASSES.filter((c) => triage.counts[c] !== undefined).map((c) => [c, triage.counts[c]]), + ); + const classes = Object.fromEntries(Object.entries(triage.classes)); + + const body = { + counts, + classes, + }; + const changed = + !previous || + JSON.stringify(previous.counts) !== JSON.stringify(body.counts) || + JSON.stringify(previous.classes) !== JSON.stringify(body.classes); + + const pinned = KNOWN_CLASSES.filter((c) => counts[c] !== undefined) + .map((c) => `${c} ${counts[c]}`) + .join(', '); + + const doc = { + $id: 'https://evolith.dev/rulesets/standards/native-evaluability-snapshot.json', + title: 'Native-engine evaluability class per rule (snapshot)', + description: + 'Per-rule evaluability class as computed by the Core native evaluator triage. This is a CAPTURED SNAPSHOT, not the source of truth: the authority is src/packages/core-domain/src/application/validators/rule-evaluability.ts and its pinned spec rule-corpus-triage.spec.ts. It is recorded here so the ISO/IEC 5055 mapping can be scoped to the real handler backlog without src/rulesets depending on a package it does not own.', + version: '1.1.0', + generatedBy: 'src/rulesets/standards/capture-native-evaluability-snapshot.mjs', + capturedOn: capturedOn(previous, changed), + capturedFrom: [ + 'src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, real handler set, classification)', + 'src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)', + 'src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)', + 'src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)', + 'src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (the pinned counts, asserted against this file)', + ], + validation: + `Captured from a live run of the Core triage over ${triage.corpusSize} rules (${pinned}). ` + + 'Recapture with `node src/rulesets/standards/capture-native-evaluability-snapshot.mjs`, then rebuild ' + + 'the mapping with `node src/rulesets/standards/build-iso-5055-mapping.mjs` — the mapping stamps ' + + 'nativeEvaluability per rule from this file, so rebuilding first would launder a stale class into it. ' + + 'Drift is caught in two places: rule-corpus-triage.spec.ts compares this file against a fresh triage ' + + '(core-domain jest), and iso-5055-mapping.test.mjs checks it against the counts pinned in that spec ' + + '(dependency-free documentation job).', + // Both numbers, because they are not the same question. `corpusSize` is what + // Core classified and what `counts` sums to; `distinctRuleIds` is how many + // keys `classes` can hold. They differ exactly when one rule id appears in + // more than one ruleset file, and a guard that assumed they were equal would + // go red on a corpus change that is not drift. + corpusSize: triage.corpusSize, + distinctRuleIds: Object.keys(classes).length, + ...body, + }; + + return { json: JSON.stringify(doc, null, 2) + '\n', changed }; +} + +// --------------------------------------------------------------------------- +// 3. Entry point +// --------------------------------------------------------------------------- + +const CHECK = process.argv.includes('--check'); + +/** + * Read the committed snapshot ONCE, and keep both views of that one read. + * + * Not `existsSync` then `readFileSync`, and not a second read for the byte + * comparison: `--check` decides whether the file on disk equals a fresh capture, + * so it must compare the bytes it actually parsed. Two reads can observe two + * different files — the generator itself writes this path — and then the diff + * reported would not be the diff that was measured. Absence is a normal state + * (first capture), so ENOENT is handled and every other error is raised. + */ +function readCommittedSnapshot() { + try { + return fs.readFileSync(OUT, 'utf8'); + } catch (err) { + if (err.code === 'ENOENT') return undefined; + throw err; + } +} + +const onDisk = readCommittedSnapshot(); +const previous = onDisk ? JSON.parse(onDisk) : undefined; +const triage = runTriage(); +const { json, changed } = render(triage, previous); + +if (CHECK) { + if (onDisk !== json) { + console.error( + 'GT-598: native-evaluability-snapshot.json no longer matches a fresh capture of the Core triage.\n' + + ' Re-run: node src/rulesets/standards/capture-native-evaluability-snapshot.mjs\n' + + ' Then: node src/rulesets/standards/build-iso-5055-mapping.mjs', + ); + if (previous) { + for (const c of KNOWN_CLASSES) { + const was = previous.counts?.[c]; + const now = triage.counts[c]; + if (was !== now) console.error(` ${c}: snapshot says ${was ?? '(absent)'}, Core says ${now ?? '(absent)'}`); + } + } + process.exit(1); + } + console.log(`GT-598: native-evaluability-snapshot.json is a faithful capture — ${triage.corpusSize} rules.`); +} else { + fs.writeFileSync(OUT, json); + console.log( + `${changed ? 'Recaptured' : 'Unchanged'} ${path.relative(process.cwd(), OUT)} — ` + + `${triage.corpusSize} rules: ${KNOWN_CLASSES.filter((c) => triage.counts[c] !== undefined).map((c) => `${c} ${triage.counts[c]}`).join(', ')}`, + ); + if (changed) console.log('Now rebuild the mapping: node src/rulesets/standards/build-iso-5055-mapping.mjs'); +} diff --git a/src/rulesets/standards/iso-5055-mapping.csv b/src/rulesets/standards/iso-5055-mapping.csv index 8f3395975..9a122a84b 100644 --- a/src/rulesets/standards/iso-5055-mapping.csv +++ b/src/rulesets/standards/iso-5055-mapping.csv @@ -5,11 +5,11 @@ ACL-03,acl/anti-corruption-layer.rules.json,architecture-decision,,,none,no,nati ACL-04,acl/anti-corruption-layer.rules.json,architecture-decision,,,none,no,native-handler ACL-05,acl/anti-corruption-layer.rules.json,architecture-decision,,,none,no,native-handler ACL-06,acl/anti-corruption-layer.rules.json,architecture-decision,,,none,no,native-handler -HXA-01,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054,Maintainability,partial,yes,unimplemented-native -HXA-02,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054,Maintainability,partial,yes,unimplemented-native +HXA-01,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054,Maintainability,partial,yes,native-handler +HXA-02,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054,Maintainability,partial,yes,native-handler HXA-03,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054,Maintainability,partial,yes,unimplemented-native -HXA-04,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054 CWE-1047,Maintainability,direct,yes,unimplemented-native -HXA-05,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,,,none,partial,unimplemented-native +HXA-04,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,CWE-1054 CWE-1047,Maintainability,direct,yes,native-handler +HXA-05,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,,,none,partial,native-handler HXA-06,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,,,none,no,unimplemented-native HXA-07,adr/adr-0002-hexagonal-architecture.rules.json,code-structure,,,none,no,needs-runtime CICD-01,adr/adr-0005-cicd-quality-gates.rules.json,process,,,none,partial,unimplemented-native @@ -23,7 +23,7 @@ MTN-01,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,CWE-1057,Perf MTN-02,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,CWE-1057,Performance Efficiency Security,partial,no,needs-external-system MTN-03,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,unimplemented-native MTN-04,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,CWE-732,Security,partial,no,needs-runtime -MTN-05,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,unimplemented-native +MTN-05,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,native-handler MTN-06,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,needs-runtime MTN-07,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,unimplemented-native MTN-08,adr/adr-0010-multi-tenancy.rules.json,architecture-decision,,,none,no,unimplemented-native @@ -56,7 +56,7 @@ GIT-04,adr/adr-0050-gitflow-branching.rules.json,process,,,none,yes,needs-extern GIT-05,adr/adr-0050-gitflow-branching.rules.json,process,,,none,no,needs-external-system GIT-06,adr/adr-0050-gitflow-branching.rules.json,process,,,none,no,needs-external-system GIT-07,adr/adr-0050-gitflow-branching.rules.json,process,,,none,no,needs-external-system -GIT-08,adr/adr-0050-gitflow-branching.rules.json,process,,,none,yes,unimplemented-native +GIT-08,adr/adr-0050-gitflow-branching.rules.json,process,,,none,yes,native-handler GIT-09,adr/adr-0050-gitflow-branching.rules.json,process,,,none,no,needs-external-system GIT-10,adr/adr-0050-gitflow-branching.rules.json,process,,,none,no,needs-external-system CORE-0001-01,adr/generated/adr-0001-monorepo-orchestration-principle.rules.json,architecture-decision,,,none,no,documentation-only @@ -146,13 +146,13 @@ CORE-0113-01,adr/generated/adr-0113-node-js-platform-lighthouse-apache-2-0-as-th CORE-0115-01,adr/generated/adr-0115-emergent-knowledge-axis-knowledge-originated-by-applying-the.rules.json,architecture-decision,,,none,no,documentation-only CORE-0116-01,adr/generated/adr-0116-canonical-finding-contract-and-an-executable-advisory-author.rules.json,architecture-decision,,,none,no,documentation-only CORE-0117-01,adr/generated/adr-0117-bilingual-parity-applies-to-authored-sources-not-generated-p.rules.json,architecture-decision,,,none,no,documentation-only -CORE-0118-01,adr/generated/adr-0118-the-core-hub-has-its-own-root-taxonomy-distinct-from-satelli.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0119-01,adr/generated/adr-0119-api-security-configuration-hardening.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0120-01,adr/generated/adr-0120-ssrf-prevention-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0121-01,adr/generated/adr-0121-input-validation-and-sanitization-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0122-01,adr/generated/adr-0122-shell-execution-safety-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0123-01,adr/generated/adr-0123-timing-safe-comparison-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot -CORE-0124-01,adr/generated/adr-0124-credential-and-secret-management-standard.rules.json,architecture-decision,,,none,no,not-in-snapshot +CORE-0118-01,adr/generated/adr-0118-the-core-hub-has-its-own-root-taxonomy-distinct-from-satelli.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0119-01,adr/generated/adr-0119-api-security-configuration-hardening.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0120-01,adr/generated/adr-0120-ssrf-prevention-standard.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0121-01,adr/generated/adr-0121-input-validation-and-sanitization-standard.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0122-01,adr/generated/adr-0122-shell-execution-safety-standard.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0123-01,adr/generated/adr-0123-timing-safe-comparison-standard.rules.json,architecture-decision,,,none,no,documentation-only +CORE-0124-01,adr/generated/adr-0124-credential-and-secret-management-standard.rules.json,architecture-decision,,,none,no,documentation-only AI-0001-01,adr/generated/adr-ai-augmented-0001-harness-engineering-for-ai-augmented-development.rules.json,architecture-decision,,,none,no,documentation-only AI-0002-01,adr/generated/adr-ai-augmented-0002-mcp-integration-protocol-for-agent-tool-invocation.rules.json,architecture-decision,,,none,no,documentation-only AI-0003-01,adr/generated/adr-ai-augmented-0003-model-selection-governance-for-ai-augmented-workflows.rules.json,architecture-decision,,,none,no,documentation-only @@ -196,9 +196,9 @@ CLI-PAR-01,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-n CLI-PAR-02,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-native CLI-PAR-03,cli/core-parity.rules.json,platform-surface,,,none,no,needs-runtime CLI-PAR-04,cli/core-parity.rules.json,platform-surface,,,none,no,unimplemented-native -CLI-EXIT-01,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot -CLI-EXIT-02,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot -CLI-EXIT-03,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,not-in-snapshot +CLI-EXIT-01,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,native-handler +CLI-EXIT-02,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,native-handler +CLI-EXIT-03,cli/exit-code-taxonomy.rules.json,platform-surface,,,none,no,native-handler CLI-RR-01,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler CLI-RR-02,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler CLI-RR-03,cli/release-readiness.rules.json,platform-surface,,,none,no,native-handler @@ -319,8 +319,8 @@ SEC-INJ-01,security/injection-prevention.rules.json,code-structure,CWE-78 CWE-77 SEC-INJ-02,security/injection-prevention.rules.json,code-structure,CWE-77 CWE-88,Security,partial,partial,unimplemented-native SEC-PATH-01,security/path-containment.rules.json,code-structure,CWE-22 CWE-23 CWE-36,Security,direct,yes,unimplemented-native SEC-PATH-02,security/path-containment.rules.json,code-structure,CWE-22,Security,direct,yes,unimplemented-native -SEC-RL-01,security/rate-limiting.rules.json,code-structure,,,none,partial,unimplemented-native -SEC-RL-02,security/rate-limiting.rules.json,code-structure,CWE-789,Security,partial,partial,unimplemented-native +SEC-RL-01,security/rate-limiting.rules.json,code-structure,,,none,partial,native-handler +SEC-RL-02,security/rate-limiting.rules.json,code-structure,CWE-789,Security,partial,partial,native-handler SEC-RL-03,security/rate-limiting.rules.json,code-structure,,,none,no,unimplemented-native SEC-TIMING-01,security/timing-safe-comparison.rules.json,code-structure,,,none,yes,unimplemented-native SEC-TIMING-02,security/timing-safe-comparison.rules.json,code-structure,,,none,partial,unimplemented-native @@ -337,7 +337,7 @@ DAM-R01,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,native-hand DAM-R02,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,native-handler DAM-R03,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,native-handler DAM-R04,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,needs-external-system -DAM-R05,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,unimplemented-native +DAM-R05,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,native-handler DAM-R06,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,needs-external-system DAM-R07,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,native-handler DAM-R08,topologies/data-mesh/data-mesh.rules.json,topology,,,none,no,unimplemented-native @@ -350,9 +350,9 @@ EC-SEC-02,topologies/edge-computing/edge-computing.rules.json,topology,,,none,no ED-R01,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler ED-R02,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler ED-R03,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler -ED-R04,topologies/event-driven/event-driven.rules.json,topology,,,none,no,unimplemented-native -ED-R05,topologies/event-driven/event-driven.rules.json,topology,,,none,no,unimplemented-native -ED-R06,topologies/event-driven/event-driven.rules.json,topology,,,none,no,unimplemented-native +ED-R04,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler +ED-R05,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler +ED-R06,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler ED-R07,topologies/event-driven/event-driven.rules.json,topology,,,none,no,unimplemented-native ED-R08,topologies/event-driven/event-driven.rules.json,topology,,,none,no,native-handler ED-R09,topologies/event-driven/event-driven.rules.json,topology,,,none,no,unimplemented-native diff --git a/src/rulesets/standards/iso-5055-mapping.json b/src/rulesets/standards/iso-5055-mapping.json index e2ea3623e..2110de05b 100644 --- a/src/rulesets/standards/iso-5055-mapping.json +++ b/src/rulesets/standards/iso-5055-mapping.json @@ -83,43 +83,37 @@ "handlerBacklog": { "source": "native-evaluability-snapshot.json", "sourceAuthority": [ + "src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, real handler set, classification)", "src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)", "src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)", "src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)", - "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (corpus loader and pinned counts)" + "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (the pinned counts, asserted against this file)" ], "note": "The handler backlog is the `unimplemented-native` class only. `documentation-only` and `underspecified` rules are not handler work at all, and the two adapter classes are closed by the enforcer seam rather than by a rule handler.", - "realBacklogSize": 60, - "adoptableFromAnalyser": 9, - "adoptableFromAnalyserIncludingPartial": 17, - "adoptedFractionOfBacklog": 0.15, - "adoptedFractionOfBacklogIncludingPartial": 0.2833, - "remainderToAuthor": 43, + "realBacklogSize": 48, + "adoptableFromAnalyser": 5, + "adoptableFromAnalyserIncludingPartial": 10, + "adoptedFractionOfBacklog": 0.1042, + "adoptedFractionOfBacklogIncludingPartial": 0.2083, + "remainderToAuthor": 38, "byEvaluabilityClass": { "unimplemented-native": { - "rules": 60, - "mappedToIso5055": 10, - "analyserAdoptable": 9, - "analyserAdoptablePartial": 8, + "rules": 48, + "mappedToIso5055": 6, + "analyserAdoptable": 5, + "analyserAdoptablePartial": 5, "adoptableRuleIds": [ - "HXA-01", - "HXA-02", "HXA-03", - "HXA-04", - "GIT-08", "SEC-INJ-01", "SEC-PATH-01", "SEC-PATH-02", "SEC-TIMING-01" ], "adoptablePartialRuleIds": [ - "HXA-05", "CICD-01", "CICD-02", "MTN-01", "SEC-INJ-02", - "SEC-RL-01", - "SEC-RL-02", "SEC-TIMING-02" ] }, @@ -144,7 +138,7 @@ "adoptablePartialRuleIds": [] }, "documentation-only": { - "rules": 129, + "rules": 136, "mappedToIso5055": 9, "analyserAdoptable": 6, "analyserAdoptablePartial": 7, @@ -175,11 +169,15 @@ "adoptablePartialRuleIds": [] }, "native-handler": { - "rules": 139, - "mappedToIso5055": 15, - "analyserAdoptable": 24, - "analyserAdoptablePartial": 8, + "rules": 154, + "mappedToIso5055": 19, + "analyserAdoptable": 28, + "analyserAdoptablePartial": 11, "adoptableRuleIds": [ + "HXA-01", + "HXA-02", + "HXA-04", + "GIT-08", "DOD-02", "DOD-09", "EM-D-01", @@ -206,18 +204,21 @@ "MM-R10" ], "adoptablePartialRuleIds": [ + "HXA-05", "EM-D-02", "EM-K-02", "EM-S-03", "EM-S-04", "QT-08", + "SEC-RL-01", + "SEC-RL-02", "AAI-R09", "MM-R12", "SV-R02" ] }, "not-in-snapshot": { - "rules": 12, + "rules": 2, "mappedToIso5055": 0, "analyserAdoptable": 0, "analyserAdoptablePartial": 0, @@ -367,7 +368,7 @@ "eslint-plugin-boundaries" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "Framework imports in the domain layer are a layering violation; 5055 models layering as the layer-skipping call." }, { @@ -395,7 +396,7 @@ "ArchUnit" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "Structural rule with no counterpart among the 138 weaknesses." }, { @@ -453,7 +454,7 @@ "ArchUnit layeredArchitecture()" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "Dependency direction is exactly what CWE-1054 and CWE-1047 measure." }, { @@ -474,7 +475,7 @@ "ESLint no-restricted-imports" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "AOP placement is an Evolith convention; import bans approximate it." }, { @@ -770,7 +771,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -1403,7 +1404,7 @@ "commitlint @commitlint/config-conventional" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "A property of the development process (pipeline, branch, review, test strategy), which is outside the scope of a source-code measure." }, { @@ -3201,7 +3202,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3220,7 +3221,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3239,7 +3240,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3258,7 +3259,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3277,7 +3278,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3296,7 +3297,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -3315,7 +3316,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "documentation-only", "note": "Conformance to a recorded architecture decision. ISO/IEC 5055 measures source structure, not whether a decision was honoured." }, { @@ -4166,7 +4167,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "native-handler", "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." }, { @@ -4185,7 +4186,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "native-handler", "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." }, { @@ -4204,7 +4205,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "not-in-snapshot", + "nativeEvaluability": "native-handler", "note": "A CLI / MCP surface contract. Specific to this product; no standards analyser models it." }, { @@ -6715,7 +6716,7 @@ "Semgrep express-rate-limit presence rules" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "CWE-770 is not in the 138; presence of a limiter is checkable, correct configuration is not." }, { @@ -6742,7 +6743,7 @@ "Semgrep express-body-parser-limit" ] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "Unbounded request bodies are the uncontrolled-allocation weakness reached through a framework default." }, { @@ -7079,7 +7080,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "A contract of a named architecture topology. The predicate is about declared topology configuration, not about code structure." }, { @@ -7326,7 +7327,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "A contract of a named architecture topology. The predicate is about declared topology configuration, not about code structure." }, { @@ -7345,7 +7346,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "A contract of a named architecture topology. The predicate is about declared topology configuration, not about code structure." }, { @@ -7364,7 +7365,7 @@ "adoptable": "no", "examples": [] }, - "nativeEvaluability": "unimplemented-native", + "nativeEvaluability": "native-handler", "note": "A contract of a named architecture topology. The predicate is about declared topology configuration, not about code structure." }, { diff --git a/src/rulesets/standards/iso-5055-mapping.test.mjs b/src/rulesets/standards/iso-5055-mapping.test.mjs index 0d74f98ab..7edc334b7 100644 --- a/src/rulesets/standards/iso-5055-mapping.test.mjs +++ b/src/rulesets/standards/iso-5055-mapping.test.mjs @@ -7,13 +7,21 @@ * * These assertions are the reason the mapping is worth anything. A mapping table * that silently stops covering the corpus is worse than no table: it reports a - * shrinking denominator as progress. Four things are pinned here — + * shrinking denominator as progress. Six things are pinned here — * * 1. the weakness index really is the 138 of ISO/IEC 5055, per measure; * 2. every CWE the mapping cites is one of those 138; * 3. every rule in the corpus has a row (add a rule, this fails); - * 4. nothing in this directory quotes the retired 2011 eight-characteristic + * 4. the evaluability snapshot still agrees with the counts Core pins, read + * OUT OF CORE rather than retyped here (see `pinnedByCore`); + * 5. the snapshot's counts, its per-rule classes and the classes stamped into + * the mapping are three views of one capture, and they agree; + * 6. nothing in this directory quotes the retired 2011 eight-characteristic * ISO/IEC 25010 model. + * + * This job runs without node_modules, so nothing here recomputes the triage. The + * recomputing guard is `rule-corpus-triage.spec.ts` in core-domain, which + * compares the committed snapshot against a fresh classification of the corpus. */ import test from 'node:test'; @@ -103,18 +111,115 @@ test('the handler backlog is scoped to the unimplemented-native class, not to th assert.equal(backlog.byEvaluabilityClass['unimplemented-native'].adoptableRuleIds.length, backlog.adoptableFromAnalyser); }); +/** + * Read the class counts Core pins, from Core. + * + * This used to be six numbers typed into this file — the same six that are + * literals inside the snapshot it was checking. It compared the snapshot to + * itself, so it passed while `documentation-only` sat at 129 and Core had long + * since moved to 136, and it passed again when twelve rules got handlers and the + * snapshot went on calling them `unimplemented-native`. A guard whose expected + * value is a copy of its actual value is not a guard. + * + * This job has no node_modules, so the counts cannot be recomputed here — they + * are read out of the spec that pins them. Parsing a source file is a real + * coupling and it is stated rather than hidden: if the const is renamed or + * reshaped, this THROWS. Failing to find the source of truth must never read as + * agreement with it. + * + * The recomputation-based check lives where the dependencies do: the same spec + * asserts the committed snapshot against a freshly computed triage, and + * `capture-native-evaluability-snapshot.mjs --check` re-derives it end to end. + */ +function pinnedByCore() { + const spec = path.resolve( + RULESETS, + '..', + 'packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts', + ); + if (!fs.existsSync(spec)) { + throw new Error(`Core's pinned counts are unreadable: ${spec} does not exist. The snapshot cannot be verified.`); + } + + const source = fs.readFileSync(spec, 'utf8'); + const literal = /const\s+PINNED_CLASS_COUNTS\s*:[^=]*=\s*\{([^}]*)\}\s*;/.exec(source); + if (!literal) { + throw new Error( + `Could not find the PINNED_CLASS_COUNTS literal in ${path.basename(spec)}. ` + + 'It was renamed or reshaped — update this parser rather than deleting the check.', + ); + } + + const counts = {}; + for (const line of literal[1].split('\n')) { + const entry = /^\s*'?([a-zA-Z-]+)'?\s*:\s*(\d+)\s*,?\s*$/.exec(line); + if (entry) counts[entry[1]] = Number(entry[2]); + } + if (Object.keys(counts).length === 0) { + throw new Error(`Parsed the PINNED_CLASS_COUNTS block in ${path.basename(spec)} and read zero counts out of it.`); + } + return counts; +} + test('the evaluability snapshot still matches the class counts pinned by Core', () => { - // These six numbers are pinned in rule-corpus-triage.spec.ts. If Core moves - // them, this snapshot is stale and the backlog arithmetic above is wrong. - assert.deepEqual(evaluability.counts, { - 'native-handler': 139, - 'documentation-only': 129, - 'unimplemented-native': 60, - 'needs-external-system': 20, - 'needs-runtime': 17, - underspecified: 14, - }); - assert.equal(Object.keys(evaluability.classes).length, 379); + assert.deepEqual( + evaluability.counts, + pinnedByCore(), + 'native-evaluability-snapshot.json disagrees with the counts pinned in rule-corpus-triage.spec.ts. ' + + 'Recapture it (node src/rulesets/standards/capture-native-evaluability-snapshot.mjs), THEN rebuild the ' + + 'mapping (node src/rulesets/standards/build-iso-5055-mapping.mjs) — the generator stamps ' + + 'nativeEvaluability per rule from the snapshot, so rebuilding first launders the stale class into the mapping.', + ); +}); + +test('the snapshot is internally consistent — its header describes its own body', () => { + // The counts and the per-rule classes are two views of one capture, and the + // mapping is stamped from the SECOND one. A header that agrees with Core while + // the body has drifted would put the wrong class on every affected row. + const summed = Object.values(evaluability.counts).reduce((a, b) => a + b, 0); + assert.equal(summed, evaluability.corpusSize, 'the class counts do not sum to the corpus the capture measured'); + assert.equal(Object.keys(evaluability.classes).length, evaluability.distinctRuleIds); + assert.ok( + evaluability.distinctRuleIds <= evaluability.corpusSize, + 'more distinct rule ids than rules classified — the capture is not a capture', + ); + + const tally = {}; + for (const klass of Object.values(evaluability.classes)) tally[klass] = (tally[klass] ?? 0) + 1; + + if (evaluability.distinctRuleIds === evaluability.corpusSize) { + // The usual case: one row per rule id, so the two views must agree exactly. + assert.deepEqual(tally, evaluability.counts, 'the per-rule classes do not add up to the stated counts'); + } else { + // A rule id carried by two ruleset files collapses to one key here, so the + // tally is a lower bound rather than an equality. + for (const [klass, n] of Object.entries(tally)) { + assert.ok(n <= evaluability.counts[klass], `${klass}: ${n} classes recorded but only ${evaluability.counts[klass]} counted`); + } + } +}); + +test('every rule the mapping stamps a class onto is a rule the snapshot actually classified', () => { + // `not-in-snapshot` is a legitimate value for the two single-rule + // infrastructure files whose rule metadata sits at the document root, which + // the Core corpus loader does not read. It is NOT a legitimate value for + // anything else: when the snapshot falls behind the corpus, rows quietly + // acquire it and drop out of the backlog. So the exception is enumerated. + const orphans = mapping.rules.filter((r) => r.nativeEvaluability === 'not-in-snapshot').map((r) => r.ruleId).sort(); + assert.deepEqual( + orphans, + ['INFRA-001', 'INFRA-OPA-001'], + `mapping rows carry no class from the snapshot: ${orphans.join(', ')}. Either the snapshot is stale ` + + '(recapture it) or the corpus grew another rule shape Core does not load (then say so here).', + ); + for (const r of mapping.rules) { + if (r.nativeEvaluability === 'not-in-snapshot') continue; + assert.equal( + r.nativeEvaluability, + evaluability.classes[r.ruleId], + `${r.ruleId} is stamped ${r.nativeEvaluability} in the mapping but ${evaluability.classes[r.ruleId]} in the snapshot`, + ); + } }); test('no artifact here quotes the retired 2011 eight-characteristic ISO/IEC 25010 model', () => { diff --git a/src/rulesets/standards/native-evaluability-snapshot.json b/src/rulesets/standards/native-evaluability-snapshot.json index 6f9fd28c1..2311abaff 100644 --- a/src/rulesets/standards/native-evaluability-snapshot.json +++ b/src/rulesets/standards/native-evaluability-snapshot.json @@ -2,21 +2,25 @@ "$id": "https://evolith.dev/rulesets/standards/native-evaluability-snapshot.json", "title": "Native-engine evaluability class per rule (snapshot)", "description": "Per-rule evaluability class as computed by the Core native evaluator triage. This is a CAPTURED SNAPSHOT, not the source of truth: the authority is src/packages/core-domain/src/application/validators/rule-evaluability.ts and its pinned spec rule-corpus-triage.spec.ts. It is recorded here so the ISO/IEC 5055 mapping can be scoped to the real handler backlog without src/rulesets depending on a package it does not own.", - "version": "1.0.0", - "capturedOn": "2026-07-28", + "version": "1.1.0", + "generatedBy": "src/rulesets/standards/capture-native-evaluability-snapshot.mjs", + "capturedOn": "2026-07-29", "capturedFrom": [ + "src/packages/core-domain/test/rule-corpus-triage.ts (corpus loader, real handler set, classification)", "src/packages/core-domain/src/application/validators/rule-evaluability.ts (RULE_TRIAGE, classifyRule, ADR_CONFORMANCE_CATEGORY)", "src/packages/core-domain/src/application/validators/evaluators/native-evaluator.ts (registered handler set)", "src/packages/core-domain/src/application/validators/evaluators/handlers/**/*.ts (canHandle predicates)", - "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (corpus loader and pinned counts)" + "src/packages/core-domain/src/application/validators/rule-corpus-triage.spec.ts (the pinned counts, asserted against this file)" ], - "validation": "The capture reproduces every class count pinned by rule-corpus-triage.spec.ts exactly (corpus 379; native-handler 139, documentation-only 129, unimplemented-native 60, needs-external-system 20, needs-runtime 17, underspecified 14). If those pinned numbers change, this snapshot is stale and must be recaptured.", + "validation": "Captured from a live run of the Core triage over 389 rules (native-handler 154, unimplemented-native 48, needs-runtime 17, needs-external-system 20, documentation-only 136, underspecified 14). Recapture with `node src/rulesets/standards/capture-native-evaluability-snapshot.mjs`, then rebuild the mapping with `node src/rulesets/standards/build-iso-5055-mapping.mjs` — the mapping stamps nativeEvaluability per rule from this file, so rebuilding first would launder a stale class into it. Drift is caught in two places: rule-corpus-triage.spec.ts compares this file against a fresh triage (core-domain jest), and iso-5055-mapping.test.mjs checks it against the counts pinned in that spec (dependency-free documentation job).", + "corpusSize": 389, + "distinctRuleIds": 389, "counts": { - "native-handler": 139, - "unimplemented-native": 60, + "native-handler": 154, + "unimplemented-native": 48, "needs-runtime": 17, "needs-external-system": 20, - "documentation-only": 129, + "documentation-only": 136, "underspecified": 14 }, "classes": { @@ -26,11 +30,11 @@ "ACL-04": "native-handler", "ACL-05": "native-handler", "ACL-06": "native-handler", - "HXA-01": "unimplemented-native", - "HXA-02": "unimplemented-native", + "HXA-01": "native-handler", + "HXA-02": "native-handler", "HXA-03": "unimplemented-native", - "HXA-04": "unimplemented-native", - "HXA-05": "unimplemented-native", + "HXA-04": "native-handler", + "HXA-05": "native-handler", "HXA-06": "unimplemented-native", "HXA-07": "needs-runtime", "CICD-01": "unimplemented-native", @@ -44,7 +48,7 @@ "MTN-02": "needs-external-system", "MTN-03": "unimplemented-native", "MTN-04": "needs-runtime", - "MTN-05": "unimplemented-native", + "MTN-05": "native-handler", "MTN-06": "needs-runtime", "MTN-07": "unimplemented-native", "MTN-08": "unimplemented-native", @@ -77,7 +81,7 @@ "GIT-05": "needs-external-system", "GIT-06": "needs-external-system", "GIT-07": "needs-external-system", - "GIT-08": "unimplemented-native", + "GIT-08": "native-handler", "GIT-09": "needs-external-system", "GIT-10": "needs-external-system", "CORE-0001-01": "documentation-only", @@ -167,6 +171,13 @@ "CORE-0115-01": "documentation-only", "CORE-0116-01": "documentation-only", "CORE-0117-01": "documentation-only", + "CORE-0118-01": "documentation-only", + "CORE-0119-01": "documentation-only", + "CORE-0120-01": "documentation-only", + "CORE-0121-01": "documentation-only", + "CORE-0122-01": "documentation-only", + "CORE-0123-01": "documentation-only", + "CORE-0124-01": "documentation-only", "AI-0001-01": "documentation-only", "AI-0002-01": "documentation-only", "AI-0003-01": "documentation-only", @@ -210,6 +221,9 @@ "CLI-PAR-02": "unimplemented-native", "CLI-PAR-03": "needs-runtime", "CLI-PAR-04": "unimplemented-native", + "CLI-EXIT-01": "native-handler", + "CLI-EXIT-02": "native-handler", + "CLI-EXIT-03": "native-handler", "CLI-RR-01": "native-handler", "CLI-RR-02": "native-handler", "CLI-RR-03": "native-handler", @@ -328,8 +342,8 @@ "SEC-INJ-02": "unimplemented-native", "SEC-PATH-01": "unimplemented-native", "SEC-PATH-02": "unimplemented-native", - "SEC-RL-01": "unimplemented-native", - "SEC-RL-02": "unimplemented-native", + "SEC-RL-01": "native-handler", + "SEC-RL-02": "native-handler", "SEC-RL-03": "unimplemented-native", "SEC-TIMING-01": "unimplemented-native", "SEC-TIMING-02": "unimplemented-native", @@ -346,7 +360,7 @@ "DAM-R02": "native-handler", "DAM-R03": "native-handler", "DAM-R04": "needs-external-system", - "DAM-R05": "unimplemented-native", + "DAM-R05": "native-handler", "DAM-R06": "needs-external-system", "DAM-R07": "native-handler", "DAM-R08": "unimplemented-native", @@ -359,9 +373,9 @@ "ED-R01": "native-handler", "ED-R02": "native-handler", "ED-R03": "native-handler", - "ED-R04": "unimplemented-native", - "ED-R05": "unimplemented-native", - "ED-R06": "unimplemented-native", + "ED-R04": "native-handler", + "ED-R05": "native-handler", + "ED-R06": "native-handler", "ED-R07": "unimplemented-native", "ED-R08": "native-handler", "ED-R09": "unimplemented-native", From 0bd58ff9b5f1b56c720ad942a08f5adddfca14a4 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 18:11:06 -0500 Subject: [PATCH 08/27] fix(evidence): qualify five recorded test commands with the path they actually live at MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dead-reference ratchet was at 309 against a budget of 305, so `Governance guards (GT-578)` has been red on main. Five of those dead references are one transcription slip repeated: the closure records for GT-424, GT-621, GT-629, GT-630 and GT-632 recorded their self-test as node --test 34-check-skill-registry-parity.test.mjs with no directory. Every one of those files exists at `.harness/scripts/ci/.test.mjs` — the `files` list in the same record names the full path — so the evidence pointed at something real and quoted a command that cannot run. That is exactly what the guard is for, and it is a repair rather than a re-write: the command now names what was actually executed. 309 -> 304, which is under the existing budget, so the job goes green on the repair alone. The budget stays at 305 in this commit ON PURPOSE. The comment above that flag records the rule and the burn behind it: it was once lowered to 303 from the guard's computed basis, the runner then counted 305, and the job went red. Only a count this job PRINTS may lower it. This branch's own run will print one; lowering follows from that number, not from my machine's. Verified: 41-validate-evidence-commands 304 dead / 0 failed; its self-test 34/34; 08-validate-tracking green over 574 closure records; the JSON still parses. Co-Authored-By: Claude Opus 5 --- .../control-center/evidence/gap-closure-evidence.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/core/control-center/evidence/gap-closure-evidence.json b/reference/core/control-center/evidence/gap-closure-evidence.json index dd0560ead..c5a410392 100644 --- a/reference/core/control-center/evidence/gap-closure-evidence.json +++ b/reference/core/control-center/evidence/gap-closure-evidence.json @@ -8484,7 +8484,7 @@ ], "validationCommands": [ "34-check-skill-registry-parity -> exit 0; 9/9 executable capabilities routable (4 hand-bound, 5 synthesized), reported separately.", - "node --test 34-check-skill-registry-parity.test.mjs -> 15/15, including three negative fixtures: derivation absent, derivation uncalled, and a synthesis narrowed by one extra conjunct.", + "node --test .harness/scripts/ci/34-check-skill-registry-parity.test.mjs -> 15/15, including three negative fixtures: derivation absent, derivation uncalled, and a synthesis narrowed by one extra conjunct.", "Wired into the governance-guards job in ci-cd.yml, so it is no longer inert." ], "closedAt": "2026-07-28", @@ -8619,7 +8619,7 @@ "validationCommands": [ "45-validate-port-inventory-honesty --verbose -> 19 ports declared, 11 on the hot path (7 required + 4 optional), 8 declared-not-reached, 53 adapters.", "Before the fix the guard exits 1 against master-view.svg: \"publishes a port/adapter count without saying how many are actually reached\".", - "node --test 45-validate-port-inventory-honesty.test.mjs -> 9/9, including four anti-vacuous fixtures (zero ports, unparseable deps, zero adapters, missing scan corpus).", + "node --test .harness/scripts/ci/45-validate-port-inventory-honesty.test.mjs -> 9/9, including four anti-vacuous fixtures (zero ports, unparseable deps, zero adapters, missing scan corpus).", "Row evidence corrected: 17/49/9 as registered vs 19/53/11 measured; the diagnostic’s \"two interaction adapters with no callers\" is six." ], "dependencyDisposition": "none" @@ -8650,7 +8650,7 @@ "validationCommands": [ "08-validate-tracking.mjs -> \"Acceptance-criteria audit: 47 non-DONE GT row(s) checked against 605/605 EN/ES catalog sections — 0 with no criteria at all.\" exit 0.", "It exited 1 on its first run against the real board, naming GT-435, GT-444 and GT-448; criteria were written for all three rather than exempting them.", - "node --test 08-validate-tracking.test.mjs -> 17/17, including the negative fixture and the vacuity floor." + "node --test .harness/scripts/ci/08-validate-tracking.test.mjs -> 17/17, including the negative fixture and the vacuity floor." ], "closedAt": "2026-07-28", "closureCommit": "c438b801", @@ -8719,7 +8719,7 @@ "validationCommands": [ "47-validate-joined-paths --verbose -> 615 files, 542 path.join calls, 89 repo-rooted, 65 resolvable, 4 fallback chains, 1 prohibition, exit 0.", "First run reported 19 broken paths; only 3 were defects. The other 16 were false positives whose 'fix' would have broken working code: 6 on a base naming the customer workspace, 9 members of fallback chains, 1 prohibition whose absence IS the passing state.", - "node --test 47-validate-joined-paths.test.mjs -> 11/11, covering both anti-vacuous floors, chain and ternary grouping, variable segments, and non-root bases.", + "node --test .harness/scripts/ci/47-validate-joined-paths.test.mjs -> 11/11, covering both anti-vacuous floors, chain and ternary grouping, variable segments, and non-root bases.", "43-validate-guard-negative-fixtures OBSERVED it red: 36 of 36 guards turned red against the empty fixture. 42 reports 11 self-guarded, 39 scanning, 0 PENDING.", "core-domain 1367/1367, mcp-server 434/434, core-api 179/179, CLI unit 1436/1436, all four tsc projects clean.", "Wired into docs.yml, which backs the required Validate documentation check.", @@ -8739,7 +8739,7 @@ ], "validationCommands": [ "46-validate-derived-artifact-order --verbose -> 2 links, 3 artifacts, current AND at a fixed point.", - "node --test 46-validate-derived-artifact-order.test.mjs -> 8/8, including a test proving the guard leaves the real tree byte-identical after replaying in place.", + "node --test .harness/scripts/ci/46-validate-derived-artifact-order.test.mjs -> 8/8, including a test proving the guard leaves the real tree byte-identical after replaying in place.", "Wired into docs.yml, which backs the required Validate documentation check.", "Registered in guard-classification.mjs: 42 reports 58 guards classified, 43 reports 35 observed failing." ], From 89366b42a0d445127594ea406608a04de6bd8380 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 18:35:35 -0500 Subject: [PATCH 09/27] chore(ci): ratchet the dead-reference budget 305 -> 304, from the count this job printed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The five repaired closure records took the count to 304, and the budget stayed at 305 in that commit on purpose: the comment above the flag records that it was once lowered from the guard's computed basis, the runner then disagreed, and the job went red. The rule it draws is that only a count THIS JOB prints may lower it. Run 30498743635 printed `304 dead (budget 305)`. That is the authorisation, so the budget moves now and not before. My local count also said 304 — worth noting and explicitly NOT the reason: the agreement is a coincidence of this checkout, and the printed line is the evidence. Leaving the slack would have meant one more dead reference could land silently, which is the whole failure mode a ratchet exists to prevent. Verified locally at the new setting: `--strict --max-dead 304` exits 0 with `304 dead (budget 304)`. Co-Authored-By: Claude Opus 5 --- .github/workflows/ci-cd.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci-cd.yml b/.github/workflows/ci-cd.yml index 72e4f860a..09ea279f4 100644 --- a/.github/workflows/ci-cd.yml +++ b/.github/workflows/ci-cd.yml @@ -459,7 +459,7 @@ jobs: - name: Execute the board's recorded validationCommands (report mode) run: node .harness/scripts/ci/41-validate-evidence-commands.mjs --execute --verbose - # Ratchet, not a cliff. The budget is 305, which is what THIS RUNNER counts. + # Ratchet, not a cliff. The budget is 304, which is what THIS RUNNER counts. # # It was briefly lowered to 303 on the strength of the guard's computed # "ratchet basis" — 288 dead locally plus 15 that resolve only through @@ -470,10 +470,20 @@ jobs: # first place, and I broke it by trusting a derived figure over a measured # one. # + # 305 -> 304 obeying that rule rather than working around it. Five closure + # records (GT-424/621/629/630/632) quoted their self-test as + # `node --test 34-check-skill-registry-parity.test.mjs`, with no directory, + # while naming the real `.harness/scripts/ci/` path in the same record's + # `files` list — evidence pointing at something real through a command that + # cannot run. Repairing the five took the count to 304, and THIS JOB printed + # `304 dead (budget 305)` in run 30498743635 before the budget moved. The + # local count agreed, and that agreement is not what authorises this: the + # printed line is. + # # Lower it as records are repaired, from a number this job printed; at 0, # delete the flag and this becomes the strict gate. - name: Dead-reference ratchet (must not grow) - run: node .harness/scripts/ci/41-validate-evidence-commands.mjs --strict --max-dead 305 + run: node .harness/scripts/ci/41-validate-evidence-commands.mjs --strict --max-dead 304 # RETIRED 2026-07-27 (GT-570): the `publish-npm` job lived here and published # ONLY `src/sdk/cli`, with `--tag beta` and no provenance, writing the token From f34a10f1e5ad3779919a9944d7615d7f03d06c58 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Wed, 29 Jul 2026 18:51:28 -0500 Subject: [PATCH 10/27] fix(evidence): migrate the recorded closure commands onto the paths this repo has MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 264 of the 304 dead references were one event: the `src/` refactor moved every workspace, and the closure evidence kept quoting the pre-refactor layout. It is the largest single block of dead evidence in the board and it had never been migrated, so `41-validate-evidence-commands` has been carrying it as permanent budget. Rewritten only where the SAME artifact moved or was renamed, and only where the new target was verified to exist in this checkout before a byte was written: • workspaces gained the `src/` prefix — sdk/cli, apps/core-api and the six packages (174 references); • the retired `@evolith/*` scope became `@beyondnet/evolith-*`, with `@evolith/core-api` becoming the bare `core-api` its package.json declares and `@evolith/smart-cli` becoming `@beyondnet/evolith-cli` (29); • `.harness/scripts/{validate-docs,check-bilingual-parity}.mjs` moved into `ci/` with numeric prefixes; `review-input.test.mjs` into `ci/agentic/` (41); • `reference/operations/` -> `product/operations/`, `reference/infrastructure/docker-compose.yml` -> `product/infra/`, `.bmad-core/agents/` -> `reference/core/foundations/agent-skills/`, and the gap catalog into `gaps/` (18); • source files quoted as targets gained the same `src/` prefix (19). A further 27 read `npm run --workspace