From fbbfba9e99986c10ba25a362fb0b57523854d6ae Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 16:41:50 +0100 Subject: [PATCH 01/12] docs(agent-loop): plan backend CI acceleration --- .../CHUNK_MAP.md | 10 ++ .../DECISIONS.md | 30 ++++ .../DISCOVERY.md | 115 +++++++++++++ .../INTENT.md | 109 ++++++++++++ .../WS-CI-001-backend-ci-acceleration/PLAN.md | 147 ++++++++++++++++ .../RISKS.md | 17 ++ .../STATUS.md | 9 + ...-CI-001-01-parallel-full-suite-coverage.md | 157 ++++++++++++++++++ .../WS-CI-001-02-safe-routing-cache-timing.md | 70 ++++++++ ...CI-001-01-preimplementation-plan-review.md | 61 +++++++ 10 files changed, 725 insertions(+) create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-02-safe-routing-cache-timing.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-preimplementation-plan-review.md diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md new file mode 100644 index 00000000..6c87aa5b --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md @@ -0,0 +1,10 @@ +# CHUNK MAP: WS-CI-001 + +| Chunk | Title | Risk | Status | +|---|---|---:|---| +| `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Plan reviewed; awaiting human implementation approval | +| `WS-CI-001-02` | Safe Routing, Cache, and Timing Refinement | L1 | Future; not started and requires separate approval | + +Each chunk maps to one PR. Chunk 01 preserves the full suite and every coverage +gate. Chunk 02 may be planned only from measured 01 evidence and cannot be named +as 01's successor unless its reviewed contract is complete. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md new file mode 100644 index 00000000..fb620443 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md @@ -0,0 +1,30 @@ +# DECISIONS: WS-CI-001 - Backend CI Acceleration + +## 2026-07-20 - Prioritize CI acceleration before explicit-start automation + +The user selected backend CI efficiency as the next initiative after successful +merge and live verification of `WS-ENG-001-04A`. `WS-ENG-001-04B` remains +stopped and is not implicitly reprioritized by merge memory. + +## 2026-07-20 - Preserve the full suite + +The first implementation will reduce wall-clock time through isolated parallel +execution, not by skipping tests or lowering coverage. + +## 2026-07-20 - Use file-level shards without a new scheduling dependency + +Test modules remain intact and are assigned deterministically from collected +inventory. Shared-database xdist and third-party sharding services are rejected. + +## 2026-07-20 - Keep routing separate + +Path-based workflow routing, caches, and persistent runtime weighting require +separate evidence and approval after parallel execution is proven. + +## 2026-07-20 - Preimplementation review repairs accepted + +Nine reviewer tracks required canonical filesystem and collected-node inventory, +exact observed-node fan-in, coverage-byte hashing, checked-out-tree provenance, +fixed artifact sets, explicit read-only permissions, digest-pinned PostgreSQL, +stable `Backend / test`, canonical operator documentation, and a real local dry +run. The plan and 01 contract now include those boundaries; all tracks pass. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md new file mode 100644 index 00000000..0366c484 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md @@ -0,0 +1,115 @@ +# DISCOVERY: WS-CI-001 - Backend CI Acceleration + +## Observed current behavior + +- `.github/workflows/backend.yml` defines one `test` job with a 240-minute + timeout, PostgreSQL 16, and a separately started pinned MinIO image. +- `Backend full-suite coverage` invokes + `backend/scripts/run_isolated_tests.py` with a 12,600-second child timeout. +- The runner creates a unique PostgreSQL role/database, migrates to the single + Alembic head, buffers child stdout/stderr, redacts database URLs, and cleans up. +- Coverage is produced once and then read by twelve sequential `coverage report` + commands enforcing the repository-wide 78 percent floor and protected 90 + percent subsystem floors. +- `API contract real API e2e` provisions another isolated database after all + coverage reports. +- The workflow runs for every pull request and every push to `main`. + +## Measured evidence + +PR #161 Backend run `29752233451`, job `88385435082`: + +| Step | Duration | +|---|---:| +| Runner/service initialization | 17 seconds | +| Dependency installation | 20 seconds | +| Lint and docstring coverage | 1 second | +| Isolated runner self-test | 17 seconds | +| Backend full-suite coverage | 25 minutes 31 seconds | +| All twelve coverage reports | about 2 seconds | +| API contract E2E | 9 seconds | +| Total job | about 26 minutes 42 seconds | + +Recent successful Backend workflows have ranged from roughly 24 to 52 minutes. +The full-suite process is therefore the dominant wall-clock cost. + +## Repository inventory + +- `backend/pyproject.toml` declares pytest, pytest-asyncio, pytest-cov, coverage, + and Ruff through the existing dev extra; it has no xdist or sharding package. +- `backend/tests/` contains 31 `test_*.py` modules and approximately 999 direct + test functions before parameter expansion. Pytest's collected node-ID set is + the executable inventory and may be larger after parameter expansion. +- The largest modules by source size are `test_auth.py`, `test_alembic.py`, + `test_projects.py`, `test_tasks.py`, and `test_authorization.py`. +- Database-heavy modules construct async engines against the isolated database + URL. Migration and transaction tests make process-level database separation + safer than concurrent workers sharing one database. +- Artifact integration tests use the real MinIO endpoint exposed by the + workflow. + +## Existing tests and gates + +- `backend/tests/test_isolated_database_runner.py` proves owned database and + cleanup behavior. +- `backend/tests/test_coverage_contract.py` protects coverage expectations. +- `scripts/test_agent_gates.py` contains workflow-structure and CI-integrity + assertions and is the natural place for fail-closed workflow tests. +- `.github/workflows/agent-gates.yml` separately validates merge intent, stale + wording, Markdown links, loop state, scope, and internal review evidence. +- Branch protection currently receives the Backend job identity `test`; the + final fan-in must preserve that identity. + +## Dependencies and integrations + +- GitHub Actions hosted Ubuntu runners. +- PostgreSQL 16 service container. +- Pinned MinIO container image. +- GitHub artifact upload/download actions, which must be pinned by immutable + commit SHA if introduced. +- Existing coverage data format from pytest-cov/coverage.py. +- MinIO is digest-pinned, but PostgreSQL currently uses the mutable + `public.ecr.aws/docker/library/postgres:16` tag. Immutable container provenance + is therefore a target for 01, not an existing property. + +## Gaps + +- No deterministic shard planner or checked shard manifest exists. +- No exact-once filesystem-module or collected-node-ID proof exists across jobs. +- No coverage-artifact provenance/fan-in validation exists. +- No live per-shard timing evidence exists. +- No current workflow test proves a failed/cancelled shard makes the final + required job fail. + +## Risks discovered + +- File size is only a weak runtime proxy; collected test count is a better + initial deterministic weight but still cannot perfectly balance slow modules. +- `test_alembic.py` may remain a long-tail shard because modules must initially + remain intact. +- Starting PostgreSQL and installing dependencies per shard increases aggregate + runner consumption while reducing elapsed time. +- Coverage aggregation is trustworthy only if each artifact is bound to the + actual checked-out tree, expected shard ID, complete module/node inventory, + and SHA-256 digest of the exact coverage bytes. +- A final job using unconditional execution must explicitly reject any upstream + non-success status; GitHub's default dependency skipping is not sufficient for + a stable required check. + +## Unknowns to measure during implementation + +- Exact collected node count per module after parameter expansion. +- Actual shard balance on GitHub-hosted runners. +- Whether all shards require MinIO or only the artifact shard. +- Practical concurrency available to the repository/account. +- Aggregate Actions-minute increase for four shards. + +## Conventions to preserve + +- Immutable action pins and the existing digest-pinned MinIO image; 01 must + replace the mutable PostgreSQL tag with a reviewed digest pin. +- `run_isolated_tests.py` as the sole database ownership boundary. +- Existing 78/90 coverage floors. +- Backend required check named `test`. +- Full exact-head hosted proof before merge. +- No implementation until a reviewed chunk contract receives human approval. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md new file mode 100644 index 00000000..d578dd0a --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md @@ -0,0 +1,109 @@ +# INTENT: WS-CI-001 - Backend CI Acceleration + +## Problem being solved + +The required Backend GitHub Actions job runs the complete backend test suite in +one sequential process. PR #161 spent 25 minutes 31 seconds in that single step +and about 27 minutes overall even though it changed no backend product code. + +## Why this work matters + +Every implementation PR must retain full backend and subsystem coverage proof, +but a 25-to-50-minute feedback loop slows review, repair, and safe delivery. The +repository needs shorter wall-clock feedback without weakening its zero-trust +CI evidence. + +## Current behavior + +- One Ubuntu job provisions PostgreSQL and MinIO. +- One isolated database is migrated and used for all 999 collected tests across + 31 test modules. +- The full suite writes one coverage data file. +- Twelve coverage reports and the API contract E2E run only after the full suite. +- The isolated runner buffers child output, so the longest step exposes no live + progress. +- PR #161 spent only 20 seconds installing dependencies; dependency setup is not + the primary bottleneck. + +## Target behavior + +- Every collected backend test still runs on every required Backend workflow. +- Canonical filesystem test modules and collected pytest node IDs are each + assigned and observed exactly once across isolated parallel shards. +- Each shard receives an independently owned migrated PostgreSQL database. +- Coverage artifacts are combined centrally before the unchanged 78 percent + repository floor and 90 percent protected-subsystem floors are enforced. +- The API contract E2E proof runs concurrently with test shards. +- The final required Backend check fails closed on missing, duplicate, failed, + cancelled, malformed, foreign-tree, or byte-tampered shard evidence. +- CI exposes shard progress and timing evidence. + +## Design chosen + +Use deterministic file-level sharding based on the collected test inventory. +Keep each test module intact, distribute modules using collected test counts, +run shards in independent GitHub jobs, upload uniquely named coverage artifacts, +and combine them in one final required `test` job. File-level isolation avoids +introducing unsafe concurrency inside a shared database and requires no new +pytest scheduling dependency. + +## Alternatives considered + +- Skip the Backend workflow for engineering-loop PRs: rejected for the first + chunk because the user requires full-suite proof and path routing can hide an + incorrectly classified backend-affecting change. +- Add `pytest-xdist` inside the existing job: rejected because tests share one + migrated database and many modules exercise transaction and migration state. +- Hash node IDs across shards: rejected because it splits module fixtures and + makes database-sensitive module behavior harder to reason about. +- Preserve one sequential process but add caching: rejected as the primary fix; + installation was 20 seconds while the suite was more than 25 minutes. +- Lower or sample coverage: rejected as CI weakening. + +## Boundaries preserved + +- All tests, migrations, real PostgreSQL behavior, required MinIO behavior, API + contract proof, and coverage thresholds remain mandatory. +- No backend application, schema, migration, authentication, authorization, + artifact, review, contribution, compensation, or payment behavior changes. +- The final branch-protection check identity remains stable. +- No third-party test-sharding service or production secret is introduced. +- Human PR approval and existing Agent Gates remain unchanged. + +## Expected risks + +- A planner may omit or duplicate a test module. +- Coverage artifacts may be mixed across commits or shards. +- One oversized module may dominate a shard and limit speedup. +- GitHub matrix cancellation may accidentally allow the aggregator to pass. +- MinIO-dependent tests may be assigned to jobs without MinIO. +- Coverage path differences may prevent correct combination. +- GitHub concurrency and aggregate runner minutes may increase. + +## What must not change + +- No test may be skipped because of its shard assignment. +- No assertion, coverage floor, required check, or failure propagation may be + weakened. +- Shards must never share a database or database role. +- Coverage from a different commit or incomplete shard set must not validate. +- `WS-ENG-001-04B` remains inactive. + +## How this will be proven + +Unit tests will prove filesystem and node-ID inventory completeness, determinism, +exact-once assignment and observation, stable ordering, malformed input +rejection, and fail-closed coverage-manifest validation. Workflow review will +prove isolated services and databases, immutable action pins, checked-out-tree +binding, byte-hashed artifacts, complete fan-in, unchanged thresholds, stable +required-check identity, and no skipped failure path. The implementation PR must +run the old full suite locally where practical and the new hosted matrix on the +exact checked-out tree; measured wall time will be compared with PR #161's +25-minute-31-second full-suite baseline. + +## Human decisions required + +- Approve implementation of `WS-CI-001-01` after plan review. +- Decide after hosted evidence whether four shards are the correct cost/latency + balance or should be adjusted in a later chunk. +- Separately approve any later path-routing policy; it is not part of 01. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md new file mode 100644 index 00000000..5a9f5124 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md @@ -0,0 +1,147 @@ +# PLAN: WS-CI-001 - Backend CI Acceleration + +## Objective + +Reduce Backend workflow wall-clock time while preserving complete test execution, +real service integration, exact-head provenance, and every current coverage and +branch-protection gate. + +## Proposed approach + +### 1. Deterministic inventory and partitioning + +Add a repository-owned planner with one shared validation implementation. +Canonical module completeness comes from symlink-safe filesystem discovery of +`backend/tests/test_*.py`, with an explicit declaration excluding only +`test_isolated_database_runner.py`. A successful pytest collection supplies the +canonical executable node IDs and per-module weights; every node must map to one +discovered canonical module. Collection errors, zero-test modules, foreign paths, +malformed node IDs, and parameterized-node collisions fail. + +Assign every non-excluded module exactly once using deterministic +largest-weight-first bin packing with lexical tie-breaking. Emit canonical, +schema-versioned JSON binding schema, actual checked-out tree SHA from +`git rev-parse HEAD`, ordered normalized module paths, explicit exclusion, +ordered node IDs, collected counts, shard count, weights, and assignments. Its +SHA-256 digest identifies the exact executable inventory and plan. + +The initial weight is collected test count per module. Runtime telemetry may +replace or augment weights only in the later optional chunk after its provenance +and stability are reviewed. + +### 2. Fast prerequisite job + +Retain evidence, install, lint, docstring, isolated-runner self-test, collection, +and shard-plan validation ahead of the expensive matrix. Upload the immutable +manifest for the current commit. A collection or planning failure prevents all +shards. + +### 3. Isolated parallel shards + +Run four matrix jobs initially. Each validates its actual checked-out tree SHA +against the manifest, provisions its own PostgreSQL service and independently +owned migrated database through `run_isolated_tests.py`, starts MinIO, and runs +only manifest modules through Python argv/subprocess construction rather than +shell-expanded module strings. It writes a unique coverage file, independently +collects the node IDs observed for its assigned modules, and uploads a fixed-name +bundle containing shard ID, tree SHA, manifest digest, observed node IDs/counts, +allowlisted non-secret metadata, coverage, and SHA-256 of the coverage bytes. + +No shard shares a database, role, filesystem artifact, or coverage filename. +No test is selected by an untrusted PR-provided shell fragment. + +### 4. Concurrent API contract proof + +Run the existing real API contract E2E command as a separate dependent job after +fast prerequisites and concurrently with shards. Preserve its isolated database +and timeout behavior. + +### 5. Fail-closed final fan-in + +Keep workflow name `Backend` and final job ID `test`, preserving the existing +`Backend / test` required-check context. Other jobs use different IDs. Run the +final job after every shard and API proof with `if: always()`. It must reject any +failed, cancelled, or skipped prerequisite; missing/extra shard; tree-SHA or +manifest-digest mismatch; malformed metadata; duplicate/omitted module or node; +collection-count mismatch; missing coverage; coverage-digest mismatch; symlink, +path traversal, non-regular file, or unexpected path. It accepts exactly four +fixed artifact names containing tree SHA and shard ID, downloads without wildcard +surplus, and recomputes canonical bindings rather than trusting repeated labels. + +Only after provenance validation does it combine coverage data and enforce the +unchanged repository-wide and twelve protected coverage reports. The final job +uploads concise timing and shard-balance evidence. + +### 6. Evidence and rollout + +Test the planner and fan-in validator locally, including a real repository +collection-to-plan-to-four-shard-selection-to-fan-in dry run. Statically test the +workflow in the agent-gate suite and run the implementation PR on GitHub. Compare hosted +wall time and shard distribution with the PR #161 baseline. Do not claim a +latency target passed until exact hosted evidence exists. + +Matrix job state supplies visible progress; final shard duration/balance metadata +supplies timing evidence. Live pytest output is not required because the +unchanged isolated runner intentionally buffers and redacts child output. + +## Alternatives rejected + +- Test or coverage sampling. +- Path-based Backend skipping in the first chunk. +- Shared-database xdist workers. +- Unpinned marketplace sharding actions. +- Mutable external timing data without commit/inventory provenance. +- Allowing the aggregator to pass when an upstream job is skipped or cancelled. + +## Architecture and security boundaries + +- Planning and execution logic remain CI-only under backend scripts and GitHub + workflows; product modules are not imported as orchestration services. +- Planner input is treated as data and emitted module paths are validated before + command construction. +- Artifacts are accepted only for the actual checked-out tree and expected + canonical manifest digest, and coverage bytes are independently hashed. +- Top-level and job permissions explicitly set `contents: read` plus only minimum + run-scoped artifact access; no repository write exists and + `pull_request_target` is prohibited. +- No database URL/password, MinIO credential, environment dump, or runner + database metadata outside an allowlisted non-secret schema enters artifacts. + +## Verification strategy + +- Planner tests: filesystem/node completeness, parameterized nodes, zero-test and + collection failure, determinism, exact-once assignment, stable ties, + empty/malformed rejection, traversal/symlink rejection, shard bounds, and + canonical manifest-digest binding. +- Fan-in tests: missing, duplicate, extra, foreign-SHA, foreign-digest, malformed, + failed, cancelled, skipped, missing/duplicate/foreign node, count mismatch, + coverage-byte mismatch, symlink, traversal, unexpected-file, and coverage-file + mismatch rejection. +- Workflow tests: immutable action pins, fixed matrix size, isolated database per + job, no shared coverage path, explicit failure propagation, stable final check, + unchanged coverage floors, and no path-based skip. +- Existing runner, agent-gate, backend, coverage, and API E2E tests. +- Hosted exact-checked-out-tree timing report against PR #161. + +## Operator documentation + +Update canonical `docs/operations_backend_testing.md` with preflight, shard, API, +and fan-in diagnosis; fixed artifact contents and retention; whole-workflow versus +failed-job reruns and stale rejection; parallel runner cost; stable check identity; +progress semantics; and one-PR rollback. + +## Rollback + +If hosted sharding is unstable or slower, revert the implementation PR as one +workflow/script unit to restore the existing sequential job. Do not lower +thresholds, disable shards, or silently fall back inside the workflow. + +## Delivery order + +1. `WS-CI-001-01`: deterministic parallel full-suite execution and centralized + coverage fan-in. +2. `WS-CI-001-02`: optional measured routing/cache/telemetry improvements after + 01 has stable hosted evidence. + +Implementation does not begin until the reviewed 01 contract receives explicit +human approval. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md new file mode 100644 index 00000000..8ab20453 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md @@ -0,0 +1,17 @@ +# RISKS: WS-CI-001 - Backend CI Acceleration + +| ID | Risk | Severity | Mitigation | +|---|---|---:|---| +| R1 | A test is omitted or duplicated | Critical | Filesystem modules plus canonical collected node IDs and exact-once shard observation/fan-in validation | +| R2 | Coverage is combined from incomplete, foreign, or altered evidence | Critical | Bind fixed artifacts and coverage SHA-256 to checked-out tree, shard ID, schema, and manifest digest | +| R3 | Upstream failure is hidden by dependency skipping | Critical | Always-run final check explicitly validates every dependency result | +| R4 | Shards interfere through shared database state | Critical | One isolated migrated database and role per shard process | +| R5 | Coverage thresholds are weakened | Critical | Preserve exact 78/90 commands and add workflow regression assertions | +| R6 | MinIO tests run without a real provider | High | Start pinned MinIO in all shards initially or prove a safe module map before narrowing | +| R7 | One large module controls wall time | Medium | Measure hosted shard duration; consider reviewed node-level split only later | +| R8 | Parallel jobs cost more runner minutes | Medium | Begin with four shards and compare aggregate minutes with PR #161 | +| R9 | Artifact names collide | High | Per-commit/per-shard names and strict unique-set fan-in | +| R10 | Untrusted paths reach shell execution | High | Canonical path validation and argument-array execution in repository script | +| R11 | Required check identity changes | Critical | Preserve final Backend `test` job and verify in workflow tests/GitHub PR | +| R12 | Plan scope expands into path-based skipping | High | Defer routing to separately approved 02 contract | +| R13 | Mutable PostgreSQL tag changes CI behavior | High | Replace `postgres:16` with a reviewed digest pin in 01 | diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md new file mode 100644 index 00000000..47160bcc --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md @@ -0,0 +1,9 @@ +# STATUS: WS-CI-001 - Backend CI Acceleration + +- Phase: planning complete; preimplementation review passed +- Active implementation chunk: none +- Proposed first chunk: `WS-CI-001-01` +- Human direction: prioritize full-suite CI acceleration before + `WS-ENG-001-04B` +- Current gate: explicit human approval to implement `WS-CI-001-01` +- `WS-ENG-001-04B`: remains inactive diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md new file mode 100644 index 00000000..d7193405 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md @@ -0,0 +1,157 @@ +# Chunk Contract: WS-CI-001-01 — Parallel Full-Suite Coverage + +## Parent initiative + +`WS-CI-001` — Backend CI Acceleration + +## Goal + +Run every collected backend test across isolated parallel jobs and combine their +coverage in one fail-closed required Backend check, reducing wall-clock latency +without weakening evidence. + +## Why this chunk exists + +PR #161 proved the sequential full-suite step consumes 25 minutes 31 seconds of +an approximately 27-minute Backend job. This chunk addresses that bottleneck +before more product chunks incur the same delay. + +## Approved plan reference + +- INTENT: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md` +- PLAN: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md` +- CHUNK_MAP: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md` + +## Risk class + +L1 + +## SLA + +P1 + +## Allowed files + +```text +.github/workflows/backend.yml +backend/scripts/ci_test_shards.py +backend/tests/test_ci_test_shards.py +scripts/test_agent_gates.py +.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/** +.agent-loop/merge-intents/WS-CI-001-01.json +docs/operations_backend_testing.md +``` + +## Not allowed + +```text +backend/app/** +backend/alembic/** +backend/pyproject.toml or dependency changes +existing backend test behavior or assertions outside the new planner tests +coverage threshold reductions +test skips, deselection, sampling, or path-based workflow suppression +shared-database parallel workers +unpinned GitHub actions or container images +repository write permissions +changes to Agent Gates, human approval, or merge authority +WS-ENG-001-04B implementation or activation +``` + +## Acceptance criteria + +- [ ] Canonical module inventory comes from symlink-safe filesystem discovery + and includes every `backend/tests/test_*.py` module except an explicit + declaration for the separately executed isolated-runner self-test; it + rejects missing, extra, duplicate, noncanonical, symlinked, escaping, or + zero-test modules and all collection errors. +- [ ] Canonical executable inventory includes every collected pytest node ID, + including parameterized nodes, maps each node to exactly one module, and + uses collected counts as initial shard weights. +- [ ] A schema-versioned deterministic manifest assigns every module exactly + once across four file-level shards using stable weights and tie-breaking; + canonical serialization binds schema, actual checked-out tree SHA, ordered + modules, exclusion, node IDs/counts, shard count, weights, and assignments. +- [ ] Every shard validates and runs at the manifest's actual checked-out tree + SHA with its own digest-pinned PostgreSQL service, + isolated migrated database/role, coverage filename, and result metadata. +- [ ] Real MinIO remains available to every shard unless executable discovery + proves and reviewers approve a narrower fail-closed mapping. +- [ ] The API contract E2E proof runs concurrently with shards using its own + isolated database. +- [ ] Test invocation uses validated Python argv/subprocess arguments, never a + shell-expanded module list; each shard records observed node IDs/counts. +- [ ] Exactly four fixed-name artifacts bind checked-out tree SHA, canonical + manifest digest, schema, shard ID, observed nodes, and SHA-256 of coverage. +- [ ] Fan-in accepts only allowlisted regular files and rejects symlinks, + traversal, unexpected files, wildcard surplus, coverage-name collisions, + credentials, database URLs, environment dumps, and unnecessary DB metadata. +- [ ] The final required Backend `test` job explicitly fails on any failed, + cancelled, skipped, missing, extra, duplicate, malformed, or foreign + prerequisite/artifact. +- [ ] Fan-in recomputes manifest and coverage digests and proves shard-observed + node union equals canonical nodes exactly once, rejecting missing, + duplicate, foreign nodes and count disagreement. +- [ ] Coverage is combined only after complete fan-in and the exact existing 78 + percent global plus twelve 90 percent subsystem reports all pass. +- [ ] Workflow `Backend` and final job ID `test` preserve `Backend / test`; the + final job uses `if: always()` and validates every `needs..result`. +- [ ] All actions, PostgreSQL, and MinIO references are immutable SHA/digest pins; + explicit permissions are `contents: read` plus minimum artifact access, no + repository write exists, and `pull_request_target` is absent. +- [ ] Hosted exact-checked-out-tree proof runs all shards and reports timing + against PR #161 without claiming success from local estimates. +- [ ] The backend testing runbook documents diagnosis, artifact retention/schema, + reruns/stale rejection, cost, check identity, progress, and atomic rollback. + +## Verification commands + +```bash +ruff check backend/scripts/ci_test_shards.py backend/tests/test_ci_test_shards.py scripts/test_agent_gates.py +python3 -m py_compile backend/scripts/ci_test_shards.py backend/tests/test_ci_test_shards.py scripts/test_agent_gates.py +PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -q backend/tests/test_ci_test_shards.py scripts/test_agent_gates.py +PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python3 -m pytest -q backend/tests/test_isolated_database_runner.py backend/tests/test_coverage_contract.py +python3 backend/scripts/ci_test_shards.py dry-run --repository-root . --shards 4 +python3 scripts/check_internal_review_evidence.py +python3 scripts/check_markdown_links.py +python3 scripts/check_stale_workstream_wording.py +python3 scripts/workstream_agent_gate.py --base origin/main --head HEAD --format markdown +git diff --check +``` + +The dry run must perform real collection, plan four selections, synthesize the +expected shard-observation metadata without executing tests, and validate exact +inventory/provenance fan-in. Hosted PR evidence must additionally show all +Backend matrix jobs, API E2E, full +combined coverage, every protected subsystem report, Agent Gates, and external +review passing on the exact reviewed head. + +## Required reviewers + +- [ ] senior engineering +- [ ] QA/test +- [ ] security/auth +- [ ] product/ops +- [ ] architecture +- [ ] CI integrity +- [ ] docs +- [ ] reuse/dedup +- [ ] test delta + +## Human review focus + +Inventory exactness, database/service isolation, failure propagation, artifact +provenance, coverage combination, unchanged thresholds, stable required-check +identity, and measured latency-versus-runner-cost tradeoff. + +## Stop conditions + +Stop and escalate if: + +- complete coverage requires shared mutable database state across shards +- a test must be skipped or weakened to make sharding pass +- the final required check cannot fail closed on upstream cancellation/skipping +- coverage artifacts cannot be bound to exact commit and inventory +- a dependency or backend product change becomes necessary +- hosted wall time does not improve enough to justify added runner cost +- scope expands into path routing or `WS-ENG-001-04B` diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-02-safe-routing-cache-timing.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-02-safe-routing-cache-timing.md new file mode 100644 index 00000000..578ec1b7 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-02-safe-routing-cache-timing.md @@ -0,0 +1,70 @@ +# Chunk Contract: WS-CI-001-02 — Safe Routing, Cache, and Timing Refinement + +## Parent initiative + +`WS-CI-001` — Backend CI Acceleration + +## Goal + +Use measured 01 evidence to consider fail-closed path routing, dependency cache, +and durable shard weights without weakening full-suite requirements. + +## Why this chunk exists + +Parallelization addresses elapsed test time first. Routing, caching, and timing +data have different trust and invalidation risks and require a separate review. + +## Approved plan reference + +- INTENT: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md` +- PLAN: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md` +- CHUNK_MAP: `.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md` + +## Risk class + +L1 + +## SLA + +P2 + +## Allowed files + +To be fixed only after 01 hosted evidence and separate discovery. + +## Not allowed + +Implementation, activation, or successor declaration before a separate human +approval; any coverage/test weakening; backend product changes; 04B activation. + +## Acceptance criteria + +- [ ] A separate reviewed amendment defines exact files and fail-closed routing. +- [ ] Cache and timing provenance cannot cross dependency or commit boundaries. +- [ ] Full-suite-required change classes default closed on ambiguity. + +## Verification commands + +To be defined from 01 hosted evidence. + +## Required reviewers + +- [ ] senior engineering +- [ ] QA/test +- [ ] security/auth +- [ ] product/ops +- [ ] architecture +- [ ] CI integrity +- [ ] docs +- [ ] reuse/dedup +- [ ] test delta + +## Human review focus + +Whether any optimization can suppress required proof and whether added cache or +telemetry complexity is justified by measured results. + +## Stop conditions + +Stop if 01 is not merged and stable, if scope is not explicit, or if the user has +not separately approved this chunk. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-preimplementation-plan-review.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-preimplementation-plan-review.md new file mode 100644 index 00000000..db5be742 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-preimplementation-plan-review.md @@ -0,0 +1,61 @@ +# WS-CI-001-01 Preimplementation Plan Review + +Reviewed scope: `INTENT.md`, `DISCOVERY.md`, `PLAN.md`, `CHUNK_MAP.md`, +`STATUS.md`, `RISKS.md`, `DECISIONS.md`, and chunk contracts 01 and 02. + +Risk route: + +- Risk class: L1 +- SLA: P1 +- Work type: CI, infrastructure, test, architecture, documentation +- Human gate: explicit approval before implementation and before PR merge +- Budget posture: proof-heavy; latency improvement cannot weaken test evidence + +## Final results + +| Track | Result | Blocking findings | +|---|---|---| +| Senior engineering | PASS | None | +| QA/test | PASS | None | +| Security/auth | PASS | None | +| Product/ops | PASS | None | +| Architecture | PASS | None | +| CI integrity | PASS | None | +| Docs | PASS | None | +| Reuse/dedup | PASS | None | +| Test delta | PASS | None | + +Reviewer runs: + +- senior engineering, architecture, reuse/dedup: `ci_plan_arch` +- QA/test, CI integrity, test delta: `ci_plan_qa` +- security/auth, product/ops, docs: `ci_plan_sec_ops` + +## Findings repaired + +- Grounded module completeness in symlink-safe filesystem discovery and bound + executable completeness to canonical collected pytest node IDs. +- Required each shard to report observed nodes and fan-in to prove their exact + union equals the prerequisite inventory once, including parameterized nodes. +- Defined canonical manifest serialization over schema, actual checked-out tree, + exclusions, ordered modules/nodes/counts, weights, assignments, and shard count. +- Required exactly four fixed artifacts, SHA-256 coverage-byte binding, allowlisted + regular files, and rejection of surplus, symlinked, traversing, malformed, + colliding, foreign, or secret-bearing content. +- Preserved workflow `Backend`, final job ID `test`, and required context + `Backend / test`; required `if: always()` plus explicit dependency-result checks. +- Required Python argv invocation, read-only permissions, no + `pull_request_target`, and a digest-pinned PostgreSQL image. +- Reused `run_isolated_tests.py` for database ownership and + `docs/operations_backend_testing.md` for operator guidance. +- Added isolation/coverage regression commands and a real + collection-to-plan-to-four-shard-to-fan-in local dry run. + +## Final assessment + +`WS-CI-001-01` is cohesive because inventory, isolated execution, evidence +fan-in, and centralized coverage form one indivisible CI trust boundary. Splitting +them into separate implementation PRs would leave an unsafe intermediate state. +The plan preserves every test and coverage threshold and is ready for explicit +human implementation approval. `WS-CI-001-02` and `WS-ENG-001-04B` remain +inactive. From b4aa7955da26e351c137f5698dc53420aacf2cbf Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 17:00:12 +0100 Subject: [PATCH 02/12] ci(backend): run full suite in isolated shards --- .../CHUNK_MAP.md | 2 +- .../STATUS.md | 6 +- .agent-loop/merge-intents/WS-CI-001-01.json | 9 + .github/workflows/backend.yml | 336 ++++++++--- backend/scripts/ci_test_shards.py | 549 ++++++++++++++++++ backend/tests/test_ci_test_shards.py | 354 +++++++++++ docs/operations_backend_testing.md | 54 ++ scripts/test_agent_gates.py | 148 +++-- 8 files changed, 1323 insertions(+), 135 deletions(-) create mode 100644 .agent-loop/merge-intents/WS-CI-001-01.json create mode 100644 backend/scripts/ci_test_shards.py create mode 100644 backend/tests/test_ci_test_shards.py diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md index 6c87aa5b..64a58bc9 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md @@ -2,7 +2,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Plan reviewed; awaiting human implementation approval | +| `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Implementation in progress; evidence pending | | `WS-CI-001-02` | Safe Routing, Cache, and Timing Refinement | L1 | Future; not started and requires separate approval | Each chunk maps to one PR. Chunk 01 preserves the full suite and every coverage diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md index 47160bcc..e1fdbaca 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md @@ -1,9 +1,9 @@ # STATUS: WS-CI-001 - Backend CI Acceleration -- Phase: planning complete; preimplementation review passed -- Active implementation chunk: none +- Phase: implementation and deterministic evidence +- Active implementation chunk: `WS-CI-001-01` - Proposed first chunk: `WS-CI-001-01` - Human direction: prioritize full-suite CI acceleration before `WS-ENG-001-04B` -- Current gate: explicit human approval to implement `WS-CI-001-01` +- Current gate: deterministic evidence, required internal review, then human PR review - `WS-ENG-001-04B`: remains inactive diff --git a/.agent-loop/merge-intents/WS-CI-001-01.json b/.agent-loop/merge-intents/WS-CI-001-01.json new file mode 100644 index 00000000..5d6e839c --- /dev/null +++ b/.agent-loop/merge-intents/WS-CI-001-01.json @@ -0,0 +1,9 @@ +{ + "chunk_id": "WS-CI-001-01", + "chunk_title": "Parallel Full-Suite Coverage", + "initiative_id": "WS-CI-001", + "next_chunk_id": null, + "next_chunk_title": null, + "next_requires_explicit_start": true, + "schema_version": 2 +} diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index b3101fb8..973c9d31 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -6,14 +6,23 @@ on: branches: - main +permissions: + contents: read + +env: + MINIO_IMAGE: quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e + jobs: - test: + preflight: runs-on: ubuntu-latest - timeout-minutes: 240 + timeout-minutes: 30 + outputs: + tree_sha: ${{ steps.identity.outputs.tree_sha }} + manifest_sha256: ${{ steps.plan.outputs.manifest_sha256 }} services: postgres: - image: public.ecr.aws/docker/library/postgres:16 + image: public.ecr.aws/docker/library/postgres:16@sha256:33f923b05f64ca54ac4401c01126a6b92afe839a0aa0a52bc5aeb5cc958e5f20 env: POSTGRES_DB: workstream_test POSTGRES_USER: workstream @@ -36,6 +45,17 @@ jobs: with: python-version: "3.12" + - id: identity + name: Bind checked-out tree + shell: bash + run: | + set -euo pipefail + tree_sha=$(git rev-parse HEAD) + if [[ ! "${tree_sha}" =~ ^[0-9a-f]{40}$ ]]; then + exit 1 + fi + echo "tree_sha=${tree_sha}" >> "${GITHUB_OUTPUT}" + - name: Internal review evidence gate run: python3 scripts/check_internal_review_evidence.py @@ -45,14 +65,96 @@ jobs: python -m pip install --upgrade pip python -m pip install -e ".[dev]" + - name: Lint + working-directory: backend + run: ruff check app tests scripts + + - name: Docstring coverage + working-directory: backend + run: docstr-coverage --config .docstr.yaml + + - name: Isolated database runner test + working-directory: backend + env: + WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres + run: python -m pytest -q tests/test_isolated_database_runner.py + + - id: plan + name: Collect and plan exact test inventory + shell: bash + run: | + set -euo pipefail + mkdir -p .ci/plan + manifest_sha256=$(python3 backend/scripts/ci_test_shards.py plan \ + --repository-root . \ + --tree-sha "${{ steps.identity.outputs.tree_sha }}" \ + --shards 4 \ + --output .ci/plan/shard-manifest.json) + echo "manifest_sha256=${manifest_sha256}" >> "${GITHUB_OUTPUT}" + + - name: Upload immutable shard plan + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 + with: + name: backend-shard-plan-${{ steps.identity.outputs.tree_sha }} + path: .ci/plan/shard-manifest.json + if-no-files-found: error + retention-days: 7 + + shards: + needs: preflight + runs-on: ubuntu-latest + timeout-minutes: 90 + strategy: + fail-fast: false + matrix: + shard: [1, 2, 3, 4] + + services: + postgres: + image: public.ecr.aws/docker/library/postgres:16@sha256:33f923b05f64ca54ac4401c01126a6b92afe839a0aa0a52bc5aeb5cc958e5f20 + env: + POSTGRES_DB: workstream_test + POSTGRES_USER: workstream + POSTGRES_PASSWORD: workstream + ports: + - 5433:5432 + options: >- + --health-cmd "pg_isready -U workstream -d workstream_test" + --health-interval 5s + --health-timeout 5s + --health-retries 10 + + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 + with: + persist-credentials: false + fetch-depth: 0 + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 + with: + python-version: "3.12" + + - name: Install backend + working-directory: backend + run: | + python -m pip install --upgrade pip + python -m pip install -e ".[dev]" + + - name: Download exact shard plan + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-shard-plan-${{ needs.preflight.outputs.tree_sha }} + path: .ci/plan + - name: Start real MinIO artifact provider + shell: bash run: | - image='quay.io/minio/minio:latest@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e' + set -euo pipefail docker run --detach --rm --name workstream-minio \ --publish 127.0.0.1:9000:9000 \ --env MINIO_ROOT_USER=workstream-minio \ --env MINIO_ROOT_PASSWORD=workstream-minio-secret-key \ - "$image" server /data --address :9000 + "${MINIO_IMAGE}" server /data --address :9000 for attempt in $(seq 1 60); do if curl --fail --silent http://127.0.0.1:9000/minio/health/live >/dev/null; then exit 0 @@ -62,29 +164,156 @@ jobs: docker logs workstream-minio exit 1 - - name: Lint - working-directory: backend - run: ruff check app tests scripts + - name: Run isolated shard ${{ matrix.shard }} + env: + WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres + WORKSTREAM_TEST_MINIO_ENDPOINT: http://127.0.0.1:9000 + run: >- + python3 backend/scripts/ci_test_shards.py run-shard + --repository-root . + --manifest .ci/plan/shard-manifest.json + --shard ${{ matrix.shard }} + --bundle-dir .ci/shard-bundle + --database-metadata "${RUNNER_TEMP}/database-${{ matrix.shard }}.json" - - name: Docstring coverage + - name: Upload authenticated shard ${{ matrix.shard }} + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 + with: + name: backend-coverage-${{ needs.preflight.outputs.tree_sha }}-${{ matrix.shard }} + path: .ci/shard-bundle + if-no-files-found: error + retention-days: 7 + + api_e2e: + needs: preflight + runs-on: ubuntu-latest + timeout-minutes: 30 + + services: + postgres: + image: public.ecr.aws/docker/library/postgres:16@sha256:33f923b05f64ca54ac4401c01126a6b92afe839a0aa0a52bc5aeb5cc958e5f20 + env: + POSTGRES_DB: workstream_test + POSTGRES_USER: workstream + POSTGRES_PASSWORD: workstream + ports: + - 5433:5432 + options: >- + --health-cmd "pg_isready -U workstream -d workstream_test" + --health-interval 5s + --health-timeout 5s + --health-retries 10 + + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 + with: + persist-credentials: false + fetch-depth: 0 + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 + with: + python-version: "3.12" + + - name: Verify checked-out tree + shell: bash + run: test "$(git rev-parse HEAD)" = "${{ needs.preflight.outputs.tree_sha }}" + + - name: Install backend working-directory: backend - run: docstr-coverage --config .docstr.yaml + run: | + python -m pip install --upgrade pip + python -m pip install -e ".[dev]" - - name: Isolated database runner test + - name: API contract real API e2e working-directory: backend env: WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres - run: python -m pytest -q tests/test_isolated_database_runner.py + run: >- + python scripts/run_isolated_tests.py + --metadata-json "${RUNNER_TEMP}/api-database.json" + --timeout-seconds 3600 + -- python scripts/api_contract_e2e.py - - name: Backend full-suite coverage - working-directory: backend + test: + if: ${{ always() }} + needs: [preflight, shards, api_e2e] + runs-on: ubuntu-latest + timeout-minutes: 30 + + steps: + - name: Require every upstream proof env: - WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres - WORKSTREAM_TEST_MINIO_ENDPOINT: http://127.0.0.1:9000 + PREFLIGHT_RESULT: ${{ needs.preflight.result }} + SHARDS_RESULT: ${{ needs.shards.result }} + API_E2E_RESULT: ${{ needs.api_e2e.result }} + shell: bash run: | - metadata_dir="$(mktemp -d)" - trap 'rm -rf "$metadata_dir"' EXIT - python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78 + set -euo pipefail + test "${PREFLIGHT_RESULT}" = success + test "${SHARDS_RESULT}" = success + test "${API_E2E_RESULT}" = success + + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 + with: + persist-credentials: false + fetch-depth: 0 + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 + with: + python-version: "3.12" + + - name: Install backend + working-directory: backend + run: | + python -m pip install --upgrade pip + python -m pip install -e ".[dev]" + + - name: Download exact shard plan + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-shard-plan-${{ needs.preflight.outputs.tree_sha }} + path: .ci/plan + + - name: Download shard 1 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-coverage-${{ needs.preflight.outputs.tree_sha }}-1 + path: .ci/bundles/shard-1 + + - name: Download shard 2 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-coverage-${{ needs.preflight.outputs.tree_sha }}-2 + path: .ci/bundles/shard-2 + + - name: Download shard 3 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-coverage-${{ needs.preflight.outputs.tree_sha }}-3 + path: .ci/bundles/shard-3 + + - name: Download shard 4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 + with: + name: backend-coverage-${{ needs.preflight.outputs.tree_sha }}-4 + path: .ci/bundles/shard-4 + + - name: Validate exact fan-in and combine coverage + shell: bash + run: | + set -euo pipefail + test "$(git rev-parse HEAD)" = "${{ needs.preflight.outputs.tree_sha }}" + python3 backend/scripts/ci_test_shards.py fan-in \ + --manifest .ci/plan/shard-manifest.json \ + --tree-sha "${{ needs.preflight.outputs.tree_sha }}" \ + --bundles-root .ci/bundles \ + --output-dir .ci/combined-coverage + cd backend + coverage combine ../.ci/combined-coverage + + - name: Backend full-suite coverage + working-directory: backend + run: coverage report --precision=2 --fail-under=78 - name: Artifact foundation coverage working-directory: backend @@ -96,83 +325,43 @@ jobs: - name: External service adapter coverage working-directory: backend - run: >- - coverage report - --include='app/interfaces/external_services.py' - --precision=2 - --fail-under=90 + run: coverage report --include='app/interfaces/external_services.py' --precision=2 --fail-under=90 - name: Artifact preparation configuration coverage working-directory: backend - run: >- - coverage report - --include='app/core/config.py' - --precision=2 - --fail-under=90 + run: coverage report --include='app/core/config.py' --precision=2 --fail-under=90 - name: Artifact worker wiring coverage working-directory: backend - run: >- - coverage report - --include='app/workers/*' - --precision=2 - --fail-under=90 + run: coverage report --include='app/workers/*' --precision=2 --fail-under=90 - name: FastAPI startup wiring coverage working-directory: backend - run: >- - coverage report - --include='app/main.py' - --precision=2 - --fail-under=90 + run: coverage report --include='app/main.py' --precision=2 --fail-under=90 - name: S3-compatible artifact adapter coverage working-directory: backend - run: >- - coverage report - --include='app/adapters/artifacts/s3_compatible.py' - --precision=2 - --fail-under=90 + run: coverage report --include='app/adapters/artifacts/s3_compatible.py' --precision=2 --fail-under=90 - name: S3 namespace validation coverage working-directory: backend - run: >- - coverage report - --include='app/core/s3_validation.py' - --precision=2 - --fail-under=90 + run: coverage report --include='app/core/s3_validation.py' --precision=2 --fail-under=90 - name: Audit subsystem coverage working-directory: backend - run: >- - coverage report - --include='app/modules/audit/*' - --precision=2 - --fail-under=90 + run: coverage report --include='app/modules/audit/*' --precision=2 --fail-under=90 - name: Actor subsystem coverage working-directory: backend - run: >- - coverage report - --include='app/modules/actors/*' - --precision=2 - --fail-under=90 + run: coverage report --include='app/modules/actors/*' --precision=2 --fail-under=90 - name: Authorization subsystem coverage working-directory: backend - run: >- - coverage report - --include='app/modules/authorization/*' - --precision=2 - --fail-under=90 + run: coverage report --include='app/modules/authorization/*' --precision=2 --fail-under=90 - name: Task subsystem coverage working-directory: backend - run: >- - coverage report - --include='app/modules/tasks/*' - --precision=2 - --fail-under=90 + run: coverage report --include='app/modules/tasks/*' --precision=2 --fail-under=90 - name: Auth verifier boundary coverage working-directory: backend @@ -181,12 +370,3 @@ jobs: --include='app/interfaces/auth.py,app/core/auth.py,app/adapters/auth/dev.py,app/adapters/auth/flow.py' --precision=2 --fail-under=90 - - - name: API contract real API e2e - working-directory: backend - env: - WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres - run: | - metadata_dir="$(mktemp -d)" - trap 'rm -rf "$metadata_dir"' EXIT - python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 3600 -- python scripts/api_contract_e2e.py diff --git a/backend/scripts/ci_test_shards.py b/backend/scripts/ci_test_shards.py new file mode 100644 index 00000000..af84b8fc --- /dev/null +++ b/backend/scripts/ci_test_shards.py @@ -0,0 +1,549 @@ +#!/usr/bin/env python3 +"""Plan, execute, and validate isolated backend CI test shards.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +from pathlib import Path, PurePosixPath +import re +import shutil +import subprocess +import sys +import tempfile +import time +from typing import Any + +SCHEMA_VERSION = 1 +SHARD_COUNT = 4 +EXCLUDED_MODULE = "backend/tests/test_isolated_database_runner.py" +MODULE_RE = re.compile(r"backend/tests/test_[a-z0-9_]+\.py") +TREE_RE = re.compile(r"[0-9a-f]{40}") +BUNDLE_FILES = {"coverage.data", "result.json"} + + +class ShardError(RuntimeError): + """A stable CI shard planning or evidence failure.""" + + +def _json_bytes(value: Any) -> bytes: + return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode() + + +def _sha256(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def _regular_file(path: Path) -> bool: + return not path.is_symlink() and path.is_file() + + +def _tree_sha(repository_root: Path) -> str: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=repository_root, + check=True, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ).stdout.strip() + if TREE_RE.fullmatch(result) is None: + raise ShardError("invalid_tree_sha") + return result + + +def _validate_tree_sha(value: str) -> str: + if TREE_RE.fullmatch(value) is None: + raise ShardError("invalid_tree_sha") + return value + + +def discover_modules(repository_root: Path) -> list[str]: + """Return the canonical symlink-free backend test module inventory.""" + tests_root = repository_root / "backend/tests" + if tests_root.is_symlink() or not tests_root.is_dir(): + raise ShardError("invalid_tests_root") + modules: list[str] = [] + for path in sorted(tests_root.glob("test_*.py")): + relative = path.relative_to(repository_root).as_posix() + if MODULE_RE.fullmatch(relative) is None or not _regular_file(path): + raise ShardError("invalid_test_module") + if relative != EXCLUDED_MODULE: + modules.append(relative) + if not modules or len(modules) != len(set(modules)): + raise ShardError("invalid_module_inventory") + return modules + + +def _module_from_node(node_id: str) -> str: + if "::" not in node_id: + raise ShardError("invalid_node_id") + module_part = node_id.split("::", 1)[0] + candidate = f"backend/{module_part}" + if MODULE_RE.fullmatch(candidate) is None: + raise ShardError("invalid_node_module") + return candidate + + +def collect_nodes(repository_root: Path, modules: list[str]) -> dict[str, list[str]]: + """Collect canonical pytest node IDs for exactly the supplied modules.""" + backend_root = repository_root / "backend" + relative_modules = [str(PurePosixPath(path).relative_to("backend")) for path in modules] + collection_env = os.environ.copy() + collection_env["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" + result = subprocess.run( + [sys.executable, "-m", "pytest", "--collect-only", "-q", *relative_modules], + cwd=backend_root, + env=collection_env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + if result.returncode != 0: + raise ShardError("pytest_collection_failed") + expected = set(modules) + collected: dict[str, list[str]] = {module: [] for module in modules} + seen: set[str] = set() + for raw_line in result.stdout.splitlines(): + line = raw_line.strip() + if not line or "::" not in line: + continue + module = _module_from_node(line) + if module not in expected or line in seen: + raise ShardError("foreign_or_duplicate_node") + collected[module].append(line) + seen.add(line) + if not seen or any(not nodes for nodes in collected.values()): + raise ShardError("zero_test_module") + return {module: sorted(nodes) for module, nodes in sorted(collected.items())} + + +def _manifest_body( + tree_sha: str, modules_to_nodes: dict[str, list[str]], shard_count: int +) -> dict[str, Any]: + _validate_tree_sha(tree_sha) + if shard_count != SHARD_COUNT: + raise ShardError("invalid_shard_count") + ordered_modules = sorted(modules_to_nodes) + if not ordered_modules or len(ordered_modules) != len(set(ordered_modules)): + raise ShardError("invalid_module_inventory") + for module, nodes in modules_to_nodes.items(): + if MODULE_RE.fullmatch(module) is None or module == EXCLUDED_MODULE: + raise ShardError("invalid_test_module") + if not nodes or nodes != sorted(set(nodes)): + raise ShardError("invalid_node_inventory") + if any(_module_from_node(node) != module for node in nodes): + raise ShardError("node_module_mismatch") + + bins: list[dict[str, Any]] = [ + {"id": shard_id, "modules": [], "weight": 0} + for shard_id in range(1, shard_count + 1) + ] + weighted = sorted( + ((len(modules_to_nodes[module]), module) for module in ordered_modules), + key=lambda item: (-item[0], item[1]), + ) + for weight, module in weighted: + target = min(bins, key=lambda item: (item["weight"], item["id"])) + target["modules"].append(module) + target["weight"] += weight + for shard in bins: + shard["modules"].sort() + if not shard["modules"]: + raise ShardError("empty_shard") + + module_rows = [ + { + "node_ids": modules_to_nodes[module], + "path": module, + "weight": len(modules_to_nodes[module]), + } + for module in ordered_modules + ] + return { + "excluded_modules": [EXCLUDED_MODULE], + "modules": module_rows, + "schema_version": SCHEMA_VERSION, + "shard_count": shard_count, + "shards": bins, + "tree_sha": tree_sha, + } + + +def build_manifest( + tree_sha: str, modules_to_nodes: dict[str, list[str]], shard_count: int +) -> dict[str, Any]: + """Build a canonical manifest with a digest over its executable body.""" + body = _manifest_body(tree_sha, modules_to_nodes, shard_count) + return {**body, "manifest_sha256": _sha256(_json_bytes(body))} + + +def validate_manifest(manifest: dict[str, Any]) -> dict[str, Any]: + """Validate and canonically reproduce one shard manifest.""" + required = { + "excluded_modules", + "manifest_sha256", + "modules", + "schema_version", + "shard_count", + "shards", + "tree_sha", + } + if not isinstance(manifest, dict) or set(manifest) != required: + raise ShardError("invalid_manifest") + digest = manifest.get("manifest_sha256") + modules = manifest.get("modules") + shard_count = manifest.get("shard_count") + if ( + manifest.get("schema_version") != SCHEMA_VERSION + or manifest.get("excluded_modules") != [EXCLUDED_MODULE] + or not isinstance(digest, str) + or re.fullmatch(r"[0-9a-f]{64}", digest) is None + or not isinstance(modules, list) + or not isinstance(shard_count, int) + or isinstance(shard_count, bool) + or not isinstance(manifest.get("shards"), list) + ): + raise ShardError("invalid_manifest") + mapping: dict[str, list[str]] = {} + for row in modules: + if not isinstance(row, dict) or set(row) != {"node_ids", "path", "weight"}: + raise ShardError("invalid_manifest") + path, nodes, weight = row["path"], row["node_ids"], row["weight"] + if not isinstance(path, str) or not isinstance(nodes, list): + raise ShardError("invalid_manifest") + if not all(isinstance(node, str) for node in nodes) or weight != len(nodes): + raise ShardError("invalid_manifest") + if path in mapping: + raise ShardError("invalid_manifest") + mapping[path] = nodes + reproduced = build_manifest( + str(manifest.get("tree_sha", "")), mapping, shard_count + ) + if manifest != reproduced or digest != reproduced["manifest_sha256"]: + raise ShardError("manifest_digest_mismatch") + return reproduced + + +def load_manifest(path: Path) -> dict[str, Any]: + if not _regular_file(path): + raise ShardError("invalid_manifest_file") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ShardError("invalid_manifest_file") from exc + return validate_manifest(value) + + +def _shard(manifest: dict[str, Any], shard_id: int) -> dict[str, Any]: + matches = [row for row in manifest["shards"] if row["id"] == shard_id] + if len(matches) != 1: + raise ShardError("invalid_shard_id") + return matches[0] + + +def _assert_checked_out_tree(repository_root: Path, expected: str) -> None: + if _tree_sha(repository_root) != _validate_tree_sha(expected): + raise ShardError("checked_out_tree_mismatch") + + +def _safe_empty_directory(path: Path) -> None: + if path.is_symlink() or (path.exists() and (not path.is_dir() or any(path.iterdir()))): + raise ShardError("invalid_output_directory") + path.mkdir(parents=True, exist_ok=True) + + +def _write_json(path: Path, value: Any) -> None: + if path.exists() or path.is_symlink(): + raise ShardError("output_exists") + path.write_bytes(_json_bytes(value)) + + +def run_shard( + repository_root: Path, + manifest_path: Path, + shard_id: int, + bundle_dir: Path, + database_metadata: Path, +) -> None: + """Run one manifest shard and emit a fixed, non-secret evidence bundle.""" + manifest = load_manifest(manifest_path) + _assert_checked_out_tree(repository_root, manifest["tree_sha"]) + shard = _shard(manifest, shard_id) + observed = collect_nodes(repository_root, shard["modules"]) + observed_nodes = sorted(node for nodes in observed.values() for node in nodes) + expected_nodes = sorted( + row["node_ids"] for row in manifest["modules"] if row["path"] in shard["modules"] + ) + expected_flat = sorted(node for group in expected_nodes for node in group) + if observed_nodes != expected_flat: + raise ShardError("shard_collection_mismatch") + + _safe_empty_directory(bundle_dir) + coverage_path = bundle_dir / "coverage.data" + relative_modules = [ + str(PurePosixPath(module).relative_to("backend")) for module in shard["modules"] + ] + env = os.environ.copy() + env["COVERAGE_FILE"] = str(coverage_path.resolve()) + started = time.monotonic() + command = [ + sys.executable, + "scripts/run_isolated_tests.py", + "--metadata-json", + str(database_metadata), + "--timeout-seconds", + "12600", + "--", + sys.executable, + "-m", + "pytest", + "-q", + *relative_modules, + "--cov=app", + "--cov-report=", + ] + result = subprocess.run(command, cwd=repository_root / "backend", env=env, check=False) + if result.returncode != 0: + raise ShardError("shard_tests_failed") + if not _regular_file(coverage_path): + raise ShardError("missing_coverage") + coverage_digest = _sha256(coverage_path.read_bytes()) + metadata = { + "coverage_file": "coverage.data", + "coverage_sha256": coverage_digest, + "duration_seconds": round(time.monotonic() - started, 3), + "manifest_sha256": manifest["manifest_sha256"], + "modules": shard["modules"], + "observed_node_ids": observed_nodes, + "schema_version": SCHEMA_VERSION, + "shard_id": shard_id, + "tree_sha": manifest["tree_sha"], + } + _write_json(bundle_dir / "result.json", metadata) + + +def _safe_bundle_files(bundle: Path) -> dict[str, Path]: + if bundle.is_symlink() or not bundle.is_dir(): + raise ShardError("invalid_bundle_directory") + found: dict[str, Path] = {} + for path in bundle.iterdir(): + if path.name not in BUNDLE_FILES or not _regular_file(path): + raise ShardError("unexpected_bundle_path") + found[path.name] = path + if set(found) != BUNDLE_FILES: + raise ShardError("incomplete_bundle") + return found + + +def validate_fan_in( + manifest: dict[str, Any], bundles_root: Path, output_dir: Path +) -> list[Path]: + """Validate the exact shard set and copy authenticated coverage for combine.""" + if bundles_root.is_symlink() or not bundles_root.is_dir(): + raise ShardError("invalid_bundles_root") + expected_dirs = {f"shard-{index}" for index in range(1, SHARD_COUNT + 1)} + entries = {path.name for path in bundles_root.iterdir()} + if entries != expected_dirs: + raise ShardError("unexpected_bundle_set") + + expected_nodes = sorted( + node for module in manifest["modules"] for node in module["node_ids"] + ) + observed_nodes: list[str] = [] + observed_modules: list[str] = [] + coverage_sources: list[Path] = [] + for shard_id in range(1, SHARD_COUNT + 1): + files = _safe_bundle_files(bundles_root / f"shard-{shard_id}") + try: + result = json.loads(files["result.json"].read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ShardError("invalid_result_metadata") from exc + required = { + "coverage_file", + "coverage_sha256", + "duration_seconds", + "manifest_sha256", + "modules", + "observed_node_ids", + "schema_version", + "shard_id", + "tree_sha", + } + if not isinstance(result, dict) or set(result) != required: + raise ShardError("invalid_result_metadata") + expected_shard = _shard(manifest, shard_id) + if ( + result["schema_version"] != SCHEMA_VERSION + or result["shard_id"] != shard_id + or result["tree_sha"] != manifest["tree_sha"] + or result["manifest_sha256"] != manifest["manifest_sha256"] + or result["modules"] != expected_shard["modules"] + or result["coverage_file"] != "coverage.data" + or not isinstance(result["duration_seconds"], (int, float)) + or isinstance(result["duration_seconds"], bool) + or result["duration_seconds"] < 0 + ): + raise ShardError("result_provenance_mismatch") + nodes = result["observed_node_ids"] + if not isinstance(nodes, list) or not all(isinstance(node, str) for node in nodes): + raise ShardError("invalid_observed_nodes") + expected_shard_nodes = sorted( + node + for module in manifest["modules"] + if module["path"] in expected_shard["modules"] + for node in module["node_ids"] + ) + if nodes != expected_shard_nodes: + raise ShardError("shard_node_mismatch") + coverage = files["coverage.data"] + if result["coverage_sha256"] != _sha256(coverage.read_bytes()): + raise ShardError("coverage_digest_mismatch") + observed_nodes.extend(nodes) + observed_modules.extend(result["modules"]) + coverage_sources.append(coverage) + if sorted(observed_nodes) != expected_nodes or len(observed_nodes) != len(set(observed_nodes)): + raise ShardError("fan_in_node_mismatch") + expected_modules = sorted(module["path"] for module in manifest["modules"]) + if sorted(observed_modules) != expected_modules or len(observed_modules) != len( + set(observed_modules) + ): + raise ShardError("fan_in_module_mismatch") + + _safe_empty_directory(output_dir) + outputs: list[Path] = [] + for shard_id, source in enumerate(coverage_sources, 1): + target = output_dir / f".coverage.shard-{shard_id}" + shutil.copyfile(source, target, follow_symlinks=False) + outputs.append(target) + return outputs + + +def _read_json(path: Path) -> dict[str, Any]: + if not _regular_file(path): + raise ShardError("invalid_json_file") + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ShardError("invalid_json_file") from exc + if not isinstance(value, dict): + raise ShardError("invalid_json_file") + return value + + +def _plan(repository_root: Path, tree_sha: str, shard_count: int) -> dict[str, Any]: + modules = discover_modules(repository_root) + return build_manifest(tree_sha, collect_nodes(repository_root, modules), shard_count) + + +def _dry_run(repository_root: Path, shard_count: int) -> None: + manifest = _plan(repository_root, _tree_sha(repository_root), shard_count) + with tempfile.TemporaryDirectory() as temporary: + root = Path(temporary) + bundles = root / "bundles" + bundles.mkdir() + for shard in manifest["shards"]: + bundle = bundles / f"shard-{shard['id']}" + bundle.mkdir() + coverage = bundle / "coverage.data" + coverage.write_bytes(f"dry-run-{shard['id']}".encode()) + nodes = sorted( + node + for module in manifest["modules"] + if module["path"] in shard["modules"] + for node in module["node_ids"] + ) + _write_json( + bundle / "result.json", + { + "coverage_file": "coverage.data", + "coverage_sha256": _sha256(coverage.read_bytes()), + "duration_seconds": 0, + "manifest_sha256": manifest["manifest_sha256"], + "modules": shard["modules"], + "observed_node_ids": nodes, + "schema_version": SCHEMA_VERSION, + "shard_id": shard["id"], + "tree_sha": manifest["tree_sha"], + }, + ) + validate_fan_in(manifest, bundles, root / "combined") + print( + json.dumps( + { + "manifest_sha256": manifest["manifest_sha256"], + "modules": len(manifest["modules"]), + "nodes": sum(len(row["node_ids"]) for row in manifest["modules"]), + "shard_weights": [row["weight"] for row in manifest["shards"]], + "tree_sha": manifest["tree_sha"], + }, + sort_keys=True, + ) + ) + + +def main() -> int: + """Run the requested shard lifecycle command.""" + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + + plan = subparsers.add_parser("plan") + plan.add_argument("--repository-root", type=Path, required=True) + plan.add_argument("--tree-sha", required=True) + plan.add_argument("--shards", type=int, default=SHARD_COUNT) + plan.add_argument("--output", type=Path, required=True) + + run = subparsers.add_parser("run-shard") + run.add_argument("--repository-root", type=Path, required=True) + run.add_argument("--manifest", type=Path, required=True) + run.add_argument("--shard", type=int, required=True) + run.add_argument("--bundle-dir", type=Path, required=True) + run.add_argument("--database-metadata", type=Path, required=True) + + fan_in = subparsers.add_parser("fan-in") + fan_in.add_argument("--manifest", type=Path, required=True) + fan_in.add_argument("--tree-sha", required=True) + fan_in.add_argument("--bundles-root", type=Path, required=True) + fan_in.add_argument("--output-dir", type=Path, required=True) + + dry_run = subparsers.add_parser("dry-run") + dry_run.add_argument("--repository-root", type=Path, required=True) + dry_run.add_argument("--shards", type=int, default=SHARD_COUNT) + + args = parser.parse_args() + try: + if args.command == "plan": + repository_root = args.repository_root.resolve() + manifest = _plan(repository_root, args.tree_sha, args.shards) + if args.output.exists() or args.output.is_symlink(): + raise ShardError("output_exists") + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_bytes(_json_bytes(manifest)) + print(manifest["manifest_sha256"]) + elif args.command == "run-shard": + run_shard( + args.repository_root.resolve(), + args.manifest, + args.shard, + args.bundle_dir, + args.database_metadata, + ) + elif args.command == "fan-in": + manifest = load_manifest(args.manifest) + if manifest["tree_sha"] != _validate_tree_sha(args.tree_sha): + raise ShardError("checked_out_tree_mismatch") + validate_fan_in(manifest, args.bundles_root, args.output_dir) + else: + _dry_run(args.repository_root.resolve(), args.shards) + except (OSError, subprocess.SubprocessError, ShardError) as exc: + code = exc.args[0] if isinstance(exc, ShardError) else "ci_shard_operation_failed" + print(f"backend CI shard operation failed: {code}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/tests/test_ci_test_shards.py b/backend/tests/test_ci_test_shards.py new file mode 100644 index 00000000..9f1c570c --- /dev/null +++ b/backend/tests/test_ci_test_shards.py @@ -0,0 +1,354 @@ +"""Tests for deterministic backend CI shard evidence.""" + +from __future__ import annotations + +import importlib.util +import json +from pathlib import Path + +import pytest + + +SCRIPT = Path(__file__).resolve().parents[1] / "scripts/ci_test_shards.py" +SPEC = importlib.util.spec_from_file_location("ci_test_shards", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +shards = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(shards) + +TREE_SHA = "a" * 40 + + +def _nodes() -> dict[str, list[str]]: + return { + "backend/tests/test_alpha.py": [ + "tests/test_alpha.py::test_a", + "tests/test_alpha.py::test_b[value]", + "tests/test_alpha.py::test_c", + "tests/test_alpha.py::test_d", + ], + "backend/tests/test_beta.py": [ + "tests/test_beta.py::test_a", + "tests/test_beta.py::test_b", + "tests/test_beta.py::test_c", + ], + "backend/tests/test_delta.py": ["tests/test_delta.py::test_a"], + "backend/tests/test_epsilon.py": ["tests/test_epsilon.py::test_a"], + "backend/tests/test_gamma.py": [ + "tests/test_gamma.py::test_a", + "tests/test_gamma.py::test_b", + ], + } + + +def _write_bundle_set(root: Path, manifest: dict) -> None: + root.mkdir() + modules = {row["path"]: row["node_ids"] for row in manifest["modules"]} + for shard in manifest["shards"]: + bundle = root / f"shard-{shard['id']}" + bundle.mkdir() + coverage = bundle / "coverage.data" + coverage.write_bytes(f"coverage-{shard['id']}".encode()) + observed = sorted(node for module in shard["modules"] for node in modules[module]) + result = { + "coverage_file": "coverage.data", + "coverage_sha256": shards._sha256(coverage.read_bytes()), + "duration_seconds": 1.25, + "manifest_sha256": manifest["manifest_sha256"], + "modules": shard["modules"], + "observed_node_ids": observed, + "schema_version": shards.SCHEMA_VERSION, + "shard_id": shard["id"], + "tree_sha": manifest["tree_sha"], + } + (bundle / "result.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + + +def test_manifest_is_deterministic_and_balanced() -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + assert manifest == shards.build_manifest(TREE_SHA, dict(reversed(_nodes().items())), 4) + assert [row["weight"] for row in manifest["shards"]] == [4, 3, 2, 2] + assigned = [module for row in manifest["shards"] for module in row["modules"]] + assert sorted(assigned) == sorted(_nodes()) + assert len(assigned) == len(set(assigned)) + assert shards.validate_manifest(manifest) == manifest + + +@pytest.mark.parametrize( + ("tree_sha", "nodes", "count", "message"), + [ + ("bad", _nodes(), 4, "invalid_tree_sha"), + (TREE_SHA, {}, 4, "invalid_module_inventory"), + (TREE_SHA, _nodes(), 3, "invalid_shard_count"), + ( + TREE_SHA, + {"backend/tests/test_zero.py": []}, + 4, + "invalid_node_inventory", + ), + ( + TREE_SHA, + {"backend/tests/test_alpha.py": ["tests/test_other.py::test_a"]}, + 4, + "node_module_mismatch", + ), + ( + TREE_SHA, + {"backend/tests/../test_escape.py": ["tests/test_escape.py::test_a"]}, + 4, + "invalid_test_module", + ), + ], +) +def test_manifest_rejects_invalid_inventory( + tree_sha: str, nodes: dict[str, list[str]], count: int, message: str +) -> None: + with pytest.raises(shards.ShardError, match=message): + shards.build_manifest(tree_sha, nodes, count) + + +def test_manifest_rejects_digest_or_assignment_tampering() -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + manifest["shards"][0]["modules"].append("backend/tests/test_foreign.py") + with pytest.raises(shards.ShardError, match="manifest_digest_mismatch"): + shards.validate_manifest(manifest) + + +def test_discovery_rejects_symlinked_module(tmp_path: Path) -> None: + tests = tmp_path / "backend/tests" + tests.mkdir(parents=True) + target = tmp_path / "target.py" + target.write_text("def test_a(): pass\n", encoding="utf-8") + (tests / "test_link.py").symlink_to(target) + with pytest.raises(shards.ShardError, match="invalid_test_module"): + shards.discover_modules(tmp_path) + + +def test_collect_nodes_rejects_collection_failure( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + class Result: + returncode = 2 + stdout = "" + stderr = "failure" + + monkeypatch.setattr(shards.subprocess, "run", lambda *args, **kwargs: Result()) + with pytest.raises(shards.ShardError, match="pytest_collection_failed"): + shards.collect_nodes(tmp_path, ["backend/tests/test_alpha.py"]) + + +def test_collect_nodes_rejects_zero_or_foreign_nodes( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + class Result: + returncode = 0 + stdout = "tests/test_other.py::test_a\n" + stderr = "" + + monkeypatch.setattr(shards.subprocess, "run", lambda *args, **kwargs: Result()) + with pytest.raises(shards.ShardError, match="foreign_or_duplicate_node"): + shards.collect_nodes(tmp_path, ["backend/tests/test_alpha.py"]) + + +def test_collect_nodes_accepts_parameterized_nodes_and_sets_safe_environment( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + observed: dict = {} + + class Result: + returncode = 0 + stdout = ( + "tests/test_alpha.py::test_a[value]\n" + "tests/test_alpha.py::test_b\n" + "2 tests collected\n" + ) + stderr = "" + + def fake_run(command, **kwargs): + observed["command"] = command + observed["env"] = kwargs["env"] + return Result() + + monkeypatch.setattr(shards.subprocess, "run", fake_run) + result = shards.collect_nodes(tmp_path, ["backend/tests/test_alpha.py"]) + assert result == { + "backend/tests/test_alpha.py": [ + "tests/test_alpha.py::test_a[value]", + "tests/test_alpha.py::test_b", + ] + } + assert observed["command"][-1] == "tests/test_alpha.py" + assert observed["env"]["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] == "1" + + +def test_discovery_accepts_regular_modules_and_explicit_exclusion(tmp_path: Path) -> None: + tests = tmp_path / "backend/tests" + tests.mkdir(parents=True) + (tests / "test_alpha.py").write_text("def test_a(): pass\n", encoding="utf-8") + (tests / "test_beta.py").write_text("def test_b(): pass\n", encoding="utf-8") + (tests / "test_isolated_database_runner.py").write_text( + "def test_runner(): pass\n", encoding="utf-8" + ) + assert shards.discover_modules(tmp_path) == [ + "backend/tests/test_alpha.py", + "backend/tests/test_beta.py", + ] + + +def test_manifest_file_round_trip_and_invalid_json(tmp_path: Path) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + path = tmp_path / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + assert shards.load_manifest(path) == manifest + path.write_text("not-json", encoding="utf-8") + with pytest.raises(shards.ShardError, match="invalid_manifest_file"): + shards.load_manifest(path) + + +@pytest.mark.parametrize( + ("field", "value"), + [ + ("schema_version", 2), + ("shard_count", True), + ("manifest_sha256", "bad"), + ("excluded_modules", []), + ("shards", "bad"), + ], +) +def test_manifest_rejects_invalid_top_level_schema(field: str, value: object) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + manifest[field] = value + with pytest.raises(shards.ShardError, match="invalid_manifest"): + shards.validate_manifest(manifest) + + +def test_run_shard_uses_python_argv_and_writes_authenticated_bundle( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + manifest_path = tmp_path / "manifest.json" + manifest_path.write_text(json.dumps(manifest), encoding="utf-8") + shard = manifest["shards"][0] + module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} + monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) + monkeypatch.setattr( + shards, + "collect_nodes", + lambda root, modules: {module: module_nodes[module] for module in modules}, + ) + captured: dict = {} + + class Result: + returncode = 0 + + def fake_run(command, **kwargs): + captured["command"] = command + captured["env"] = kwargs["env"] + Path(kwargs["env"]["COVERAGE_FILE"]).write_bytes(b"real-coverage") + return Result() + + monkeypatch.setattr(shards.subprocess, "run", fake_run) + bundle = tmp_path / "bundle" + shards.run_shard( + tmp_path, + manifest_path, + shard["id"], + bundle, + tmp_path / "database.json", + ) + result = json.loads((bundle / "result.json").read_text(encoding="utf-8")) + assert captured["command"][:2] == [shards.sys.executable, "scripts/run_isolated_tests.py"] + assert all(" " not in argument for argument in captured["command"] if argument.startswith("tests/")) + assert result["modules"] == shard["modules"] + assert result["coverage_sha256"] == shards._sha256(b"real-coverage") + + +def test_run_shard_rejects_failed_test_process( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + path = tmp_path / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + shard = manifest["shards"][0] + module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} + monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) + monkeypatch.setattr( + shards, + "collect_nodes", + lambda root, modules: {module: module_nodes[module] for module in modules}, + ) + + class Result: + returncode = 1 + + monkeypatch.setattr(shards.subprocess, "run", lambda *args, **kwargs: Result()) + with pytest.raises(shards.ShardError, match="shard_tests_failed"): + shards.run_shard(tmp_path, path, shard["id"], tmp_path / "bundle", tmp_path / "db") + + +def test_fan_in_accepts_exact_authenticated_bundles(tmp_path: Path) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + bundles = tmp_path / "bundles" + _write_bundle_set(bundles, manifest) + outputs = shards.validate_fan_in(manifest, bundles, tmp_path / "combined") + assert [path.name for path in outputs] == [ + ".coverage.shard-1", + ".coverage.shard-2", + ".coverage.shard-3", + ".coverage.shard-4", + ] + + +@pytest.mark.parametrize("mutation", ["missing", "extra", "coverage", "nodes", "tree"]) +def test_fan_in_rejects_incomplete_or_tampered_evidence( + tmp_path: Path, mutation: str +) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + bundles = tmp_path / "bundles" + _write_bundle_set(bundles, manifest) + if mutation == "missing": + (bundles / "shard-1/result.json").unlink() + elif mutation == "extra": + (bundles / "surplus").mkdir() + elif mutation == "coverage": + (bundles / "shard-1/coverage.data").write_bytes(b"changed") + else: + path = bundles / "shard-1/result.json" + result = json.loads(path.read_text(encoding="utf-8")) + if mutation == "nodes": + result["observed_node_ids"] = result["observed_node_ids"][1:] + else: + result["tree_sha"] = "b" * 40 + path.write_text(json.dumps(result), encoding="utf-8") + with pytest.raises(shards.ShardError): + shards.validate_fan_in(manifest, bundles, tmp_path / "combined") + + +def test_fan_in_rejects_symlink_and_unexpected_file(tmp_path: Path) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + bundles = tmp_path / "bundles" + _write_bundle_set(bundles, manifest) + (bundles / "shard-1/unexpected").write_text("no", encoding="utf-8") + with pytest.raises(shards.ShardError, match="unexpected_bundle_path"): + shards.validate_fan_in(manifest, bundles, tmp_path / "combined") + (bundles / "shard-1/unexpected").unlink() + coverage = bundles / "shard-1/coverage.data" + coverage.unlink() + coverage.symlink_to(bundles / "shard-2/coverage.data") + with pytest.raises(shards.ShardError, match="unexpected_bundle_path"): + shards.validate_fan_in(manifest, bundles, tmp_path / "combined") + + +def test_fan_in_rejects_duplicate_node_across_shards(tmp_path: Path) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + bundles = tmp_path / "bundles" + _write_bundle_set(bundles, manifest) + first = bundles / "shard-1/result.json" + second = bundles / "shard-2/result.json" + one = json.loads(first.read_text(encoding="utf-8")) + two = json.loads(second.read_text(encoding="utf-8")) + two["observed_node_ids"][0] = one["observed_node_ids"][0] + second.write_text(json.dumps(two), encoding="utf-8") + with pytest.raises(shards.ShardError, match="shard_node_mismatch"): + shards.validate_fan_in(manifest, bundles, tmp_path / "combined") diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index 18dd1ea7..d934458a 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -56,3 +56,57 @@ WORKSTREAM_DATABASE_URL='postgresql+asyncpg://USER:PASSWORD@localhost:5433/works Do not use `WORKSTREAM_ALLOW_NONLOCAL_E2E_DATABASE` for ordinary proof. If provisioning fails, confirm the local PostgreSQL provisioning credential can create/drop databases and roles, terminate owned sessions, and reach the named admin database. Diagnostics omit credentials. + +## Hosted parallel full-suite proof + +The required GitHub check remains `Backend / test`. It is the final fan-in for: + +1. `preflight`: evidence gate, lint, docstrings, isolated-runner test, exact test + collection, and deterministic four-shard plan; +2. `shards`: four independent jobs, each with its own digest-pinned PostgreSQL + service, runner-owned migrated database, real digest-pinned MinIO, and + coverage file; +3. `api_e2e`: the real API contract proof in a separate isolated database; and +4. `test`: exact artifact validation, coverage combination, the 78 percent + repository floor, and all protected 90 percent subsystem floors. + +Matrix job state is the live progress view. The isolated runner continues to +buffer and redact pytest output, so a running shard does not stream individual +test names. The final check reports shard duration and balance metadata after +all evidence is authenticated. + +### Evidence bundles + +The preflight plan and four fixed shard bundles are retained for seven days. +Their names include the actual checked-out tree SHA. Each shard bundle contains +only `coverage.data` and allowlisted `result.json`; the result binds the tree, +manifest, shard, modules, observed pytest node IDs, duration, and SHA-256 of the +exact coverage bytes. Bundles never contain database URLs or passwords, MinIO +credentials, environment dumps, or runner database metadata. + +The fan-in accepts exactly four expected regular-file bundles. It rejects stale +tree or manifest bindings, missing/extra/duplicate nodes or modules, altered +coverage, symlinks, path traversal, unexpected files, failed/cancelled/skipped +upstream jobs, and missing artifacts before `coverage combine` runs. + +### Failure diagnosis and reruns + +- `preflight` failure: inspect evidence/lint/runner/collection output. No shard + evidence is valid until preflight succeeds. +- one `shards` matrix failure: inspect that shard's database, MinIO, collection, + or test failure. The final required check must fail even if other shards pass. +- `api_e2e` failure: inspect the independent API contract job; coverage cannot + compensate for it. +- `test` failure: inspect dependency-result validation, exact bundle fan-in, then + the named global or subsystem coverage report. + +A complete workflow rerun creates evidence for the same checked-out tree and is +the clearest recovery. GitHub may rerun failed jobs, but the fan-in still rejects +missing or stale artifacts; never upload or edit bundles manually. A new commit +always requires a complete new run because its tree SHA differs. + +Four shards reduce wall-clock latency by using more concurrent runner minutes. +Review shard durations and total Actions consumption after deployment before +changing the shard count. If parallel execution is unstable or does not justify +its cost, revert the single implementation PR to restore the prior sequential +workflow; do not lower coverage, skip a shard, or add a silent fallback. diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index f287311b..68564022 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -5200,70 +5200,112 @@ def test_local_minio_compose_is_regression_protected() -> None: def test_backend_coverage_thresholds_are_regression_protected() -> None: - """Keep both the approved global floor and stricter artifact floor fail closed.""" + """Keep parallel full-suite fan-in and every coverage floor fail closed.""" workflow_path = ROOT / ".github/workflows/backend.yml" workflow = workflow_path.read_text(encoding="utf-8") parsed_workflow = yaml.safe_load(workflow) + assert parsed_workflow["name"] == "Backend" + assert parsed_workflow["permissions"] == {"contents": "read"} + assert "pull_request_target" not in workflow + assert "paths-ignore" not in workflow and "continue-on-error" not in workflow + jobs = parsed_workflow["jobs"] + assert set(jobs) == {"preflight", "shards", "api_e2e", "test"} + + postgres_image = ( + "public.ecr.aws/docker/library/postgres:16@sha256:" + "33f923b05f64ca54ac4401c01126a6b92afe839a0aa0a52bc5aeb5cc958e5f20" + ) + for job_name in ("preflight", "shards", "api_e2e"): + assert jobs[job_name]["services"]["postgres"]["image"] == postgres_image + + preflight = jobs["preflight"] + assert set(preflight["outputs"]) == {"tree_sha", "manifest_sha256"} + assert any( + step.get("name") == "Isolated database runner test" + and step.get("run") == "python -m pytest -q tests/test_isolated_database_runner.py" + for step in preflight["steps"] + ) + plan_steps = [ + step for step in preflight["steps"] if step.get("name") == "Collect and plan exact test inventory" + ] + assert len(plan_steps) == 1 + assert "ci_test_shards.py plan" in str(plan_steps[0]["run"]) + assert "--shards 4" in str(plan_steps[0]["run"]) + + shard_job = jobs["shards"] + assert shard_job["needs"] == "preflight" + assert shard_job["strategy"] == { + "fail-fast": False, + "matrix": {"shard": [1, 2, 3, 4]}, + } + shard_steps = shard_job["steps"] + minio_steps = [ + step for step in shard_steps if step.get("name") == "Start real MinIO artifact provider" + ] + assert len(minio_steps) == 1 + assert "${MINIO_IMAGE}" in str(minio_steps[0]["run"]) + run_shard_steps = [ + step for step in shard_steps if str(step.get("name", "")).startswith("Run isolated shard") + ] + assert len(run_shard_steps) == 1 + assert "ci_test_shards.py run-shard" in str(run_shard_steps[0]["run"]) + assert "--cov-fail-under" not in str(run_shard_steps[0]["run"]) + + api_steps = jobs["api_e2e"]["steps"] + api_e2e_steps = [ + step for step in api_steps if step.get("name") == "API contract real API e2e" + ] + assert len(api_e2e_steps) == 1 + assert "scripts/run_isolated_tests.py" in str(api_e2e_steps[0]["run"]) + assert "scripts/api_contract_e2e.py" in str(api_e2e_steps[0]["run"]) + test_job = parsed_workflow["jobs"]["test"] - assert set(test_job) == {"runs-on", "timeout-minutes", "services", "steps"} + assert set(test_job) == {"if", "needs", "runs-on", "timeout-minutes", "steps"} + assert test_job["if"] == "${{ always() }}" + assert test_job["needs"] == ["preflight", "shards", "api_e2e"] steps = test_job["steps"] - minio_steps = [ + upstream = [step for step in steps if step.get("name") == "Require every upstream proof"] + assert len(upstream) == 1 + assert upstream[0]["env"] == { + "PREFLIGHT_RESULT": "${{ needs.preflight.result }}", + "SHARDS_RESULT": "${{ needs.shards.result }}", + "API_E2E_RESULT": "${{ needs.api_e2e.result }}", + } + assert str(upstream[0]["run"]).count('= success') == 3 + downloads = [step for step in steps if "actions/download-artifact@" in step.get("uses", "")] + assert len(downloads) == 5 + assert all( + step["uses"] + == "actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093" + and "pattern" not in step.get("with", {}) + for step in downloads + ) + uploads = [ step - for step in steps - if step.get("name") == "Start real MinIO artifact provider" + for job in jobs.values() + for step in job["steps"] + if "actions/upload-artifact@" in step.get("uses", "") ] - assert len(minio_steps) == 1 - assert set(minio_steps[0]) == {"name", "run"} - assert str(minio_steps[0]["run"]).strip() == BACKEND_MINIO_START_COMMAND + assert len(uploads) == 2 + assert all( + step["uses"] + == "actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02" + for step in uploads + ) + fan_in = [step for step in steps if step.get("name") == "Validate exact fan-in and combine coverage"] + assert len(fan_in) == 1 + assert "ci_test_shards.py fan-in" in str(fan_in[0]["run"]) + assert "coverage combine ../.ci/combined-coverage" in str(fan_in[0]["run"]) + full_suite_steps = [ step for step in steps if step.get("name") == "Backend full-suite coverage" ] assert len(full_suite_steps) == 1 - full_suite_run = full_suite_steps[0]["run"] assert full_suite_steps[0].get("working-directory") == "backend" - assert full_suite_steps[0].get("env") == { - "WORKSTREAM_TEST_ADMIN_DATABASE_URL": ( - "postgresql+asyncpg://workstream:workstream@localhost:5433/postgres" - ), - "WORKSTREAM_TEST_MINIO_ENDPOINT": "http://127.0.0.1:9000", - } - for forbidden_key in ("if", "continue-on-error", "shell"): + assert full_suite_steps[0]["run"] == "coverage report --precision=2 --fail-under=78" + for forbidden_key in ("if", "continue-on-error", "shell", "env"): assert forbidden_key not in full_suite_steps[0] - assert full_suite_run.strip() == BACKEND_FULL_SUITE_COVERAGE_COMMAND - assert "/tmp/workstream-database.json" not in workflow full_suite_index = steps.index(full_suite_steps[0]) - assert steps.index(minio_steps[0]) < full_suite_index - isolated_steps = [ - step for step in steps if step.get("name") == "Isolated database runner test" - ] - assert len(isolated_steps) == 1 - isolated_step = isolated_steps[0] - assert isolated_step == { - "name": "Isolated database runner test", - "working-directory": "backend", - "env": { - "WORKSTREAM_TEST_ADMIN_DATABASE_URL": ( - "postgresql+asyncpg://workstream:workstream@localhost:5433/postgres" - ) - }, - "run": "python -m pytest -q tests/test_isolated_database_runner.py", - } - assert steps.index(isolated_step) < full_suite_index - api_e2e_steps = [ - step for step in steps if step.get("name") == "API contract real API e2e" - ] - assert len(api_e2e_steps) == 1 - api_e2e_step = api_e2e_steps[0] - assert set(api_e2e_step) == {"name", "working-directory", "env", "run"} - assert api_e2e_step["working-directory"] == "backend" - assert api_e2e_step["env"] == { - "WORKSTREAM_TEST_ADMIN_DATABASE_URL": ( - "postgresql+asyncpg://workstream:workstream@localhost:5433/postgres" - ) - } - assert str(api_e2e_step["run"]).strip() == BACKEND_API_CONTRACT_E2E_COMMAND - assert steps.index(api_e2e_step) > full_suite_index - assert "WORKSTREAM_DATABASE_URL" not in api_e2e_step["env"] auth_coverage_steps = [ step for step in steps @@ -5273,7 +5315,7 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: AUTH_09B_COVERAGE_COMMANDS ) for coverage_step in auth_coverage_steps: - assert full_suite_index < steps.index(coverage_step) < steps.index(api_e2e_step) + assert full_suite_index < steps.index(coverage_step) assert coverage_step.get("working-directory") == "backend" for forbidden_key in ("if", "continue-on-error", "shell", "env"): assert forbidden_key not in coverage_step @@ -5299,8 +5341,8 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: later_commands = artifact_expected_coverage_commands_for("06B") assert later_commands[0] == FOUNDATION_ARTIFACT_COVERAGE_COMMAND assert any("app/modules/checkers/*" in command for command in later_commands) - assert workflow.count("--cov-fail-under=78") == 1 - assert ("--cov=app --cov-report=term-missing --cov-fail-under=78") in workflow + assert workflow.count("--fail-under=78") == 1 + assert "--cov-fail-under" not in workflow assert workflow.count("--fail-under=90") == len(expected_coverage) + len( AUTH_09B_COVERAGE_COMMANDS ) From d99683fd5428513dfe95709ad5ce26f375358b05 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 17:02:50 +0100 Subject: [PATCH 03/12] docs(agent-loop): clarify CI process terminology --- .../initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md | 2 +- .../initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md | 2 +- .../chunks/WS-CI-001-01-parallel-full-suite-coverage.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md index 0366c484..6df4f21e 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DISCOVERY.md @@ -44,7 +44,7 @@ The full-suite process is therefore the dominant wall-clock cost. `test_projects.py`, `test_tasks.py`, and `test_authorization.py`. - Database-heavy modules construct async engines against the isolated database URL. Migration and transaction tests make process-level database separation - safer than concurrent workers sharing one database. + safer than concurrent test processes sharing one database. - Artifact integration tests use the real MinIO endpoint exposed by the workflow. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md index 5a9f5124..e5a901dd 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md @@ -88,7 +88,7 @@ unchanged isolated runner intentionally buffers and redacts child output. - Test or coverage sampling. - Path-based Backend skipping in the first chunk. -- Shared-database xdist workers. +- Shared-database xdist processes. - Unpinned marketplace sharding actions. - Mutable external timing data without commit/inventory provenance. - Allowing the aggregator to pass when an upstream job is skipped or cancelled. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md index d7193405..c826084b 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md @@ -51,7 +51,7 @@ backend/pyproject.toml or dependency changes existing backend test behavior or assertions outside the new planner tests coverage threshold reductions test skips, deselection, sampling, or path-based workflow suppression -shared-database parallel workers +shared-database parallel test processes unpinned GitHub actions or container images repository write permissions changes to Agent Gates, human approval, or merge authority From 14c50b464efca95da4f57b30272e0ce7e0435c11 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 17:15:10 +0100 Subject: [PATCH 04/12] fix(ci): bind shard evidence to completed tests --- .../WS-CI-001-backend-ci-acceleration/PLAN.md | 7 +- ...-CI-001-01-parallel-full-suite-coverage.md | 5 +- .github/workflows/backend.yml | 21 ++-- backend/scripts/ci_test_shards.py | 118 ++++++++++++++---- backend/tests/test_ci_test_shards.py | 70 ++++++++++- docs/operations_backend_testing.md | 5 +- scripts/test_agent_gates.py | 4 +- 7 files changed, 189 insertions(+), 41 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md index e5a901dd..3f78b671 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md @@ -42,9 +42,10 @@ Run four matrix jobs initially. Each validates its actual checked-out tree SHA against the manifest, provisions its own PostgreSQL service and independently owned migrated database through `run_isolated_tests.py`, starts MinIO, and runs only manifest modules through Python argv/subprocess construction rather than -shell-expanded module strings. It writes a unique coverage file, independently -collects the node IDs observed for its assigned modules, and uploads a fixed-name -bundle containing shard ID, tree SHA, manifest digest, observed node IDs/counts, +shell-expanded module strings. It executes the manifest's validated node IDs, +uses a repository-owned pytest hook to record each node only after its runtime +lifecycle finishes, writes a unique coverage file, and uploads a fixed-name +bundle containing shard ID, tree SHA, manifest digest, completed node IDs/counts, allowlisted non-secret metadata, coverage, and SHA-256 of the coverage bytes. No shard shares a database, role, filesystem artifact, or coverage filename. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md index c826084b..876cdd6d 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md @@ -79,8 +79,9 @@ WS-ENG-001-04B implementation or activation proves and reviewers approve a narrower fail-closed mapping. - [ ] The API contract E2E proof runs concurrently with shards using its own isolated database. -- [ ] Test invocation uses validated Python argv/subprocess arguments, never a - shell-expanded module list; each shard records observed node IDs/counts. +- [ ] Test invocation uses validated manifest node IDs through Python + argv/subprocess arguments, never a shell-expanded list; a repository-owned + pytest hook records each node only after its runtime lifecycle finishes. - [ ] Exactly four fixed-name artifacts bind checked-out tree SHA, canonical manifest digest, schema, shard ID, observed nodes, and SHA-256 of coverage. - [ ] Fan-in accepts only allowlisted regular files and rejects symlinks, diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml index 973c9d31..5adced8a 100644 --- a/.github/workflows/backend.yml +++ b/.github/workflows/backend.yml @@ -18,7 +18,6 @@ jobs: timeout-minutes: 30 outputs: tree_sha: ${{ steps.identity.outputs.tree_sha }} - manifest_sha256: ${{ steps.plan.outputs.manifest_sha256 }} services: postgres: @@ -79,18 +78,16 @@ jobs: WORKSTREAM_TEST_ADMIN_DATABASE_URL: postgresql+asyncpg://workstream:workstream@localhost:5433/postgres run: python -m pytest -q tests/test_isolated_database_runner.py - - id: plan - name: Collect and plan exact test inventory + - name: Collect and plan exact test inventory shell: bash run: | set -euo pipefail mkdir -p .ci/plan - manifest_sha256=$(python3 backend/scripts/ci_test_shards.py plan \ + python3 backend/scripts/ci_test_shards.py plan \ --repository-root . \ --tree-sha "${{ steps.identity.outputs.tree_sha }}" \ --shards 4 \ - --output .ci/plan/shard-manifest.json) - echo "manifest_sha256=${manifest_sha256}" >> "${GITHUB_OUTPUT}" + --output .ci/plan/shard-manifest.json - name: Upload immutable shard plan uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 @@ -307,10 +304,20 @@ jobs: --manifest .ci/plan/shard-manifest.json \ --tree-sha "${{ needs.preflight.outputs.tree_sha }}" \ --bundles-root .ci/bundles \ - --output-dir .ci/combined-coverage + --output-dir .ci/combined-coverage \ + --summary-output .ci/fan-in-summary.json + python3 -m json.tool .ci/fan-in-summary.json cd backend coverage combine ../.ci/combined-coverage + - name: Upload authenticated timing and balance report + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 + with: + name: backend-fan-in-${{ needs.preflight.outputs.tree_sha }} + path: .ci/fan-in-summary.json + if-no-files-found: error + retention-days: 7 + - name: Backend full-suite coverage working-directory: backend run: coverage report --precision=2 --fail-under=78 diff --git a/backend/scripts/ci_test_shards.py b/backend/scripts/ci_test_shards.py index af84b8fc..1e29554f 100644 --- a/backend/scripts/ci_test_shards.py +++ b/backend/scripts/ci_test_shards.py @@ -22,12 +22,38 @@ MODULE_RE = re.compile(r"backend/tests/test_[a-z0-9_]+\.py") TREE_RE = re.compile(r"[0-9a-f]{40}") BUNDLE_FILES = {"coverage.data", "result.json"} +OBSERVED_NODES_ENV = "WORKSTREAM_CI_OBSERVED_NODES" +PYTEST_PLUGINS = ( + "-p", + "pytest_asyncio.plugin", + "-p", + "pytest_cov.plugin", + "-p", + "scripts.ci_test_shards", +) class ShardError(RuntimeError): """A stable CI shard planning or evidence failure.""" +def pytest_runtest_logfinish(nodeid: str, location: tuple[str, int | None, str]) -> None: + """Record a node only after its actual pytest lifecycle has finished.""" + del location + destination = os.environ.get(OBSERVED_NODES_ENV) + if not destination: + return + data = (json.dumps(nodeid) + "\n").encode() + flags = os.O_WRONLY | os.O_APPEND + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(destination, flags) + try: + os.write(descriptor, data) + finally: + os.close(descriptor) + + def _json_bytes(value: Any) -> bytes: return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode() @@ -94,7 +120,15 @@ def collect_nodes(repository_root: Path, modules: list[str]) -> dict[str, list[s collection_env = os.environ.copy() collection_env["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" result = subprocess.run( - [sys.executable, "-m", "pytest", "--collect-only", "-q", *relative_modules], + [ + sys.executable, + "-m", + "pytest", + "--collect-only", + "-q", + *PYTEST_PLUGINS, + *relative_modules, + ], cwd=backend_root, env=collection_env, text=True, @@ -273,22 +307,22 @@ def run_shard( manifest = load_manifest(manifest_path) _assert_checked_out_tree(repository_root, manifest["tree_sha"]) shard = _shard(manifest, shard_id) - observed = collect_nodes(repository_root, shard["modules"]) - observed_nodes = sorted(node for nodes in observed.values() for node in nodes) expected_nodes = sorted( row["node_ids"] for row in manifest["modules"] if row["path"] in shard["modules"] ) expected_flat = sorted(node for group in expected_nodes for node in group) - if observed_nodes != expected_flat: - raise ShardError("shard_collection_mismatch") _safe_empty_directory(bundle_dir) coverage_path = bundle_dir / "coverage.data" - relative_modules = [ - str(PurePosixPath(module).relative_to("backend")) for module in shard["modules"] - ] + observed_path = database_metadata.with_name(f"observed-nodes-{shard_id}.jsonl") + if observed_path.exists() or observed_path.is_symlink(): + raise ShardError("observed_nodes_path_exists") + descriptor = os.open(observed_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + os.close(descriptor) env = os.environ.copy() env["COVERAGE_FILE"] = str(coverage_path.resolve()) + env["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" + env[OBSERVED_NODES_ENV] = str(observed_path.resolve()) started = time.monotonic() command = [ sys.executable, @@ -302,13 +336,27 @@ def run_shard( "-m", "pytest", "-q", - *relative_modules, + *PYTEST_PLUGINS, + *expected_flat, "--cov=app", "--cov-report=", ] result = subprocess.run(command, cwd=repository_root / "backend", env=env, check=False) if result.returncode != 0: raise ShardError("shard_tests_failed") + try: + observed_nodes = sorted( + json.loads(line) + for line in observed_path.read_text(encoding="utf-8").splitlines() + ) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ShardError("invalid_observed_nodes") from exc + if ( + not all(isinstance(node, str) for node in observed_nodes) + or observed_nodes != expected_flat + or len(observed_nodes) != len(set(observed_nodes)) + ): + raise ShardError("shard_execution_mismatch") if not _regular_file(coverage_path): raise ShardError("missing_coverage") coverage_digest = _sha256(coverage_path.read_bytes()) @@ -340,7 +388,10 @@ def _safe_bundle_files(bundle: Path) -> dict[str, Path]: def validate_fan_in( - manifest: dict[str, Any], bundles_root: Path, output_dir: Path + manifest: dict[str, Any], + bundles_root: Path, + output_dir: Path, + summary_output: Path | None = None, ) -> list[Path]: """Validate the exact shard set and copy authenticated coverage for combine.""" if bundles_root.is_symlink() or not bundles_root.is_dir(): @@ -356,6 +407,7 @@ def validate_fan_in( observed_nodes: list[str] = [] observed_modules: list[str] = [] coverage_sources: list[Path] = [] + durations: list[float] = [] for shard_id in range(1, SHARD_COUNT + 1): files = _safe_bundle_files(bundles_root / f"shard-{shard_id}") try: @@ -405,6 +457,7 @@ def validate_fan_in( observed_nodes.extend(nodes) observed_modules.extend(result["modules"]) coverage_sources.append(coverage) + durations.append(float(result["duration_seconds"])) if sorted(observed_nodes) != expected_nodes or len(observed_nodes) != len(set(observed_nodes)): raise ShardError("fan_in_node_mismatch") expected_modules = sorted(module["path"] for module in manifest["modules"]) @@ -419,21 +472,34 @@ def validate_fan_in( target = output_dir / f".coverage.shard-{shard_id}" shutil.copyfile(source, target, follow_symlinks=False) outputs.append(target) + if summary_output is not None: + _write_json( + summary_output, + { + "manifest_sha256": manifest["manifest_sha256"], + "module_count": len(expected_modules), + "node_count": len(expected_nodes), + "schema_version": SCHEMA_VERSION, + "timing": { + "imbalance_seconds": round(max(durations) - min(durations), 3), + "maximum_seconds": max(durations), + "total_runner_seconds": round(sum(durations), 3), + }, + "shards": [ + { + "duration_seconds": durations[index - 1], + "id": index, + "module_count": len(_shard(manifest, index)["modules"]), + "node_count": _shard(manifest, index)["weight"], + } + for index in range(1, SHARD_COUNT + 1) + ], + "tree_sha": manifest["tree_sha"], + }, + ) return outputs -def _read_json(path: Path) -> dict[str, Any]: - if not _regular_file(path): - raise ShardError("invalid_json_file") - try: - value = json.loads(path.read_text(encoding="utf-8")) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise ShardError("invalid_json_file") from exc - if not isinstance(value, dict): - raise ShardError("invalid_json_file") - return value - - def _plan(repository_root: Path, tree_sha: str, shard_count: int) -> dict[str, Any]: modules = discover_modules(repository_root) return build_manifest(tree_sha, collect_nodes(repository_root, modules), shard_count) @@ -508,6 +574,7 @@ def main() -> int: fan_in.add_argument("--tree-sha", required=True) fan_in.add_argument("--bundles-root", type=Path, required=True) fan_in.add_argument("--output-dir", type=Path, required=True) + fan_in.add_argument("--summary-output", type=Path, required=True) dry_run = subparsers.add_parser("dry-run") dry_run.add_argument("--repository-root", type=Path, required=True) @@ -535,7 +602,12 @@ def main() -> int: manifest = load_manifest(args.manifest) if manifest["tree_sha"] != _validate_tree_sha(args.tree_sha): raise ShardError("checked_out_tree_mismatch") - validate_fan_in(manifest, args.bundles_root, args.output_dir) + validate_fan_in( + manifest, + args.bundles_root, + args.output_dir, + args.summary_output, + ) else: _dry_run(args.repository_root.resolve(), args.shards) except (OSError, subprocess.SubprocessError, ShardError) as exc: diff --git a/backend/tests/test_ci_test_shards.py b/backend/tests/test_ci_test_shards.py index 9f1c570c..e1c297e7 100644 --- a/backend/tests/test_ci_test_shards.py +++ b/backend/tests/test_ci_test_shards.py @@ -246,6 +246,12 @@ def fake_run(command, **kwargs): captured["command"] = command captured["env"] = kwargs["env"] Path(kwargs["env"]["COVERAGE_FILE"]).write_bytes(b"real-coverage") + Path(kwargs["env"][shards.OBSERVED_NODES_ENV]).write_text( + "".join(json.dumps(node) + "\n" for node in sorted( + node for module in shard["modules"] for node in module_nodes[module] + )), + encoding="utf-8", + ) return Result() monkeypatch.setattr(shards.subprocess, "run", fake_run) @@ -259,7 +265,12 @@ def fake_run(command, **kwargs): ) result = json.loads((bundle / "result.json").read_text(encoding="utf-8")) assert captured["command"][:2] == [shards.sys.executable, "scripts/run_isolated_tests.py"] - assert all(" " not in argument for argument in captured["command"] if argument.startswith("tests/")) + invoked_nodes = [argument for argument in captured["command"] if argument.startswith("tests/")] + assert invoked_nodes == sorted( + node for module in shard["modules"] for node in module_nodes[module] + ) + assert "scripts.ci_test_shards" in captured["command"] + assert captured["env"]["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] == "1" assert result["modules"] == shard["modules"] assert result["coverage_sha256"] == shards._sha256(b"real-coverage") @@ -287,17 +298,72 @@ class Result: shards.run_shard(tmp_path, path, shard["id"], tmp_path / "bundle", tmp_path / "db") +def test_run_shard_rejects_collected_but_not_executed_node( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) + path = tmp_path / "manifest.json" + path.write_text(json.dumps(manifest), encoding="utf-8") + shard = manifest["shards"][0] + module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} + monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) + + class Result: + returncode = 0 + + def fake_run(command, **kwargs): + Path(kwargs["env"]["COVERAGE_FILE"]).write_bytes(b"coverage") + expected = sorted( + node for module in shard["modules"] for node in module_nodes[module] + ) + Path(kwargs["env"][shards.OBSERVED_NODES_ENV]).write_text( + "".join(json.dumps(node) + "\n" for node in expected[1:]), + encoding="utf-8", + ) + return Result() + + monkeypatch.setattr(shards.subprocess, "run", fake_run) + with pytest.raises(shards.ShardError, match="shard_execution_mismatch"): + shards.run_shard(tmp_path, path, shard["id"], tmp_path / "bundle", tmp_path / "db") + + +def test_pytest_hook_records_only_when_execution_destination_exists( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +) -> None: + destination = tmp_path / "observed.jsonl" + destination.write_text("", encoding="utf-8") + monkeypatch.setenv(shards.OBSERVED_NODES_ENV, str(destination)) + shards.pytest_runtest_logfinish("tests/test_alpha.py::test_a", ("file", 1, "test_a")) + assert destination.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' + monkeypatch.delenv(shards.OBSERVED_NODES_ENV) + shards.pytest_runtest_logfinish("tests/test_alpha.py::test_b", ("file", 2, "test_b")) + assert destination.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' + + def test_fan_in_accepts_exact_authenticated_bundles(tmp_path: Path) -> None: manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) bundles = tmp_path / "bundles" _write_bundle_set(bundles, manifest) - outputs = shards.validate_fan_in(manifest, bundles, tmp_path / "combined") + summary_path = tmp_path / "fan-in-summary.json" + outputs = shards.validate_fan_in( + manifest, bundles, tmp_path / "combined", summary_path + ) assert [path.name for path in outputs] == [ ".coverage.shard-1", ".coverage.shard-2", ".coverage.shard-3", ".coverage.shard-4", ] + summary = json.loads(summary_path.read_text(encoding="utf-8")) + assert summary["tree_sha"] == TREE_SHA + assert summary["manifest_sha256"] == manifest["manifest_sha256"] + assert summary["node_count"] == sum(len(nodes) for nodes in _nodes().values()) + assert [row["node_count"] for row in summary["shards"]] == [4, 3, 2, 2] + assert summary["timing"] == { + "imbalance_seconds": 0.0, + "maximum_seconds": 1.25, + "total_runner_seconds": 5.0, + } @pytest.mark.parametrize("mutation", ["missing", "extra", "coverage", "nodes", "tree"]) diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index d934458a..f83b69a0 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -80,8 +80,9 @@ all evidence is authenticated. The preflight plan and four fixed shard bundles are retained for seven days. Their names include the actual checked-out tree SHA. Each shard bundle contains only `coverage.data` and allowlisted `result.json`; the result binds the tree, -manifest, shard, modules, observed pytest node IDs, duration, and SHA-256 of the -exact coverage bytes. Bundles never contain database URLs or passwords, MinIO +manifest, shard, modules, completed pytest node IDs, duration, and SHA-256 of the +exact coverage bytes. A repository-owned pytest hook records a node only after +its runtime lifecycle finishes. Bundles never contain database URLs or passwords, MinIO credentials, environment dumps, or runner database metadata. The fan-in accepts exactly four expected regular-file bundles. It rejects stale diff --git a/scripts/test_agent_gates.py b/scripts/test_agent_gates.py index 68564022..11dbeee6 100644 --- a/scripts/test_agent_gates.py +++ b/scripts/test_agent_gates.py @@ -5219,7 +5219,7 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: assert jobs[job_name]["services"]["postgres"]["image"] == postgres_image preflight = jobs["preflight"] - assert set(preflight["outputs"]) == {"tree_sha", "manifest_sha256"} + assert set(preflight["outputs"]) == {"tree_sha"} assert any( step.get("name") == "Isolated database runner test" and step.get("run") == "python -m pytest -q tests/test_isolated_database_runner.py" @@ -5286,7 +5286,7 @@ def test_backend_coverage_thresholds_are_regression_protected() -> None: for step in job["steps"] if "actions/upload-artifact@" in step.get("uses", "") ] - assert len(uploads) == 2 + assert len(uploads) == 3 assert all( step["uses"] == "actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02" From a6141d5b7155c178d533e718404cb04576c900d7 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 17:22:22 +0100 Subject: [PATCH 05/12] docs(agent-loop): bind CI shard review evidence --- .../CHUNK_MAP.md | 2 +- .../STATUS.md | 4 +- .../WS-CI-001-01-internal-review-evidence.md | 73 +++++++++++++++++++ .../reviews/WS-CI-001-01-pr-trust-bundle.md | 63 ++++++++++++++++ 4 files changed, 139 insertions(+), 3 deletions(-) create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md create mode 100644 .agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md index 64a58bc9..3156ae6b 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/CHUNK_MAP.md @@ -2,7 +2,7 @@ | Chunk | Title | Risk | Status | |---|---|---:|---| -| `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Implementation in progress; evidence pending | +| `WS-CI-001-01` | Parallel Full-Suite Coverage | L1 | Implemented and internally reviewed; awaiting hosted PR proof/human review | | `WS-CI-001-02` | Safe Routing, Cache, and Timing Refinement | L1 | Future; not started and requires separate approval | Each chunk maps to one PR. Chunk 01 preserves the full suite and every coverage diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md index e1fdbaca..1f5d7004 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md @@ -1,9 +1,9 @@ # STATUS: WS-CI-001 - Backend CI Acceleration -- Phase: implementation and deterministic evidence +- Phase: implementation and internal review complete - Active implementation chunk: `WS-CI-001-01` - Proposed first chunk: `WS-CI-001-01` - Human direction: prioritize full-suite CI acceleration before `WS-ENG-001-04B` -- Current gate: deterministic evidence, required internal review, then human PR review +- Current gate: PR publication, hosted matrix proof, and human review - `WS-ENG-001-04B`: remains inactive diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md new file mode 100644 index 00000000..2cfcbcc1 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -0,0 +1,73 @@ +# Internal Review Evidence: WS-CI-001-01 + +Open sub-agent sessions: none + +Valid findings addressed: yes + +Reviewed code SHA: `14c50b464efca95da4f57b30272e0ce7e0435c11` + +Reviewed at: 2026-07-20T16:21:10Z + +Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_impl_arch`; QA/test/CI-integrity/test-delta=`ci_impl_qa`; security/auth/product/ops/docs=`ci_impl_sec_ops` + +## Binding + +- Base: `c12ba1c8d4bbde86d0e2c19826f5791afc130489` +- Final reviewed implementation: `14c50b464efca95da4f57b30272e0ce7e0435c11` +- Contract: `WS-CI-001-01 — Parallel Full-Suite Coverage` +- Scope: CI workflow, one shared shard tool, additive tests, operator runbook, + initiative evidence, and exactly one merge intent; no product runtime change + +## Reviewer Results + +| Reviewer | Result | Blocking findings | Notes | +|---|---|---|---| +| senior engineering | PASS | None | Cohesive inventory-to-fan-in trust boundary. | +| QA/test | PASS AFTER FIXES | None | Exact runtime-finished node proof replaces pre-run collection claims. | +| security/auth | PASS AFTER FIXES | None | Fixed artifacts, byte hashes, read-only permissions, safe hook path. | +| product/ops | PASS AFTER FIXES | None | Authenticated timing/cost evidence now reaches operators. | +| architecture | PASS | None | Shared Python policy boundary; YAML only orchestrates. | +| CI integrity | PASS AFTER FIXES | None | Stable required check and unchanged 78/90 thresholds. | +| docs | PASS AFTER FIXES | None | Runbook matches timing report and completed-node semantics. | +| reuse/dedup | PASS | None | Existing isolated database runner remains sole DB owner. | +| test delta | PASS AFTER FIXES | None | Additive tests; no skip, assertion, or threshold weakening. | + +## Findings repaired + +- Replaced shard metadata derived from a second collection with exact manifest + node argv and a repository-owned pytest hook recording each node after its + actual runtime lifecycle finishes. A successful shard requires the completed + set to equal the canonical manifest exactly once. +- Added a regression proving a collected-but-not-executed node fails. +- Added authenticated fan-in summary containing tree, manifest, per-shard + duration/node/module balance, total runner seconds, and timing imbalance; + the final check prints and uploads it through a pinned action. +- Removed one unused JSON helper and one unused workflow output. +- Replaced ambiguous CI “worker” wording with test-process terminology. + +## Deterministic Evidence + +```text +Ruff: passed +Python compilation: passed +planner + agent-gate + coverage-contract tests: 294 passed +isolated-runner + coverage-contract tests: 188 passed +clean-environment real collection/fan-in dry run: 31 modules, 1774 nodes +shard weights: 443 / 444 / 443 / 444 +merge-intent validation: passed +loop-memory state validation: passed +Markdown links: passed for 11 changed Markdown files +stale Workstream/authorization/artifact/review scans: passed +git diff --check: passed +``` + +The static sensor reports `REVIEW_REQUIRED` for a large L1 CI diff and the moved +global-threshold enforcement point. This is expected review routing, not a bypass: +the final authenticated fan-in enforces the same 78 percent global floor and all +twelve 90 percent subsystem floors, and all nine reviewer tracks passed. + +## Remaining Gate + +The new GitHub matrix and actual latency/cost can only be proven on the pushed PR +head. Hosted `Backend / test`, Agent Gates, external review, and human merge +approval remain mandatory. `WS-CI-001-02` and `WS-ENG-001-04B` are inactive. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md new file mode 100644 index 00000000..a7639206 --- /dev/null +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -0,0 +1,63 @@ +# WS-CI-001-01 PR Trust Bundle + +## Goal + +Reduce Backend CI wall-clock time by running the complete test inventory in four +isolated jobs, then authenticate and combine all evidence before the unchanged +required coverage gates pass. + +## Design + +- Preflight discovers every canonical test module and collected pytest node. +- Deterministic file-level bin packing keeps modules intact across four shards. +- Every shard validates the checked-out tree, uses its own PostgreSQL database + and MinIO instance, executes exact manifest node IDs, and records nodes only + after their runtime lifecycle finishes. +- Fixed bundles bind tree SHA, manifest digest, exact completed nodes, coverage + bytes, shard identity, and non-secret duration metadata. +- Final `Backend / test` runs with `if: always()`, rejects every upstream or + artifact inconsistency, combines coverage, and enforces 78 percent global plus + twelve 90 percent subsystem floors. +- API contract E2E runs concurrently in its own isolated database. + +## Scope Control + +No backend application code, schema, migration, dependency, existing test +behavior, assertion, skip, coverage floor, authentication, authorization, +artifact product behavior, payment, review, approval, or merge authority changed. +No path-based workflow suppression was added. + +## Evidence + +- 294 focused planner/agent-gate/coverage-contract tests pass. +- 188 isolated-runner/coverage-contract tests pass against local PostgreSQL. +- Clean installed environment discovers 31 modules and 1,774 nodes and validates + complete four-shard fan-in at weights 443/444/443/444. +- Ruff, compilation, merge-intent, loop-state, stale scans, Markdown links, and + diff integrity pass. +- Actions are SHA-pinned; PostgreSQL and MinIO are digest-pinned. +- Workflow permissions are read-only and checkout credentials are not persisted. + +## Internal Review + +Exact SHA `14c50b464efca95da4f57b30272e0ce7e0435c11` passes senior engineering, +QA/test, security/auth, product/ops, architecture, CI integrity, docs, +reuse/dedup, and test-delta review after all valid findings were repaired. + +## Remaining Risk + +Collected-node counts may not predict runtime for migration-heavy modules. +Hosted execution must show actual per-shard durations, wall-clock improvement, +and aggregate runner cost. The workflow fails closed rather than silently falling +back if sharding is unstable. + +## Human Review Focus + +Exact runtime-node evidence, action/plugin provenance, database/MinIO isolation, +fixed artifact set and byte hashes, cancellation propagation, coverage combine, +unchanged thresholds, stable `Backend / test`, and latency-versus-runner-cost. + +## Human Merge Ownership + +Only the human may approve and merge the PR after hosted exact-head evidence. +`WS-CI-001-02` and `WS-ENG-001-04B` remain inactive. From a048122a547a476b10687205b190f48160efaabb Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 17:24:25 +0100 Subject: [PATCH 06/12] docs(agent-loop): finalize CI shard review binding --- .../reviews/WS-CI-001-01-internal-review-evidence.md | 4 ++-- .../reviews/WS-CI-001-01-pr-trust-bundle.md | 7 ++++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md index 2cfcbcc1..f3aacec6 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -4,9 +4,9 @@ Open sub-agent sessions: none Valid findings addressed: yes -Reviewed code SHA: `14c50b464efca95da4f57b30272e0ce7e0435c11` +Reviewed code SHA: `a6141d5b7155c178d533e718404cb04576c900d7` -Reviewed at: 2026-07-20T16:21:10Z +Reviewed at: 2026-07-20T16:23:40Z Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_impl_arch`; QA/test/CI-integrity/test-delta=`ci_impl_qa`; security/auth/product/ops/docs=`ci_impl_sec_ops` diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index a7639206..976f98d1 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -40,9 +40,10 @@ No path-based workflow suppression was added. ## Internal Review -Exact SHA `14c50b464efca95da4f57b30272e0ce7e0435c11` passes senior engineering, -QA/test, security/auth, product/ops, architecture, CI integrity, docs, -reuse/dedup, and test-delta review after all valid findings were repaired. +Implementation SHA `14c50b464efca95da4f57b30272e0ce7e0435c11` and final evidence head +`a6141d5b7155c178d533e718404cb04576c900d7` pass senior engineering, QA/test, +security/auth, product/ops, architecture, CI integrity, docs, reuse/dedup, and +test-delta review after all valid findings were repaired. ## Remaining Risk From bbad02b8715da3e23f9fa80150b13f2d423ba9c6 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:07:15 +0100 Subject: [PATCH 07/12] fix(ci): tolerate runtime parameter node ids --- .../DECISIONS.md | 9 + .../INTENT.md | 9 +- .../WS-CI-001-backend-ci-acceleration/PLAN.md | 14 +- .../RISKS.md | 3 +- .../STATUS.md | 5 +- ...-CI-001-01-parallel-full-suite-coverage.md | 18 +- .../reviews/WS-CI-001-01-pr-trust-bundle.md | 9 +- backend/scripts/ci_test_shards.py | 257 ++++++++++++------ backend/tests/test_ci_test_shards.py | 130 +++++---- docs/operations_backend_testing.md | 8 +- 10 files changed, 297 insertions(+), 165 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md index fb620443..7fe13ee5 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/DECISIONS.md @@ -28,3 +28,12 @@ exact observed-node fan-in, coverage-byte hashing, checked-out-tree provenance, fixed artifact sets, explicit read-only permissions, digest-pinned PostgreSQL, stable `Backend / test`, canonical operator documentation, and a real local dry run. The plan and 01 contract now include those boundaries; all tracks pass. + +## 2026-07-20 - Bind nondeterministic parameter IDs safely + +The first hosted run proved that parameter display IDs containing import-time +UUID values are not stable across preflight and shard processes. Raw preflight +node IDs are therefore not executable cross-process authority. Shards execute +validated whole modules, record final collection and completion in the same +pytest process, require those exact sets to match, and bind their stable +test-base cardinalities to the authenticated preflight manifest. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md index d578dd0a..c66aa23a 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/INTENT.md @@ -28,8 +28,9 @@ CI evidence. ## Target behavior - Every collected backend test still runs on every required Backend workflow. -- Canonical filesystem test modules and collected pytest node IDs are each - assigned and observed exactly once across isolated parallel shards. +- Canonical filesystem test modules are assigned exactly once. Stable test-base + cardinality signatures bind runtime inventory to preflight, while each shard + proves its exact same-process collection equals its completions. - Each shard receives an independently owned migrated PostgreSQL database. - Coverage artifacts are combined centrally before the unchanged 78 percent repository floor and 90 percent protected-subsystem floors are enforced. @@ -91,8 +92,8 @@ pytest scheduling dependency. ## How this will be proven -Unit tests will prove filesystem and node-ID inventory completeness, determinism, -exact-once assignment and observation, stable ordering, malformed input +Unit tests will prove filesystem and stable node-signature inventory completeness, +determinism, exact same-process collection/completion, stable ordering, malformed input rejection, and fail-closed coverage-manifest validation. Workflow review will prove isolated services and databases, immutable action pins, checked-out-tree binding, byte-hashed artifacts, complete fan-in, unchanged thresholds, stable diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md index 3f78b671..b5d247be 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/PLAN.md @@ -14,7 +14,7 @@ Add a repository-owned planner with one shared validation implementation. Canonical module completeness comes from symlink-safe filesystem discovery of `backend/tests/test_*.py`, with an explicit declaration excluding only `test_isolated_database_runner.py`. A successful pytest collection supplies the -canonical executable node IDs and per-module weights; every node must map to one +canonical per-test-base cardinality signatures and per-module weights; every node must map to one discovered canonical module. Collection errors, zero-test modules, foreign paths, malformed node IDs, and parameterized-node collisions fail. @@ -22,7 +22,7 @@ Assign every non-excluded module exactly once using deterministic largest-weight-first bin packing with lexical tie-breaking. Emit canonical, schema-versioned JSON binding schema, actual checked-out tree SHA from `git rev-parse HEAD`, ordered normalized module paths, explicit exclusion, -ordered node IDs, collected counts, shard count, weights, and assignments. Its +ordered stable node signatures, collected counts, shard count, weights, and assignments. Its SHA-256 digest identifies the exact executable inventory and plan. The initial weight is collected test count per module. Runtime telemetry may @@ -42,10 +42,12 @@ Run four matrix jobs initially. Each validates its actual checked-out tree SHA against the manifest, provisions its own PostgreSQL service and independently owned migrated database through `run_isolated_tests.py`, starts MinIO, and runs only manifest modules through Python argv/subprocess construction rather than -shell-expanded module strings. It executes the manifest's validated node IDs, -uses a repository-owned pytest hook to record each node only after its runtime -lifecycle finishes, writes a unique coverage file, and uploads a fixed-name -bundle containing shard ID, tree SHA, manifest digest, completed node IDs/counts, +shell-expanded module strings. It executes the manifest's validated whole-module +paths and uses repository-owned pytest hooks to record final collection and +runtime completion in that same pytest process. It requires those exact runtime +sets to match and their stable test-base cardinalities to match preflight, writes +a unique coverage file, and uploads a fixed-name bundle containing shard ID, +tree SHA, manifest digest, collected/completed node IDs and counts, allowlisted non-secret metadata, coverage, and SHA-256 of the coverage bytes. No shard shares a database, role, filesystem artifact, or coverage filename. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md index 8ab20453..46196ab5 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/RISKS.md @@ -2,7 +2,7 @@ | ID | Risk | Severity | Mitigation | |---|---|---:|---| -| R1 | A test is omitted or duplicated | Critical | Filesystem modules plus canonical collected node IDs and exact-once shard observation/fan-in validation | +| R1 | A test is omitted or duplicated | Critical | Filesystem modules plus stable preflight cardinality signatures and exact same-process collection/completion validation | | R2 | Coverage is combined from incomplete, foreign, or altered evidence | Critical | Bind fixed artifacts and coverage SHA-256 to checked-out tree, shard ID, schema, and manifest digest | | R3 | Upstream failure is hidden by dependency skipping | Critical | Always-run final check explicitly validates every dependency result | | R4 | Shards interfere through shared database state | Critical | One isolated migrated database and role per shard process | @@ -15,3 +15,4 @@ | R11 | Required check identity changes | Critical | Preserve final Backend `test` job and verify in workflow tests/GitHub PR | | R12 | Plan scope expands into path-based skipping | High | Defer routing to separately approved 02 contract | | R13 | Mutable PostgreSQL tag changes CI behavior | High | Replace `postgres:16` with a reviewed digest pin in 01 | +| R14 | Parameter display values change across pytest processes | Critical | Execute whole modules; compare exact collection/completion within one process and bind only stable test-base cardinalities across processes | diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md index 1f5d7004..425bfbff 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md @@ -1,9 +1,10 @@ # STATUS: WS-CI-001 - Backend CI Acceleration -- Phase: implementation and internal review complete +- Phase: hosted-failure repair and renewed internal review - Active implementation chunk: `WS-CI-001-01` - Proposed first chunk: `WS-CI-001-01` - Human direction: prioritize full-suite CI acceleration before `WS-ENG-001-04B` -- Current gate: PR publication, hosted matrix proof, and human review +- Current gate: repair evidence, renewed internal review, hosted matrix proof, + and human review - `WS-ENG-001-04B`: remains inactive diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md index 876cdd6d..2d101ed3 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/chunks/WS-CI-001-01-parallel-full-suite-coverage.md @@ -71,7 +71,8 @@ WS-ENG-001-04B implementation or activation - [ ] A schema-versioned deterministic manifest assigns every module exactly once across four file-level shards using stable weights and tie-breaking; canonical serialization binds schema, actual checked-out tree SHA, ordered - modules, exclusion, node IDs/counts, shard count, weights, and assignments. + modules, exclusion, stable test-base cardinalities, shard count, weights, + and assignments. - [ ] Every shard validates and runs at the manifest's actual checked-out tree SHA with its own digest-pinned PostgreSQL service, isolated migrated database/role, coverage filename, and result metadata. @@ -79,19 +80,22 @@ WS-ENG-001-04B implementation or activation proves and reviewers approve a narrower fail-closed mapping. - [ ] The API contract E2E proof runs concurrently with shards using its own isolated database. -- [ ] Test invocation uses validated manifest node IDs through Python - argv/subprocess arguments, never a shell-expanded list; a repository-owned - pytest hook records each node only after its runtime lifecycle finishes. +- [ ] Test invocation uses validated whole-module paths through Python + argv/subprocess arguments, never a shell-expanded list; repository-owned + pytest hooks prove exact same-process collection/completion equality and + bind stable test-base cardinalities back to preflight. - [ ] Exactly four fixed-name artifacts bind checked-out tree SHA, canonical - manifest digest, schema, shard ID, observed nodes, and SHA-256 of coverage. + manifest digest, schema, shard ID, collected/completed nodes, and SHA-256 + of coverage. - [ ] Fan-in accepts only allowlisted regular files and rejects symlinks, traversal, unexpected files, wildcard surplus, coverage-name collisions, credentials, database URLs, environment dumps, and unnecessary DB metadata. - [ ] The final required Backend `test` job explicitly fails on any failed, cancelled, skipped, missing, extra, duplicate, malformed, or foreign prerequisite/artifact. -- [ ] Fan-in recomputes manifest and coverage digests and proves shard-observed - node union equals canonical nodes exactly once, rejecting missing, +- [ ] Fan-in recomputes manifest and coverage digests, proves each shard's exact + runtime collection equals its completions, and proves runtime stable + signatures equal the canonical preflight signatures, rejecting missing, duplicate, foreign nodes and count disagreement. - [ ] Coverage is combined only after complete fan-in and the exact existing 78 percent global plus twelve 90 percent subsystem reports all pass. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index 976f98d1..2191c5b6 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -8,12 +8,13 @@ required coverage gates pass. ## Design -- Preflight discovers every canonical test module and collected pytest node. +- Preflight discovers every canonical test module and stable test-base cardinality. - Deterministic file-level bin packing keeps modules intact across four shards. - Every shard validates the checked-out tree, uses its own PostgreSQL database - and MinIO instance, executes exact manifest node IDs, and records nodes only - after their runtime lifecycle finishes. -- Fixed bundles bind tree SHA, manifest digest, exact completed nodes, coverage + and MinIO instance, executes whole assigned modules, and proves exact + same-process runtime collection/completion equality against the preflight + cardinality signature. +- Fixed bundles bind tree SHA, manifest digest, exact collected/completed nodes, coverage bytes, shard identity, and non-secret duration metadata. - Final `Backend / test` runs with `if: always()`, rejects every upstream or artifact inconsistency, combines coverage, and enforces 78 percent global plus diff --git a/backend/scripts/ci_test_shards.py b/backend/scripts/ci_test_shards.py index 1e29554f..20acf481 100644 --- a/backend/scripts/ci_test_shards.py +++ b/backend/scripts/ci_test_shards.py @@ -16,13 +16,14 @@ import time from typing import Any -SCHEMA_VERSION = 1 +SCHEMA_VERSION = 2 SHARD_COUNT = 4 EXCLUDED_MODULE = "backend/tests/test_isolated_database_runner.py" MODULE_RE = re.compile(r"backend/tests/test_[a-z0-9_]+\.py") TREE_RE = re.compile(r"[0-9a-f]{40}") BUNDLE_FILES = {"coverage.data", "result.json"} -OBSERVED_NODES_ENV = "WORKSTREAM_CI_OBSERVED_NODES" +COLLECTED_NODES_ENV = "WORKSTREAM_CI_COLLECTED_NODES" +COMPLETED_NODES_ENV = "WORKSTREAM_CI_COMPLETED_NODES" PYTEST_PLUGINS = ( "-p", "pytest_asyncio.plugin", @@ -37,13 +38,8 @@ class ShardError(RuntimeError): """A stable CI shard planning or evidence failure.""" -def pytest_runtest_logfinish(nodeid: str, location: tuple[str, int | None, str]) -> None: - """Record a node only after its actual pytest lifecycle has finished.""" - del location - destination = os.environ.get(OBSERVED_NODES_ENV) - if not destination: - return - data = (json.dumps(nodeid) + "\n").encode() +def _append_node(destination: str, node_id: str) -> None: + data = (json.dumps(node_id) + "\n").encode() flags = os.O_WRONLY | os.O_APPEND if hasattr(os, "O_NOFOLLOW"): flags |= os.O_NOFOLLOW @@ -54,6 +50,24 @@ def pytest_runtest_logfinish(nodeid: str, location: tuple[str, int | None, str]) os.close(descriptor) +def pytest_collection_finish(session: Any) -> None: + """Record the final selected inventory from the executing pytest process.""" + destination = os.environ.get(COLLECTED_NODES_ENV) + if not destination: + return + for item in session.items: + _append_node(destination, item.nodeid) + + +def pytest_runtest_logfinish(nodeid: str, location: tuple[str, int | None, str]) -> None: + """Record a node only after its actual pytest lifecycle has finished.""" + del location + destination = os.environ.get(COMPLETED_NODES_ENV) + if not destination: + return + _append_node(destination, nodeid) + + def _json_bytes(value: Any) -> bytes: return (json.dumps(value, indent=2, sort_keys=True) + "\n").encode() @@ -113,6 +127,33 @@ def _module_from_node(node_id: str) -> str: return candidate +def _node_base(node_id: str) -> str: + """Remove only pytest's final parameter display value from a node ID.""" + _module_from_node(node_id) + leaf = node_id.rsplit("::", 1)[-1] + if leaf.endswith("]") and "[" in leaf: + return node_id[: -(len(leaf) - leaf.find("["))] + return node_id + + +def _node_signature(nodes: list[str]) -> list[dict[str, Any]]: + counts: dict[str, int] = {} + for node in nodes: + base = _node_base(node) + counts[base] = counts.get(base, 0) + 1 + return [{"base": base, "count": counts[base]} for base in sorted(counts)] + + +def _read_node_log(path: Path) -> list[str]: + try: + nodes = [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()] + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise ShardError("invalid_runtime_nodes") from exc + if not nodes or not all(isinstance(node, str) for node in nodes): + raise ShardError("invalid_runtime_nodes") + return sorted(nodes) + + def collect_nodes(repository_root: Path, modules: list[str]) -> dict[str, list[str]]: """Collect canonical pytest node IDs for exactly the supplied modules.""" backend_root = repository_root / "backend" @@ -156,28 +197,46 @@ def collect_nodes(repository_root: Path, modules: list[str]) -> dict[str, list[s def _manifest_body( - tree_sha: str, modules_to_nodes: dict[str, list[str]], shard_count: int + tree_sha: str, module_rows: list[dict[str, Any]], shard_count: int ) -> dict[str, Any]: _validate_tree_sha(tree_sha) if shard_count != SHARD_COUNT: raise ShardError("invalid_shard_count") - ordered_modules = sorted(modules_to_nodes) + ordered_modules = sorted(row["path"] for row in module_rows) if not ordered_modules or len(ordered_modules) != len(set(ordered_modules)): raise ShardError("invalid_module_inventory") - for module, nodes in modules_to_nodes.items(): + rows_by_module = {row["path"]: row for row in module_rows} + if len(rows_by_module) != len(module_rows): + raise ShardError("invalid_module_inventory") + for module, row in rows_by_module.items(): if MODULE_RE.fullmatch(module) is None or module == EXCLUDED_MODULE: raise ShardError("invalid_test_module") - if not nodes or nodes != sorted(set(nodes)): + signature, weight = row.get("node_signature"), row.get("weight") + if not isinstance(signature, list) or not signature or not isinstance(weight, int): raise ShardError("invalid_node_inventory") - if any(_module_from_node(node) != module for node in nodes): + if any( + not isinstance(item, dict) + or set(item) != {"base", "count"} + or not isinstance(item["base"], str) + or not isinstance(item["count"], int) + or isinstance(item["count"], bool) + or item["count"] < 1 + or _module_from_node(item["base"]) != module + for item in signature + ): raise ShardError("node_module_mismatch") + if signature != sorted(signature, key=lambda item: item["base"]): + raise ShardError("invalid_node_inventory") + if len({item["base"] for item in signature}) != len(signature) or weight != sum( + item["count"] for item in signature + ): + raise ShardError("invalid_node_inventory") bins: list[dict[str, Any]] = [ - {"id": shard_id, "modules": [], "weight": 0} - for shard_id in range(1, shard_count + 1) + {"id": shard_id, "modules": [], "weight": 0} for shard_id in range(1, shard_count + 1) ] weighted = sorted( - ((len(modules_to_nodes[module]), module) for module in ordered_modules), + ((rows_by_module[module]["weight"], module) for module in ordered_modules), key=lambda item: (-item[0], item[1]), ) for weight, module in weighted: @@ -189,17 +248,10 @@ def _manifest_body( if not shard["modules"]: raise ShardError("empty_shard") - module_rows = [ - { - "node_ids": modules_to_nodes[module], - "path": module, - "weight": len(modules_to_nodes[module]), - } - for module in ordered_modules - ] + canonical_rows = [rows_by_module[module] for module in ordered_modules] return { "excluded_modules": [EXCLUDED_MODULE], - "modules": module_rows, + "modules": canonical_rows, "schema_version": SCHEMA_VERSION, "shard_count": shard_count, "shards": bins, @@ -211,7 +263,18 @@ def build_manifest( tree_sha: str, modules_to_nodes: dict[str, list[str]], shard_count: int ) -> dict[str, Any]: """Build a canonical manifest with a digest over its executable body.""" - body = _manifest_body(tree_sha, modules_to_nodes, shard_count) + for module, nodes in modules_to_nodes.items(): + if MODULE_RE.fullmatch(module) is None or module == EXCLUDED_MODULE: + raise ShardError("invalid_test_module") + if not nodes or nodes != sorted(set(nodes)): + raise ShardError("invalid_node_inventory") + if any(_module_from_node(node) != module for node in nodes): + raise ShardError("node_module_mismatch") + rows = [ + {"node_signature": _node_signature(nodes), "path": module, "weight": len(nodes)} + for module, nodes in sorted(modules_to_nodes.items()) + ] + body = _manifest_body(tree_sha, rows, shard_count) return {**body, "manifest_sha256": _sha256(_json_bytes(body))} @@ -242,21 +305,15 @@ def validate_manifest(manifest: dict[str, Any]) -> dict[str, Any]: or not isinstance(manifest.get("shards"), list) ): raise ShardError("invalid_manifest") - mapping: dict[str, list[str]] = {} + rows: list[dict[str, Any]] = [] for row in modules: - if not isinstance(row, dict) or set(row) != {"node_ids", "path", "weight"}: + if not isinstance(row, dict) or set(row) != {"node_signature", "path", "weight"}: raise ShardError("invalid_manifest") - path, nodes, weight = row["path"], row["node_ids"], row["weight"] - if not isinstance(path, str) or not isinstance(nodes, list): + if not isinstance(row["path"], str): raise ShardError("invalid_manifest") - if not all(isinstance(node, str) for node in nodes) or weight != len(nodes): - raise ShardError("invalid_manifest") - if path in mapping: - raise ShardError("invalid_manifest") - mapping[path] = nodes - reproduced = build_manifest( - str(manifest.get("tree_sha", "")), mapping, shard_count - ) + rows.append(row) + body = _manifest_body(str(manifest.get("tree_sha", "")), rows, shard_count) + reproduced = {**body, "manifest_sha256": _sha256(_json_bytes(body))} if manifest != reproduced or digest != reproduced["manifest_sha256"]: raise ShardError("manifest_digest_mismatch") return reproduced @@ -307,22 +364,22 @@ def run_shard( manifest = load_manifest(manifest_path) _assert_checked_out_tree(repository_root, manifest["tree_sha"]) shard = _shard(manifest, shard_id) - expected_nodes = sorted( - row["node_ids"] for row in manifest["modules"] if row["path"] in shard["modules"] - ) - expected_flat = sorted(node for group in expected_nodes for node in group) + expected_rows = [row for row in manifest["modules"] if row["path"] in shard["modules"]] _safe_empty_directory(bundle_dir) coverage_path = bundle_dir / "coverage.data" - observed_path = database_metadata.with_name(f"observed-nodes-{shard_id}.jsonl") - if observed_path.exists() or observed_path.is_symlink(): - raise ShardError("observed_nodes_path_exists") - descriptor = os.open(observed_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) - os.close(descriptor) + collected_path = database_metadata.with_name(f"collected-nodes-{shard_id}.jsonl") + completed_path = database_metadata.with_name(f"completed-nodes-{shard_id}.jsonl") + for path in (collected_path, completed_path): + if path.exists() or path.is_symlink(): + raise ShardError("runtime_nodes_path_exists") + descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + os.close(descriptor) env = os.environ.copy() env["COVERAGE_FILE"] = str(coverage_path.resolve()) env["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] = "1" - env[OBSERVED_NODES_ENV] = str(observed_path.resolve()) + env[COLLECTED_NODES_ENV] = str(collected_path.resolve()) + env[COMPLETED_NODES_ENV] = str(completed_path.resolve()) started = time.monotonic() command = [ sys.executable, @@ -337,7 +394,7 @@ def run_shard( "pytest", "-q", *PYTEST_PLUGINS, - *expected_flat, + *(str(PurePosixPath(module).relative_to("backend")) for module in shard["modules"]), "--cov=app", "--cov-report=", ] @@ -345,18 +402,28 @@ def run_shard( if result.returncode != 0: raise ShardError("shard_tests_failed") try: - observed_nodes = sorted( - json.loads(line) - for line in observed_path.read_text(encoding="utf-8").splitlines() - ) - except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: - raise ShardError("invalid_observed_nodes") from exc - if ( - not all(isinstance(node, str) for node in observed_nodes) - or observed_nodes != expected_flat - or len(observed_nodes) != len(set(observed_nodes)) - ): + collected_nodes = _read_node_log(collected_path) + completed_nodes = _read_node_log(completed_path) + except ShardError: + raise + if collected_nodes != completed_nodes or len(collected_nodes) != len(set(collected_nodes)): raise ShardError("shard_execution_mismatch") + runtime_by_module = {module: [] for module in shard["modules"]} + for node in collected_nodes: + module = _module_from_node(node) + if module not in runtime_by_module: + raise ShardError("shard_execution_mismatch") + runtime_by_module[module].append(node) + runtime_signatures = [ + { + "node_signature": _node_signature(runtime_by_module[row["path"]]), + "path": row["path"], + "weight": len(runtime_by_module[row["path"]]), + } + for row in expected_rows + ] + if runtime_signatures != expected_rows: + raise ShardError("shard_signature_mismatch") if not _regular_file(coverage_path): raise ShardError("missing_coverage") coverage_digest = _sha256(coverage_path.read_bytes()) @@ -366,7 +433,8 @@ def run_shard( "duration_seconds": round(time.monotonic() - started, 3), "manifest_sha256": manifest["manifest_sha256"], "modules": shard["modules"], - "observed_node_ids": observed_nodes, + "collected_node_ids": collected_nodes, + "completed_node_ids": completed_nodes, "schema_version": SCHEMA_VERSION, "shard_id": shard_id, "tree_sha": manifest["tree_sha"], @@ -401,10 +469,7 @@ def validate_fan_in( if entries != expected_dirs: raise ShardError("unexpected_bundle_set") - expected_nodes = sorted( - node for module in manifest["modules"] for node in module["node_ids"] - ) - observed_nodes: list[str] = [] + expected_node_count = sum(module["weight"] for module in manifest["modules"]) observed_modules: list[str] = [] coverage_sources: list[Path] = [] durations: list[float] = [] @@ -420,7 +485,8 @@ def validate_fan_in( "duration_seconds", "manifest_sha256", "modules", - "observed_node_ids", + "collected_node_ids", + "completed_node_ids", "schema_version", "shard_id", "tree_sha", @@ -440,26 +506,40 @@ def validate_fan_in( or result["duration_seconds"] < 0 ): raise ShardError("result_provenance_mismatch") - nodes = result["observed_node_ids"] - if not isinstance(nodes, list) or not all(isinstance(node, str) for node in nodes): - raise ShardError("invalid_observed_nodes") - expected_shard_nodes = sorted( - node - for module in manifest["modules"] - if module["path"] in expected_shard["modules"] - for node in module["node_ids"] - ) - if nodes != expected_shard_nodes: + collected = result["collected_node_ids"] + completed = result["completed_node_ids"] + if ( + not isinstance(collected, list) + or not all(isinstance(node, str) for node in collected) + or collected != completed + or collected != sorted(set(collected)) + ): + raise ShardError("shard_node_mismatch") + runtime_by_module = {module: [] for module in expected_shard["modules"]} + for node in collected: + module = _module_from_node(node) + if module not in runtime_by_module: + raise ShardError("shard_node_mismatch") + runtime_by_module[module].append(node) + expected_rows = [ + row for row in manifest["modules"] if row["path"] in expected_shard["modules"] + ] + runtime_rows = [ + { + "node_signature": _node_signature(runtime_by_module[row["path"]]), + "path": row["path"], + "weight": len(runtime_by_module[row["path"]]), + } + for row in expected_rows + ] + if runtime_rows != expected_rows: raise ShardError("shard_node_mismatch") coverage = files["coverage.data"] if result["coverage_sha256"] != _sha256(coverage.read_bytes()): raise ShardError("coverage_digest_mismatch") - observed_nodes.extend(nodes) observed_modules.extend(result["modules"]) coverage_sources.append(coverage) durations.append(float(result["duration_seconds"])) - if sorted(observed_nodes) != expected_nodes or len(observed_nodes) != len(set(observed_nodes)): - raise ShardError("fan_in_node_mismatch") expected_modules = sorted(module["path"] for module in manifest["modules"]) if sorted(observed_modules) != expected_modules or len(observed_modules) != len( set(observed_modules) @@ -478,7 +558,7 @@ def validate_fan_in( { "manifest_sha256": manifest["manifest_sha256"], "module_count": len(expected_modules), - "node_count": len(expected_nodes), + "node_count": expected_node_count, "schema_version": SCHEMA_VERSION, "timing": { "imbalance_seconds": round(max(durations) - min(durations), 3), @@ -516,12 +596,14 @@ def _dry_run(repository_root: Path, shard_count: int) -> None: bundle.mkdir() coverage = bundle / "coverage.data" coverage.write_bytes(f"dry-run-{shard['id']}".encode()) - nodes = sorted( - node + nodes = [ + f"{item['base']}[{index}]" if item["count"] > 1 else item["base"] for module in manifest["modules"] if module["path"] in shard["modules"] - for node in module["node_ids"] - ) + for item in module["node_signature"] + for index in range(item["count"]) + ] + nodes.sort() _write_json( bundle / "result.json", { @@ -530,7 +612,8 @@ def _dry_run(repository_root: Path, shard_count: int) -> None: "duration_seconds": 0, "manifest_sha256": manifest["manifest_sha256"], "modules": shard["modules"], - "observed_node_ids": nodes, + "collected_node_ids": nodes, + "completed_node_ids": nodes, "schema_version": SCHEMA_VERSION, "shard_id": shard["id"], "tree_sha": manifest["tree_sha"], @@ -542,7 +625,7 @@ def _dry_run(repository_root: Path, shard_count: int) -> None: { "manifest_sha256": manifest["manifest_sha256"], "modules": len(manifest["modules"]), - "nodes": sum(len(row["node_ids"]) for row in manifest["modules"]), + "nodes": sum(row["weight"] for row in manifest["modules"]), "shard_weights": [row["weight"] for row in manifest["shards"]], "tree_sha": manifest["tree_sha"], }, diff --git a/backend/tests/test_ci_test_shards.py b/backend/tests/test_ci_test_shards.py index e1c297e7..ff2ff38f 100644 --- a/backend/tests/test_ci_test_shards.py +++ b/backend/tests/test_ci_test_shards.py @@ -42,7 +42,14 @@ def _nodes() -> dict[str, list[str]]: def _write_bundle_set(root: Path, manifest: dict) -> None: root.mkdir() - modules = {row["path"]: row["node_ids"] for row in manifest["modules"]} + modules = { + row["path"]: [ + f"{item['base']}[runtime-{index}]" if item["count"] > 1 else item["base"] + for item in row["node_signature"] + for index in range(item["count"]) + ] + for row in manifest["modules"] + } for shard in manifest["shards"]: bundle = root / f"shard-{shard['id']}" bundle.mkdir() @@ -55,7 +62,8 @@ def _write_bundle_set(root: Path, manifest: dict) -> None: "duration_seconds": 1.25, "manifest_sha256": manifest["manifest_sha256"], "modules": shard["modules"], - "observed_node_ids": observed, + "collected_node_ids": observed, + "completed_node_ids": observed, "schema_version": shards.SCHEMA_VERSION, "shard_id": shard["id"], "tree_sha": manifest["tree_sha"], @@ -159,9 +167,7 @@ def test_collect_nodes_accepts_parameterized_nodes_and_sets_safe_environment( class Result: returncode = 0 stdout = ( - "tests/test_alpha.py::test_a[value]\n" - "tests/test_alpha.py::test_b\n" - "2 tests collected\n" + "tests/test_alpha.py::test_a[value]\ntests/test_alpha.py::test_b\n2 tests collected\n" ) stderr = "" @@ -209,7 +215,7 @@ def test_manifest_file_round_trip_and_invalid_json(tmp_path: Path) -> None: @pytest.mark.parametrize( ("field", "value"), [ - ("schema_version", 2), + ("schema_version", 99), ("shard_count", True), ("manifest_sha256", "bad"), ("excluded_modules", []), @@ -230,13 +236,8 @@ def test_run_shard_uses_python_argv_and_writes_authenticated_bundle( manifest_path = tmp_path / "manifest.json" manifest_path.write_text(json.dumps(manifest), encoding="utf-8") shard = manifest["shards"][0] - module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} + module_nodes = {module: nodes for module, nodes in _nodes().items()} monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) - monkeypatch.setattr( - shards, - "collect_nodes", - lambda root, modules: {module: module_nodes[module] for module in modules}, - ) captured: dict = {} class Result: @@ -246,12 +247,11 @@ def fake_run(command, **kwargs): captured["command"] = command captured["env"] = kwargs["env"] Path(kwargs["env"]["COVERAGE_FILE"]).write_bytes(b"real-coverage") - Path(kwargs["env"][shards.OBSERVED_NODES_ENV]).write_text( - "".join(json.dumps(node) + "\n" for node in sorted( - node for module in shard["modules"] for node in module_nodes[module] - )), - encoding="utf-8", - ) + runtime = sorted(node for module in shard["modules"] for node in module_nodes[module]) + for variable in (shards.COLLECTED_NODES_ENV, shards.COMPLETED_NODES_ENV): + Path(kwargs["env"][variable]).write_text( + "".join(json.dumps(node) + "\n" for node in runtime), encoding="utf-8" + ) return Result() monkeypatch.setattr(shards.subprocess, "run", fake_run) @@ -265,10 +265,12 @@ def fake_run(command, **kwargs): ) result = json.loads((bundle / "result.json").read_text(encoding="utf-8")) assert captured["command"][:2] == [shards.sys.executable, "scripts/run_isolated_tests.py"] - invoked_nodes = [argument for argument in captured["command"] if argument.startswith("tests/")] - assert invoked_nodes == sorted( - node for module in shard["modules"] for node in module_nodes[module] - ) + invoked_modules = [ + argument for argument in captured["command"] if argument.startswith("tests/") + ] + assert invoked_modules == [ + str(Path(module).relative_to("backend")) for module in shard["modules"] + ] assert "scripts.ci_test_shards" in captured["command"] assert captured["env"]["PYTEST_DISABLE_PLUGIN_AUTOLOAD"] == "1" assert result["modules"] == shard["modules"] @@ -282,13 +284,7 @@ def test_run_shard_rejects_failed_test_process( path = tmp_path / "manifest.json" path.write_text(json.dumps(manifest), encoding="utf-8") shard = manifest["shards"][0] - module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) - monkeypatch.setattr( - shards, - "collect_nodes", - lambda root, modules: {module: module_nodes[module] for module in modules}, - ) class Result: returncode = 1 @@ -305,7 +301,7 @@ def test_run_shard_rejects_collected_but_not_executed_node( path = tmp_path / "manifest.json" path.write_text(json.dumps(manifest), encoding="utf-8") shard = manifest["shards"][0] - module_nodes = {row["path"]: row["node_ids"] for row in manifest["modules"]} + module_nodes = _nodes() monkeypatch.setattr(shards, "_assert_checked_out_tree", lambda *args: None) class Result: @@ -313,10 +309,12 @@ class Result: def fake_run(command, **kwargs): Path(kwargs["env"]["COVERAGE_FILE"]).write_bytes(b"coverage") - expected = sorted( - node for module in shard["modules"] for node in module_nodes[module] + expected = sorted(node for module in shard["modules"] for node in module_nodes[module]) + Path(kwargs["env"][shards.COLLECTED_NODES_ENV]).write_text( + "".join(json.dumps(node) + "\n" for node in expected), + encoding="utf-8", ) - Path(kwargs["env"][shards.OBSERVED_NODES_ENV]).write_text( + Path(kwargs["env"][shards.COMPLETED_NODES_ENV]).write_text( "".join(json.dumps(node) + "\n" for node in expected[1:]), encoding="utf-8", ) @@ -327,17 +325,53 @@ def fake_run(command, **kwargs): shards.run_shard(tmp_path, path, shard["id"], tmp_path / "bundle", tmp_path / "db") -def test_pytest_hook_records_only_when_execution_destination_exists( +def test_pytest_hooks_record_runtime_collection_and_completion( monkeypatch: pytest.MonkeyPatch, tmp_path: Path ) -> None: - destination = tmp_path / "observed.jsonl" - destination.write_text("", encoding="utf-8") - monkeypatch.setenv(shards.OBSERVED_NODES_ENV, str(destination)) + collected = tmp_path / "collected.jsonl" + completed = tmp_path / "completed.jsonl" + collected.write_text("", encoding="utf-8") + completed.write_text("", encoding="utf-8") + monkeypatch.setenv(shards.COLLECTED_NODES_ENV, str(collected)) + monkeypatch.setenv(shards.COMPLETED_NODES_ENV, str(completed)) + session = type( + "Session", (), {"items": [type("Item", (), {"nodeid": "tests/test_alpha.py::test_a"})()]} + )() + shards.pytest_collection_finish(session) shards.pytest_runtest_logfinish("tests/test_alpha.py::test_a", ("file", 1, "test_a")) - assert destination.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' - monkeypatch.delenv(shards.OBSERVED_NODES_ENV) - shards.pytest_runtest_logfinish("tests/test_alpha.py::test_b", ("file", 2, "test_b")) - assert destination.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' + assert collected.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' + assert completed.read_text(encoding="utf-8") == '"tests/test_alpha.py::test_a"\n' + + +def test_runtime_signature_tolerates_changed_parameter_display_values() -> None: + before = ["tests/test_alpha.py::test_a[550e8400-e29b-41d4-a716-446655440000]"] + after = ["tests/test_alpha.py::test_a[6ba7b810-9dad-11d1-80b4-00c04fd430c8]"] + assert shards._node_signature(before) == shards._node_signature(after) + first = shards.build_manifest( + TREE_SHA, + { + "backend/tests/test_alpha.py": before, + **{ + key: value + for key, value in _nodes().items() + if key != "backend/tests/test_alpha.py" + }, + }, + 4, + ) + second = shards.build_manifest( + TREE_SHA, + { + "backend/tests/test_alpha.py": after, + **{ + key: value + for key, value in _nodes().items() + if key != "backend/tests/test_alpha.py" + }, + }, + 4, + ) + assert first == second def test_fan_in_accepts_exact_authenticated_bundles(tmp_path: Path) -> None: @@ -345,9 +379,7 @@ def test_fan_in_accepts_exact_authenticated_bundles(tmp_path: Path) -> None: bundles = tmp_path / "bundles" _write_bundle_set(bundles, manifest) summary_path = tmp_path / "fan-in-summary.json" - outputs = shards.validate_fan_in( - manifest, bundles, tmp_path / "combined", summary_path - ) + outputs = shards.validate_fan_in(manifest, bundles, tmp_path / "combined", summary_path) assert [path.name for path in outputs] == [ ".coverage.shard-1", ".coverage.shard-2", @@ -367,9 +399,7 @@ def test_fan_in_accepts_exact_authenticated_bundles(tmp_path: Path) -> None: @pytest.mark.parametrize("mutation", ["missing", "extra", "coverage", "nodes", "tree"]) -def test_fan_in_rejects_incomplete_or_tampered_evidence( - tmp_path: Path, mutation: str -) -> None: +def test_fan_in_rejects_incomplete_or_tampered_evidence(tmp_path: Path, mutation: str) -> None: manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) bundles = tmp_path / "bundles" _write_bundle_set(bundles, manifest) @@ -383,7 +413,7 @@ def test_fan_in_rejects_incomplete_or_tampered_evidence( path = bundles / "shard-1/result.json" result = json.loads(path.read_text(encoding="utf-8")) if mutation == "nodes": - result["observed_node_ids"] = result["observed_node_ids"][1:] + result["completed_node_ids"] = result["completed_node_ids"][1:] else: result["tree_sha"] = "b" * 40 path.write_text(json.dumps(result), encoding="utf-8") @@ -406,15 +436,13 @@ def test_fan_in_rejects_symlink_and_unexpected_file(tmp_path: Path) -> None: shards.validate_fan_in(manifest, bundles, tmp_path / "combined") -def test_fan_in_rejects_duplicate_node_across_shards(tmp_path: Path) -> None: +def test_fan_in_rejects_runtime_collection_completion_mismatch(tmp_path: Path) -> None: manifest = shards.build_manifest(TREE_SHA, _nodes(), 4) bundles = tmp_path / "bundles" _write_bundle_set(bundles, manifest) - first = bundles / "shard-1/result.json" second = bundles / "shard-2/result.json" - one = json.loads(first.read_text(encoding="utf-8")) two = json.loads(second.read_text(encoding="utf-8")) - two["observed_node_ids"][0] = one["observed_node_ids"][0] + two["completed_node_ids"] = two["completed_node_ids"][1:] second.write_text(json.dumps(two), encoding="utf-8") with pytest.raises(shards.ShardError, match="shard_node_mismatch"): shards.validate_fan_in(manifest, bundles, tmp_path / "combined") diff --git a/docs/operations_backend_testing.md b/docs/operations_backend_testing.md index f83b69a0..3fb88f93 100644 --- a/docs/operations_backend_testing.md +++ b/docs/operations_backend_testing.md @@ -80,9 +80,11 @@ all evidence is authenticated. The preflight plan and four fixed shard bundles are retained for seven days. Their names include the actual checked-out tree SHA. Each shard bundle contains only `coverage.data` and allowlisted `result.json`; the result binds the tree, -manifest, shard, modules, completed pytest node IDs, duration, and SHA-256 of the -exact coverage bytes. A repository-owned pytest hook records a node only after -its runtime lifecycle finishes. Bundles never contain database URLs or passwords, MinIO +manifest, shard, modules, collected and completed pytest node IDs, duration, and +SHA-256 of the exact coverage bytes. Repository-owned pytest hooks record the +final selected inventory and lifecycle completions in the same process; fan-in +requires exact equality and matches stable test-base cardinalities to preflight. +Bundles never contain database URLs or passwords, MinIO credentials, environment dumps, or runner database metadata. The fan-in accepts exactly four expected regular-file bundles. It rejects stale From ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:08:51 +0100 Subject: [PATCH 08/12] test(ci): cover structured parameter ids --- backend/scripts/ci_test_shards.py | 5 ++--- backend/tests/test_ci_test_shards.py | 4 ++-- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/backend/scripts/ci_test_shards.py b/backend/scripts/ci_test_shards.py index 20acf481..50974a33 100644 --- a/backend/scripts/ci_test_shards.py +++ b/backend/scripts/ci_test_shards.py @@ -130,9 +130,8 @@ def _module_from_node(node_id: str) -> str: def _node_base(node_id: str) -> str: """Remove only pytest's final parameter display value from a node ID.""" _module_from_node(node_id) - leaf = node_id.rsplit("::", 1)[-1] - if leaf.endswith("]") and "[" in leaf: - return node_id[: -(len(leaf) - leaf.find("["))] + if node_id.endswith("]") and "[" in node_id: + return node_id.split("[", 1)[0] return node_id diff --git a/backend/tests/test_ci_test_shards.py b/backend/tests/test_ci_test_shards.py index ff2ff38f..d2506a40 100644 --- a/backend/tests/test_ci_test_shards.py +++ b/backend/tests/test_ci_test_shards.py @@ -344,8 +344,8 @@ def test_pytest_hooks_record_runtime_collection_and_completion( def test_runtime_signature_tolerates_changed_parameter_display_values() -> None: - before = ["tests/test_alpha.py::test_a[550e8400-e29b-41d4-a716-446655440000]"] - after = ["tests/test_alpha.py::test_a[6ba7b810-9dad-11d1-80b4-00c04fd430c8]"] + before = ["tests/test_alpha.py::test_a[550e8400::nested[value]]"] + after = ["tests/test_alpha.py::test_a[6ba7b810::other[value]]"] assert shards._node_signature(before) == shards._node_signature(after) first = shards.build_manifest( TREE_SHA, From f2f35493681ad45ff7e249de0e99b210639cf684 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:12:25 +0100 Subject: [PATCH 09/12] docs(agent-loop): disclose hosted shard repair --- .../WS-CI-001-01-internal-review-evidence.md | 41 ++++++++++-------- .../reviews/WS-CI-001-01-pr-trust-bundle.md | 42 +++++++++++++++---- 2 files changed, 57 insertions(+), 26 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md index f3aacec6..98220bbd 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -4,16 +4,16 @@ Open sub-agent sessions: none Valid findings addressed: yes -Reviewed code SHA: `a6141d5b7155c178d533e718404cb04576c900d7` +Reviewed code SHA: `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b` -Reviewed at: 2026-07-20T16:23:40Z +Reviewed at: 2026-07-20T17:11:39Z -Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_impl_arch`; QA/test/CI-integrity/test-delta=`ci_impl_qa`; security/auth/product/ops/docs=`ci_impl_sec_ops` +Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_repair_arch`; QA/test/CI-integrity/test-delta=`ci_repair_qa`; security/auth/product/ops/docs=`ci_repair_sec_ops` ## Binding - Base: `c12ba1c8d4bbde86d0e2c19826f5791afc130489` -- Final reviewed implementation: `14c50b464efca95da4f57b30272e0ce7e0435c11` +- Final reviewed implementation: `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b` - Contract: `WS-CI-001-01 — Parallel Full-Suite Coverage` - Scope: CI workflow, one shared shard tool, additive tests, operator runbook, initiative evidence, and exactly one merge intent; no product runtime change @@ -25,20 +25,24 @@ Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_impl_arch`; QA | senior engineering | PASS | None | Cohesive inventory-to-fan-in trust boundary. | | QA/test | PASS AFTER FIXES | None | Exact runtime-finished node proof replaces pre-run collection claims. | | security/auth | PASS AFTER FIXES | None | Fixed artifacts, byte hashes, read-only permissions, safe hook path. | -| product/ops | PASS AFTER FIXES | None | Authenticated timing/cost evidence now reaches operators. | +| product/ops | PENDING CONFIRMATION | Stale trust bundle | This evidence-only correction discloses the hosted failure and repaired SHA. | | architecture | PASS | None | Shared Python policy boundary; YAML only orchestrates. | | CI integrity | PASS AFTER FIXES | None | Stable required check and unchanged 78/90 thresholds. | -| docs | PASS AFTER FIXES | None | Runbook matches timing report and completed-node semantics. | +| docs | PENDING CONFIRMATION | Stale trust bundle | Runbook is accurate; repaired PR evidence now requires confirmation. | | reuse/dedup | PASS | None | Existing isolated database runner remains sole DB owner. | | test delta | PASS AFTER FIXES | None | Additive tests; no skip, assertion, or threshold weakening. | ## Findings repaired -- Replaced shard metadata derived from a second collection with exact manifest - node argv and a repository-owned pytest hook recording each node after its - actual runtime lifecycle finishes. A successful shard requires the completed - set to equal the canonical manifest exactly once. -- Added a regression proving a collected-but-not-executed node fails. +- Hosted run `29759523305` proved raw parameter display IDs can change across + pytest processes. Replaced cross-process raw-node execution with validated + whole-module execution, exact same-process collection/completion equality, + and stable preflight test-base cardinality binding. +- Added regressions for changed UUID-like/nested parameter displays and for a + collected node that does not complete. +- Disclosed the failed/cancelled hosted run and rebound the PR trust bundle to + the repaired implementation SHA; hosted rerun remains pending. + - Added authenticated fan-in summary containing tree, manifest, per-shard duration/node/module balance, total runner seconds, and timing imbalance; the final check prints and uploads it through a pinned action. @@ -50,13 +54,15 @@ Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_impl_arch`; QA ```text Ruff: passed Python compilation: passed -planner + agent-gate + coverage-contract tests: 294 passed -isolated-runner + coverage-contract tests: 188 passed -clean-environment real collection/fan-in dry run: 31 modules, 1774 nodes -shard weights: 443 / 444 / 443 / 444 +shard + coverage-contract tests at repaired head: 204 passed +agent-gate tests at repaired head: 91 passed +clean-environment collection/fan-in dry run: 31 modules, 1775 nodes +shard weights: 445 / 444 / 443 / 443 +real local shard diagnostic: 445 collected, 246 completed before deliberate interrupt +diagnostic database and role cleanup: verified absent merge-intent validation: passed loop-memory state validation: passed -Markdown links: passed for 11 changed Markdown files +Markdown links: passed for 13 changed Markdown files stale Workstream/authorization/artifact/review scans: passed git diff --check: passed ``` @@ -64,7 +70,8 @@ git diff --check: passed The static sensor reports `REVIEW_REQUIRED` for a large L1 CI diff and the moved global-threshold enforcement point. This is expected review routing, not a bypass: the final authenticated fan-in enforces the same 78 percent global floor and all -twelve 90 percent subsystem floors, and all nine reviewer tracks passed. +twelve 90 percent subsystem floors. Seven reviewer tracks passed; product/ops +and docs require confirmation of this evidence-only correction. ## Remaining Gate diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index 2191c5b6..5deaf5ac 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -30,10 +30,18 @@ No path-based workflow suppression was added. ## Evidence -- 294 focused planner/agent-gate/coverage-contract tests pass. -- 188 isolated-runner/coverage-contract tests pass against local PostgreSQL. -- Clean installed environment discovers 31 modules and 1,774 nodes and validates - complete four-shard fan-in at weights 443/444/443/444. +- At repaired implementation SHA `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b`, + 204 shard/coverage-contract tests and 91 agent-gate tests pass; Ruff, + compilation, loop-state, stale scans, Markdown links, and diff integrity pass. +- The repaired clean-environment dry run discovers 31 modules and 1,775 nodes + and validates complete four-shard fan-in at weights 445/444/443/443. +- A real local shard proved 445 same-process nodes were collected and whole + module paths—not preflight parameter IDs—reached pytest. It was intentionally + interrupted after 246 completions because runtime was no longer a + proportionate local check; its exact owned database and role were removed and + verified absent. This is diagnostic evidence, not a passing full-shard claim. +- Earlier unchanged isolated-runner/coverage-contract proof passed 188 tests + against local PostgreSQL before the hosted repair. - Ruff, compilation, merge-intent, loop-state, stale scans, Markdown links, and diff integrity pass. - Actions are SHA-pinned; PostgreSQL and MinIO are digest-pinned. @@ -41,14 +49,30 @@ No path-based workflow suppression was added. ## Internal Review -Implementation SHA `14c50b464efca95da4f57b30272e0ce7e0435c11` and final evidence head -`a6141d5b7155c178d533e718404cb04576c900d7` pass senior engineering, QA/test, -security/auth, product/ops, architecture, CI integrity, docs, reuse/dedup, and -test-delta review after all valid findings were repaired. +Repaired implementation SHA `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b` +passes senior engineering, QA/test, security/auth, architecture, CI integrity, +reuse/dedup, and test-delta review. Product/ops and docs identified this stale +trust bundle as blocking; this evidence-only correction records their finding +and requires exact-head confirmation before push. + +## Hosted Failure and Repair + +GitHub Actions run `29759523305` reached green preflight and API E2E, then shard +jobs failed because import-time UUID parameter display values changed between +the preflight and shard pytest processes. The invalid run was cancelled after +root cause confirmation to avoid wasting runner minutes. It is not success +evidence. + +The repair no longer executes raw cross-process node IDs. It executes validated +whole modules, proves exact final collection equals exact completion inside the +same pytest process, and binds stable test-base cardinalities to preflight. A +new hosted run has not started yet; exact-head hosted success and timing remain +mandatory before human merge approval. ## Remaining Risk -Collected-node counts may not predict runtime for migration-heavy modules. +Collected-node counts may not predict runtime for migration-heavy modules; the +partial local shard reinforced that risk without proving hosted wall time. Hosted execution must show actual per-shard durations, wall-clock improvement, and aggregate runner cost. The workflow fails closed rather than silently falling back if sharding is unstable. From 76b1d242878c49353178091e41f0435a6c9885d8 Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:13:39 +0100 Subject: [PATCH 10/12] docs(agent-loop): close shard repair reviews --- .../reviews/WS-CI-001-01-internal-review-evidence.md | 10 ++++++---- .../reviews/WS-CI-001-01-pr-trust-bundle.md | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md index 98220bbd..e66dd924 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -6,6 +6,8 @@ Valid findings addressed: yes Reviewed code SHA: `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b` +Reviewed evidence SHA: `f2f35493681ad45ff7e249de0e99b210639cf684` + Reviewed at: 2026-07-20T17:11:39Z Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_repair_arch`; QA/test/CI-integrity/test-delta=`ci_repair_qa`; security/auth/product/ops/docs=`ci_repair_sec_ops` @@ -25,10 +27,10 @@ Reviewer run IDs: senior-engineering/architecture/reuse-dedup=`ci_repair_arch`; | senior engineering | PASS | None | Cohesive inventory-to-fan-in trust boundary. | | QA/test | PASS AFTER FIXES | None | Exact runtime-finished node proof replaces pre-run collection claims. | | security/auth | PASS AFTER FIXES | None | Fixed artifacts, byte hashes, read-only permissions, safe hook path. | -| product/ops | PENDING CONFIRMATION | Stale trust bundle | This evidence-only correction discloses the hosted failure and repaired SHA. | +| product/ops | PASS AFTER FIXES | None | Hosted failure, diagnostic limit, repair, and remaining gate are disclosed. | | architecture | PASS | None | Shared Python policy boundary; YAML only orchestrates. | | CI integrity | PASS AFTER FIXES | None | Stable required check and unchanged 78/90 thresholds. | -| docs | PENDING CONFIRMATION | Stale trust bundle | Runbook is accurate; repaired PR evidence now requires confirmation. | +| docs | PASS AFTER FIXES | None | Runbook and repaired PR evidence accurately describe current state. | | reuse/dedup | PASS | None | Existing isolated database runner remains sole DB owner. | | test delta | PASS AFTER FIXES | None | Additive tests; no skip, assertion, or threshold weakening. | @@ -70,8 +72,8 @@ git diff --check: passed The static sensor reports `REVIEW_REQUIRED` for a large L1 CI diff and the moved global-threshold enforcement point. This is expected review routing, not a bypass: the final authenticated fan-in enforces the same 78 percent global floor and all -twelve 90 percent subsystem floors. Seven reviewer tracks passed; product/ops -and docs require confirmation of this evidence-only correction. +twelve 90 percent subsystem floors, and all nine reviewer tracks passed after +valid findings were repaired. ## Remaining Gate diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index 5deaf5ac..483a1284 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -51,9 +51,10 @@ No path-based workflow suppression was added. Repaired implementation SHA `ab75b96ba5eba3fad0f127dc1b892c3a804b2c7b` passes senior engineering, QA/test, security/auth, architecture, CI integrity, -reuse/dedup, and test-delta review. Product/ops and docs identified this stale -trust bundle as blocking; this evidence-only correction records their finding -and requires exact-head confirmation before push. +reuse/dedup, and test-delta review. Product/ops and docs identified the stale +trust bundle as blocking, then passed evidence SHA +`f2f35493681ad45ff7e249de0e99b210639cf684` after the hosted failure, repaired +proof model, diagnostic limitation, and remaining gate were recorded. ## Hosted Failure and Repair From 3a0af386422a140372d3e9f0123ecd732078005e Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:32:28 +0100 Subject: [PATCH 11/12] docs(agent-loop): record hosted CI acceleration --- .../STATUS.md | 5 ++--- .../WS-CI-001-01-internal-review-evidence.md | 14 +++++++++--- .../reviews/WS-CI-001-01-pr-trust-bundle.md | 22 +++++++++++++------ 3 files changed, 28 insertions(+), 13 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md index 425bfbff..f52a98e5 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/STATUS.md @@ -1,10 +1,9 @@ # STATUS: WS-CI-001 - Backend CI Acceleration -- Phase: hosted-failure repair and renewed internal review +- Phase: implementation proof complete; awaiting human PR review - Active implementation chunk: `WS-CI-001-01` - Proposed first chunk: `WS-CI-001-01` - Human direction: prioritize full-suite CI acceleration before `WS-ENG-001-04B` -- Current gate: repair evidence, renewed internal review, hosted matrix proof, - and human review +- Current gate: final evidence-only exact-head CI and human PR review - `WS-ENG-001-04B`: remains inactive diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md index e66dd924..fea712af 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -62,6 +62,13 @@ clean-environment collection/fan-in dry run: 31 modules, 1775 nodes shard weights: 445 / 444 / 443 / 443 real local shard diagnostic: 445 collected, 246 completed before deliberate interrupt diagnostic database and role cleanup: verified absent +hosted repaired Backend run 29762868497: passed +hosted Agent Gates run 29762868454: passed +hosted inventory: 31 modules / 1775 nodes +hosted shard seconds: 711.782 / 135.947 / 130.978 / 411.147 +hosted imbalance: 580.804 seconds +hosted aggregate shard runner time: 1389.854 seconds +hosted Backend wall time: approximately 14 minutes 40 seconds merge-intent validation: passed loop-memory state validation: passed Markdown links: passed for 13 changed Markdown files @@ -77,6 +84,7 @@ valid findings were repaired. ## Remaining Gate -The new GitHub matrix and actual latency/cost can only be proven on the pushed PR -head. Hosted `Backend / test`, Agent Gates, external review, and human merge -approval remain mandatory. `WS-CI-001-02` and `WS-ENG-001-04B` are inactive. +Hosted `Backend / test`, Agent Gates, and CodeRabbit passed on the repaired +branch. The evidence-only commit recording that result must receive final +exact-head CI, then human review and explicit merge approval remain mandatory. +`WS-CI-001-02` and `WS-ENG-001-04B` are inactive. diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index 483a1284..52f5761d 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -67,16 +67,24 @@ evidence. The repair no longer executes raw cross-process node IDs. It executes validated whole modules, proves exact final collection equals exact completion inside the same pytest process, and binds stable test-base cardinalities to preflight. A -new hosted run has not started yet; exact-head hosted success and timing remain -mandatory before human merge approval. +hosted rerun at branch head `76b1d242878c49353178091e41f0435a6c9885d8` +passed Agent Gates, CodeRabbit, preflight, API E2E, all four shards, complete +fan-in, the 78 percent repository floor, and all twelve 90 percent subsystem +floors. The Backend run completed in about 14 minutes 40 seconds versus about +27 minutes on PR #161. + +Authenticated fan-in recorded 31 modules and 1,775 nodes. Shard execution was +711.782, 135.947, 130.978, and 411.147 seconds, with 580.804 seconds imbalance +and 1,389.854 aggregate runner seconds. This proves the acceleration and also +supports later, separately approved runtime-weight routing; it does not activate +`WS-CI-001-02`. ## Remaining Risk -Collected-node counts may not predict runtime for migration-heavy modules; the -partial local shard reinforced that risk without proving hosted wall time. -Hosted execution must show actual per-shard durations, wall-clock improvement, -and aggregate runner cost. The workflow fails closed rather than silently falling -back if sharding is unstable. +Collected-node counts do not predict runtime well for migration-heavy modules: +hosted shard durations ranged from about 131 to 712 seconds despite nearly equal +node counts. The workflow still fails closed rather than silently falling back. +Runtime-weight routing remains a separately approved follow-up decision. ## Human Review Focus From 1fa072b89407e6a705d85eb06cd8ac26f774e4fd Mon Sep 17 00:00:00 2001 From: Abiorh001 Date: Mon, 20 Jul 2026 18:34:56 +0100 Subject: [PATCH 12/12] docs(agent-loop): clarify hosted tree provenance --- .../reviews/WS-CI-001-01-internal-review-evidence.md | 2 ++ .../reviews/WS-CI-001-01-pr-trust-bundle.md | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md index fea712af..21fcfab7 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-internal-review-evidence.md @@ -64,6 +64,8 @@ real local shard diagnostic: 445 collected, 246 completed before deliberate inte diagnostic database and role cleanup: verified absent hosted repaired Backend run 29762868497: passed hosted Agent Gates run 29762868454: passed +hosted PR branch head: 76b1d242878c49353178091e41f0435a6c9885d8 +hosted authenticated checked-out merge tree: c2a0ce7c13bb76f6a769dcbca0726de8b244431e hosted inventory: 31 modules / 1775 nodes hosted shard seconds: 711.782 / 135.947 / 130.978 / 411.147 hosted imbalance: 580.804 seconds diff --git a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md index 52f5761d..3bc7d235 100644 --- a/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md +++ b/.agent-loop/initiatives/WS-CI-001-backend-ci-acceleration/reviews/WS-CI-001-01-pr-trust-bundle.md @@ -67,11 +67,12 @@ evidence. The repair no longer executes raw cross-process node IDs. It executes validated whole modules, proves exact final collection equals exact completion inside the same pytest process, and binds stable test-base cardinalities to preflight. A -hosted rerun at branch head `76b1d242878c49353178091e41f0435a6c9885d8` -passed Agent Gates, CodeRabbit, preflight, API E2E, all four shards, complete -fan-in, the 78 percent repository floor, and all twelve 90 percent subsystem -floors. The Backend run completed in about 14 minutes 40 seconds versus about -27 minutes on PR #161. +hosted rerun associated with PR branch head +`76b1d242878c49353178091e41f0435a6c9885d8`, executing authenticated checked-out +merge tree `c2a0ce7c13bb76f6a769dcbca0726de8b244431e`, passed Agent Gates, +CodeRabbit, preflight, API E2E, all four shards, complete fan-in, the 78 percent +repository floor, and all twelve 90 percent subsystem floors. The Backend run +completed in about 14 minutes 40 seconds versus about 27 minutes on PR #161. Authenticated fan-in recorded 31 modules and 1,775 nodes. Shard execution was 711.782, 135.947, 130.978, and 411.147 seconds, with 580.804 seconds imbalance