Skip to content

CI: reduce repeated verification overhead #105

Description

@noeltock

Problem

Routine changes wait on repeated WordPress environment setup, and documentation/demo pushes to main still run the full CI matrix. The v0.9.6 CI run spent 17m 57s in the WordPress job, including 13m 45s in automated acceptance. The real-WordPress test file accounted for 12m 57s of that step. Across four recent runs, the WordPress job took 17m 13s to 18m 26s.

The current proof cases stop the environment between sequential tests. Each case still needs to install and verify its exact generated ZIP, but rebuilding the environment lifecycle between cases may be avoidable. Separately, README/demo-only main pushes pay for package and browser checks that their changes do not affect.

Approach

Change Current behavior Proposed behavior
Sequential WordPress proofs Stop between cases Retain the environment, reinstall each exact ZIP, run every existing assertion, then clean up
Docs/demo pushes to main Full matrix Classify the complete before/after range and run focused contracts for explicitly allowlisted paths
Runtime, dependency, unknown, deleted or renamed paths Full checks Continue requiring full checks

Keep the release workflow on its existing full checks. Preserve an environment that was already running before a local proof suite began; an unknown status must not grant cleanup ownership.

Proposed control flow:

 main push
-  run full CI for every path
+  classify the complete before..after range
+  added/modified allowlisted docs/demo → focused contracts
+  runtime, dependency or uncertain changes → full checks or visible classification failure

 sequential WordPress proofs
+  inspect initial environment status and establish cleanup ownership
   for each case
     stage and install its exact ZIP
     run browser assertions and retain its receipt
-    stop WordPress
+    retain WordPress for the next case
+  final cleanup → stop only an environment owned by this suite

Implementation entry points: .github/workflows/ci.yml, scripts/ci-scope.mjs and dev/test/proof-real-wordpress.test.ts. Regression coverage belongs alongside the classifier and lifecycle helper. PR #106 implements this scope.

Acceptance criteria

  • Every existing WordPress browser assertion, profile gate and artifact receipt still runs against the correct ZIP.
  • Cleanup covers failure paths and preserves pre-existing environments.
  • PR and main-push classification uses the complete event range; unavailable diffs fail closed and missing/zero boundaries select full checks.
  • Docs/demo-only changes run focused documentation, shell and demo contracts; mixed runtime changes take the full route.
  • The final CI scope check requires every selected job and accepts only intentional skips.
  • Full CI passes on the implementation revision, with timings compared against the linked baseline. Report observed savings separately from work intentionally skipped.

Scope

This issue covers environment reuse and conservative path routing. CLI-test sharding, packed-consumer install reuse, compiler/linter replacement and release-check reuse are separate follow-ups. No package release is needed for these CI/test changes.

Implementation evidence

PR #106, revision 57f1516, passes full CI: Node 20/22/24, packed consumers, all 9 WordPress tests and the final scope gate. Focused local checks passed: classifier 9/9, lifecycle/staged-ZIP setup 11/11, documentation contracts 9/9 and typecheck. PR/push classifier-shell scenarios and required-job aggregation checks produced the expected outcomes.

The WordPress job changed from 17m 57s to 12m 10s; acceptance changed from 13m 45s to 7m 31s. This is one observed before/after comparison, with runner variability, not a guaranteed percentage improvement. The smaller route has not yet run as a live main-push event; its workflow shell has been exercised locally. The PR remains open for merge.

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions