Skip to content

[CI] Optimize pr-ci.yml and hack/ci with Change-Aware Test Selection, PR Smoke Tiering, and Build Decoupling #1813

Closed

Description

@skevetter

Context & Problem Statement

PRs currently take 45–75+ minutes to pass status checks in .github/workflows/pr-ci.yml. Even minor or isolated changes trigger all 87 jobs (including 72 E2E test suites across Linux Docker/Kind, Linux Podman, Linux MicroSandbox KVM, and Windows Podman/Kind/Docker-WSL) because:

  1. Single Coarse go Filter: changes in .github/workflows/pr-ci.yml (L23–L50) only outputs a single go boolean. Touching any .go file triggers all 72 E2E suites, provider-update-windows, github-credential-host-tests (3 OSes), and licenses.
  2. Unrelated Long-Tail E2E Suites Run on Every PR:
    • up-provider-kubernetes-inactivity has a 60m test timeout (75m job timeout) and runs on every Go PR even when Kubernetes or inactivity code is untouched.
    • All 16 microsandbox KVM suites, 10 podman suites, and 8 windows-latest E2E suites run on every Go PR even when touching isolated packages like pkg/mcp, pkg/ide, pkg/compose, or a single e2e/tests/<suite> package.
  3. Critical-Path Serialization in build-cli:
    • build-cli (L222–L225) has needs: [changes, precommit, lint], delaying all E2E binary compilation by ~4–6 minutes until both precommit and lint finish.
    • build-cli runs a static matrix over ubuntu-latest, macos-latest, and windows-latest. All 64 Linux E2E jobs block until the slow windows-latest GoReleaser + e2e.test.exe build completes, and macos-latest builds devsy-darwin even though zero E2E jobs consume it.
  4. Redundant Linting & Unfiltered Jobs on Non-Go PRs:
    • precommit runs prek run --all-files (which runs golangci-lint v2.13.2 across the repo) while lint runs golangci-lint v2.14.0 again.
    • lint, agent-skill, and plan run even on docs-only (sites/**) or desktop-only (desktop/**) PRs.

Goal

Reduce PR CI turnaround time while maintaining high confidence:

  • push to main and workflow_dispatch: Run 100% of checks and all 72 E2E suites.
  • pull_request:
    • Non-Go PRs (sites/**, desktop/**, docs): Skip Go lint, unit, build, and E2E jobs.
    • Isolated Subsystem / Leaf Go PRs: Run unit tests + only the E2E suites mapped to the modified packages/files.
    • Core / Shared Go PRs: Run unit tests + a curated 13-suite Core Smoke Tier (pr_smoke: true) spanning every runtime (Linux unprivileged, Docker, Compose, Features, SSH, Build, Kubernetes kind, Podman rootless/rootful, MicroSandbox KVM parity, Windows WSL) plus any directly matched suites.
    • CI / Build / Dependency PRs (.github/workflows/pr-ci.yml, hack/ci/**, go.mod, go.sum, .goreleaser.yml, e2e/framework/**): Run the full 72-suite E2E matrix.
  • Deliver this as a single PR with clean, atomic commits executed by an agent team / subagents.

Architecture Overview

flowchart TD
    subgraph Discovery["1. Change Detection & Planning"]
        Changes["changes (dorny/paths-filter + changed files list)"]
        Plan["plan (hack/ci matrix --mode=pr|full)"]
        Changes -->|changed files + filter flags| Plan
    end

    subgraph FastChecks["2. Parallel Fast Checks"]
        Precommit["precommit (prek, skip duplicate golangci-lint)"]
        Lint["lint (if: changes.go == 'true')"]
        Skill["agent-skill (if: changes.agent_skill == 'true')"]
        Licenses["licenses (if: changes.licenses == 'true')"]
        Unit["unit-tests (if: changes.go == 'true')"]
        GHHost["github-credential-host-tests (if: changes.gh_credentials == 'true')"]
        ProvWin["provider-update-windows (if: changes.provider_update == 'true')"]
    end

    subgraph Build["3. Decoupled Dynamic CLI & E2E Build"]
        BuildCLI["build-cli (needs: [changes, plan], dynamic OS matrix from plan.build)"]
        Plan -->|plan.build != empty| BuildCLI
    end

    subgraph E2E["4. Change-Selected E2E Tiers"]
        Unpriv["integration-tests-unprivileged (if: plan.unprivileged != empty)"]
        Integ["integration-tests (if: plan.integration != empty)"]
        PrepMSB["prepare-microsandbox-provider (if: plan.microsandbox != empty)"]
        MSB["integration-tests-microsandbox (if: plan.microsandbox != empty)"]
        BuildCLI --> Unpriv
        BuildCLI --> Integ
        BuildCLI --> PrepMSB --> MSB
    end

    subgraph Gate["5. Fail-Closed Required Status Gate"]
        CISuccess["ci-success (hack/ci success validates all jobs against changes + plan outputs)"]
    end

    Precommit & Lint & Skill & Licenses & Unit & GHHost & ProvWin --> CISuccess
    Unpriv & Integ & PrepMSB & MSB --> CISuccess
Loading

Subagent & Commit Decomposition Plan

Execute this change in 4 tasks (1 commit per task) within a single PR:

  1. Task 1 (Subagent A — Manifest & Selector Engine): Extend hack/ci/suites.yaml and hack/ci/manifest.go with selection, pr_smoke, and paths, and implement hack/ci/select.go + unit/coverage tests (commit 1: feat(ci): add suite path mappings and PR smoke selection rules).
  2. Task 2 (Subagent B — Dynamic Matrix & Build Planner): Update hack/ci/matrix.go and hack/ci/matrix_test.go to support --mode=full|pr, --changed-files, and the dynamic build matrix group (commit 2: feat(ci): add change-aware matrix and build planning to hack/ci).
  3. Task 3 (Subagent C — Gate Evaluator): Update hack/ci/success.go and hack/ci/success_test.go to validate per-domain changes outputs and empty plan matrix skips fail-closed (commit 3: feat(ci): support conditional job and empty matrix skips in ci-success).
  4. Task 4 (Subagent D — Workflow Wiring): Update .github/workflows/pr-ci.yml to wire granular path filters, plan mode/changed-files, decoupled dynamic build-cli, empty-matrix if: guards, and SKIP=golangci-lint in precommit (commit 4: ci: wire change-aware test selection and decoupled builds in pr-ci.yml).

Detailed Task Breakdown

Task 1: Manifest Schema, Suite Path Mappings & Selector Engine (hack/ci/)

Files to Modify / Create

  • hack/ci/manifest.go
  • hack/ci/suites.yaml
  • hack/ci/select.go (NEW)
  • hack/ci/select_test.go (NEW)
  • hack/ci/manifest_test.go
  • hack/ci/coverage_test.go

Implementation Instructions

  1. Schema Additions in hack/ci/manifest.go:
    • Add Selection SelectionConfig (yaml:"selection") to Manifest:
      type SelectionConfig struct {
          FullRunPaths []string `yaml:"full_run_paths"`
          CorePaths    []string `yaml:"core_paths"`
      }
    • Add PRSmoke bool (yaml:"pr_smoke") and Paths []string (yaml:"paths") to Suite (hack/ci/manifest.go:116-142).
    • In Manifest.validate() (hack/ci/manifest.go:314-332) and Suite.validate() (hack/ci/manifest.go:208-227):
      • Validate that any glob pattern present in Selection.FullRunPaths, Selection.CorePaths, and Suite.Paths is non-empty and syntactically valid.
      • Enforce in coverage_test.go (TestEverySuiteHasSelectionPaths) that the embedded suites.yaml has non-empty Selection.FullRunPaths, non-empty Selection.CorePaths, and len(s.Paths) > 0 on every suite in manifest.E2E.Suites (keeping minimal synthetic YAML fixtures in manifest_test.go working without boilerplate).
  2. Selector Engine in hack/ci/select.go:
    • Implement matchGlob(pattern, filePath string) bool supporting * (single path segment wildcard) and ** (zero or more path segments) on slash-normalized repo-relative paths.
    • Implement selectSuites(m *Manifest, group string, secretAvailable bool, mode string, changedFiles []string) []Suite:
      • Filter m.SuitesIn(group) by secretAvailable (if s.RequiresSecret && !secretAvailable { continue }).
      • If mode == "" || mode == "full": return all secret-eligible suites in group.
      • If mode == "pr":
        • If len(changedFiles) == 0: return []Suite{}.
        • If matchesAny(changedFiles, m.Selection.FullRunPaths): return all secret-eligible suites in group.
        • Compute coreChanged := matchesAny(changedFiles, m.Selection.CorePaths).
        • Keep suite s if (coreChanged && s.PRSmoke) || matchesAny(changedFiles, s.Paths).
  3. Populate hack/ci/suites.yaml:
    • Add selection.full_run_paths:
      • .github/workflows/pr-ci.yml, hack/ci/**, go.mod, go.sum, .goreleaser.yml, e2e/framework/**, e2e/*.go
    • Add selection.core_paths covering shared CLI/workspace/agent/devcontainer/driver-factory packages:
      • cmd/devsy/**, cmd/root.go, cmd/up.go, cmd/flags/**, cmd/workspace/up/**, cmd/internal/agent*.go, cmd/internal/agentworkspace/**, cmd/internal/agentcontainer/**, cmd/internal/runusercommands.go
      • pkg/agent/**, pkg/client/**, pkg/command/**, pkg/config/**, pkg/copy/**, pkg/devcontainer/*.go, pkg/devcontainer/build/**, pkg/devcontainer/config/**, pkg/devcontainer/dockerfile/**, pkg/devcontainer/lifecycle/**, pkg/devcontainer/metadata/**, pkg/dockerfile/**, pkg/driver/*.go, pkg/driver/drivercreate/create.go, pkg/encoding/**, pkg/envfile/**, pkg/exitcode/**, pkg/extract/**, pkg/flags/**, pkg/git/**, pkg/image/**, pkg/inject/**, pkg/language/**, pkg/log/**, pkg/options/**, pkg/output/**, pkg/provider/**, pkg/secrets/**, pkg/setup/**, pkg/shell/**, pkg/ssh/**, pkg/status/**, pkg/telemetry/**, pkg/types/**, pkg/util/**, pkg/workspace/**, providers/providers.go
    • Mark the 13 Core Smoke Tier suites with pr_smoke: true:
      • unprivileged: context, extends
      • integration (Linux Docker/Kind): up-workspaces, up-provider-docker, up-docker-compose, up-features, ssh, build, up-provider-kubernetes
      • integration (Linux Podman): up-provider-podman-rootless-basic, up-provider-podman-rootful-basic
      • integration (Windows): up-docker-wsl
      • microsandbox: up-provider-microsandbox
    • Populate paths on all 72 suites in suites.yaml:
      • context: ["e2e/tests/context/**", "cmd/context/**", "pkg/config/**", "pkg/options/**"]
      • extends: ["e2e/tests/extends/**", "cmd/workspace/describe.go", "pkg/devcontainer/config/**", "pkg/compose/**"]
      • outdated: ["e2e/tests/outdated/**", "cmd/outdated.go", "pkg/devcontainer/feature/**", "pkg/image/**"]
      • machine: ["e2e/tests/machine/**", "cmd/machine/**", "pkg/driver/custom/**", "providers/ssh/**"]
      • self-update: ["e2e/tests/self/**", "cmd/version.go", "cmd/upgrade.go", "pkg/version/**"]
      • upgrade: ["e2e/tests/upgrade/**", "cmd/upgrade.go", "pkg/version/**"]
      • docker-context: ["e2e/tests/dockercontext/**", "pkg/docker/**", "pkg/driver/docker/**", "providers/docker/**"]
      • ide: ["e2e/tests/ide/**", "cmd/ide/**", "cmd/workspace/set_ide.go", "cmd/internal/browser_tunnel.go", "pkg/ide/**", "pkg/browser/**"]
      • machine-provider: ["e2e/tests/machineprovider/**", "cmd/machine/**", "cmd/provider/**", "pkg/driver/custom/**", "providers/ssh/**"]
      • integration: ["e2e/tests/integration/**", "cmd/workspace/**", "pkg/workspace/**"]
      • tunnel: ["e2e/tests/tunnel/**", "cmd/workspace/**", "pkg/tunnel/**", "pkg/port/**", "pkg/credentials/**"]
      • mcp: ["e2e/tests/mcp/**", "cmd/mcp.go", "pkg/mcp/**"]
      • provider: ["e2e/tests/provider/**", "cmd/provider/**", "pkg/provider/**", "pkg/options/**", "providers/**"]
      • ssh: ["e2e/tests/ssh/**", "cmd/ssh.go", "cmd/workspace/ssh.go", "cmd/workspace/port_forward.go", "cmd/workspace/gpg_tunnel.go", "cmd/internal/ssh_*.go", "cmd/internal/container_tunnel.go", "cmd/internal/gh.go", "cmd/internal/install_gh_wrapper.go", "cmd/internal/git_*.go", "pkg/ssh/**", "pkg/gpg/**", "pkg/credentials/**", "pkg/ghcredentials/**", "pkg/gitcredentials/**", "pkg/gitsshsigning/**", "pkg/port/**"]
      • ssh-agent-forward: ["e2e/tests/ssh/**", "cmd/internal/ssh_*.go", "pkg/ssh/**"]
      • ssh-ports-attributes: ["e2e/tests/ssh/**", "cmd/internal/container_tunnel.go", "pkg/port/**", "pkg/netstat/**", "pkg/devcontainer/config/**"]
      • ssh-tunnel-mode: ["e2e/tests/ssh/**", "cmd/workspace/up/**", "cmd/workspace/task*.go", "pkg/ssh/**", "pkg/task/**", "pkg/command/**"]
      • task-cancel (Linux & Windows): ["e2e/tests/ssh/**", "cmd/workspace/task*.go", "cmd/workspace/stop.go", "pkg/task/**", "pkg/command/**"]
      • ssh-credentials-server-race: ["e2e/tests/ssh/**", "cmd/workspace/ssh.go", "cmd/internal/container_tunnel.go", "pkg/credentials/**"]
      • build: ["e2e/tests/build/**", "cmd/build.go", "pkg/devcontainer/build/**", "pkg/devcontainer/feature/**", "pkg/devcontainer/dockerfile/**", "pkg/devcontainer/metadata/**", "pkg/buildkit/**", "pkg/driver/docker/**"]
      • docker-install: ["e2e/tests/dockerinstall/**", "pkg/docker/**", "pkg/driver/docker/**", "providers/docker/**"]
      • exec: ["e2e/tests/exec/**", "cmd/workspace/exec.go", "pkg/driver/docker/**", "pkg/ssh/**"]
      • ci: ["e2e/tests/ci/**", "cmd/up.go", "cmd/build.go", "cmd/workspace/**", "pkg/git/**", "pkg/workspace/**"]
      • feature: ["e2e/tests/feature/**", "cmd/features/**", "pkg/devcontainer/feature/**", "pkg/image/**"]
      • extends-up: ["e2e/tests/extends-up/**", "pkg/devcontainer/config/**"]
      • down: ["e2e/tests/down/**", "cmd/down.go", "cmd/workspace/stop.go", "cmd/workspace/delete.go", "pkg/workspace/**"]
      • rename: ["e2e/tests/rename/**", "cmd/workspace/rename.go", "pkg/workspace/**", "pkg/ssh/**", "pkg/ide/vscode/**"]
      • delivery: ["e2e/tests/delivery/**", "cmd/up.go", "cmd/workspace/up/**", "pkg/devcontainer/**", "pkg/driver/docker/**"]
      • logs: ["e2e/tests/logs/**", "cmd/workspace/logs.go", "cmd/workspace/setup_logs.go", "cmd/internal/logs_daemon.go", "cmd/internal/runusercommands.go", "pkg/setup/**"]
      • templates: ["e2e/tests/template/**", "cmd/templates/**", "pkg/devcontainer/template/**"]
      • config-apply: ["e2e/tests/configapply/**", "pkg/devcontainer/config/**", "pkg/driver/docker/**"]
      • run-user-commands: ["e2e/tests/runusercommands/**", "cmd/internal/runusercommands.go", "pkg/devcontainer/lifecycle/**", "pkg/setup/**"]
      • config-read: ["e2e/tests/configread/**", "cmd/workspace/describe.go", "pkg/devcontainer/config/**", "pkg/compose/**"]
      • up-workspaces: ["e2e/tests/up/**", "cmd/up.go", "cmd/workspace/events.go", "cmd/container/**", "pkg/git/**", "pkg/language/**", "pkg/envfile/**"]
      • up-behaviors: ["e2e/tests/up/**", "pkg/dotfiles/**", "pkg/ide/openvscode/**", "pkg/devcontainer/**"]
      • up-handle-errors: ["e2e/tests/up/**", "cmd/workspace/up/**", "pkg/exitcode/**", "pkg/devcontainer/**"]
      • up-private-token: ["e2e/tests/up/up_private_token.go", "cmd/internal/git_credentials.go", "pkg/git/**", "pkg/gitcredentials/**"]
      • up-provider-kubernetes & up-provider-kubernetes-restricted-scc: ["e2e/tests/up/provider_kubernetes*.go", "pkg/driver/kubernetes/**", "providers/kubernetes/**", "cmd/internal/ssh_git_clone.go"]
      • up-provider-kubernetes-inactivity: ["e2e/tests/up/provider_kubernetes_inactivity.go", "pkg/driver/kubernetes/**", "providers/kubernetes/**", "cmd/internal/agent_daemon.go", "cmd/internal/ssh_server.go"]
      • up-provider-docker: ["e2e/tests/up/provider_docker.go", "e2e/tests/up/dockerfile_build.go", "e2e/tests/up/interactive_post_start.go", "e2e/tests/up/sops.go", "cmd/secret/**", "pkg/driver/docker/**", "pkg/docker/**", "pkg/secrets/**", "providers/docker/**"]
      • snapshot: ["e2e/tests/snapshot/**", "cmd/workspace/snapshot*.go", "pkg/snapshot/**", "pkg/driver/docker/**"]
      • up-docker-compose: ["e2e/tests/up-docker-compose/**", "pkg/compose/**", "pkg/driver/docker/**", "pkg/secrets/**"]
      • up-features (Linux & Windows): ["e2e/tests/up-features/**", "pkg/devcontainer/feature/**", "pkg/devcontainer/build/**"]
      • All 10 up-provider-podman-* suites: ["e2e/tests/up/*podman*", "pkg/driver/docker/**", "pkg/docker/**", "providers/docker/**", "hack/podman_ci/**"]
      • Windows provider, up-docker-wsl, ssh-proxy-command, ssh-tunnel-mode: include their respective feature paths plus ["e2e/framework/dockerwsl/**", "hack/podman_ci/**"]
      • Windows docker-internal-buildkit: ["e2e/tests/dockerbuildkit/**", "e2e/framework/dockerwsl/**", "pkg/buildkit/**", "pkg/driver/docker/**", "pkg/docker/**"]
      • Windows wslg: ["e2e/tests/dockerbuildkit/wslg.go", "e2e/framework/dockerwsl/**", "pkg/wslg/**", "pkg/compose/**"]
      • All 16 up-provider-microsandbox* suites: ["e2e/tests/up/*microsandbox*", "e2e/tests/up/managed_policy_fixture_linux.go", "pkg/driver/microsandbox/**", "pkg/driver/external/**", "pkg/driver/drivercreate/microsandbox_images.go", "providers/microsandbox/**", "hack/microsandbox/**", "cmd/internal/runtime_supervisor.go", "cmd/internal/managedvalue/**"]
  4. Coverage Invariants in hack/ci/coverage_test.go:
    • Add TestEverySuiteHasSelectionPaths: asserts every suite in manifest.E2E.Suites has len(s.Paths) > 0, and that static directory/file prefixes of every pattern in Selection.FullRunPaths, Selection.CorePaths, and Suite.Paths exist on disk under s.root.

Task 2: Dynamic Matrix & Build Planner (hack/ci/matrix.go)

Files to Modify

  • hack/ci/matrix.go
  • hack/ci/matrix_test.go

Implementation Instructions

  1. CLI Flags in matrixCLI (hack/ci/matrix.go:65-103):
    • Add --mode flag ("full" or "pr", default "full"). Reject unknown modes with exit code 2 (errUnknownMatrixMode).
    • Add --changed-files flag (path to file containing newline-separated changed file paths; empty means no file).
  2. Add "build" Matrix Group (hack/ci/matrix.go:14-63):
    • Define matrixBuildGroup = "build" and add "build" to matrixGroupNames.
    • Define matrixBuildEntry:
      type matrixBuildEntry struct {
          Runner             string `json:"runner"`
          JobTimeoutMinutes  int    `json:"job_timeout_minutes,omitempty"`
          BuildMicrosandbox  bool   `json:"build_microsandbox"`
      }
    • In buildMatrixEntries(m *Manifest, secretAvailable bool, mode string, changedFiles []string) []matrixBuildEntry:
      • In mode == "full": return the 3 entries matching current pr-ci.yml:
        1. {Runner: "ubuntu-latest", JobTimeoutMinutes: 45, BuildMicrosandbox: true}
        2. {Runner: "macos-latest", JobTimeoutMinutes: 20, BuildMicrosandbox: false}
        3. {Runner: "windows-latest", JobTimeoutMinutes: 45, BuildMicrosandbox: false}
      • In mode == "pr":
        • Compute selected suites across unprivileged, integration, and microsandbox.
        • If .goreleaser.yml or FullRunPaths matched, return all 3 runners (ubuntu-latest, macos-latest, windows-latest) with BuildMicrosandbox: true.
        • Otherwise:
          • needLinux := len(unprivileged) > 0 || len(integration) > 0 || len(microsandbox) > 0 (Note: Windows E2E suites also download devsy-linux for container injection, so needLinux is true whenever any E2E suite runs).
          • needWindows := false (true if any selected suite in integration has s.OS() == "windows").
          • needMicrosandbox := len(microsandbox) > 0.
          • Append ubuntu-latest (JobTimeoutMinutes: 45, BuildMicrosandbox: needMicrosandbox) if needLinux.
          • Append windows-latest (JobTimeoutMinutes: 45) if needWindows.
  3. Wire e2eMatrixEntries to selectSuites:
    • Pass mode and changedFiles to selectSuites(m, group, secretAvailable, mode, changedFiles).
  4. Unit Tests in hack/ci/matrix_test.go:
    • Test --mode=full emits all 5 groups (unit, build, unprivileged, integration, microsandbox) with all suites.
    • Test --mode=pr with a leaf change (e.g. pkg/mcp/server.go) selects only mcp in integration, empty unprivileged and microsandbox, and only ubuntu-latest (with build_microsandbox: false) in build.
    • Test --mode=pr with a core change (e.g. cmd/up.go) selects the 13 pr_smoke suites and includes both ubuntu-latest (build_microsandbox: true) and windows-latest in build.
    • Test --mode=pr with no Go/E2E changes (e.g. sites/docs-devsy-sh/README.md) emits {"include":[]} for build, unprivileged, integration, and microsandbox.

Task 3: Update ci-success Gate Evaluator (hack/ci/success.go)

Files to Modify

  • hack/ci/success.go
  • hack/ci/success_test.go

Implementation Instructions

  1. Per-Job Legitimate Skip Rules in hack/ci/success.go:
    • Keep --always-run flag support (in pr-ci.yml, --always-run changes,plan,precommit).
    • Validate changes job in needs: must exist and succeed when checking skips.
    • Define the expected skip condition for each conditional job in needs:
      • agent-skill: may be "skipped" only if changes.outputs["agent_skill"] == "false".
      • licenses: may be "skipped" only if changes.outputs["licenses"] == "false".
      • github-credential-host-tests: may be "skipped" only if changes.outputs["gh_credentials"] == "false".
      • provider-update-windows: may be "skipped" only if changes.outputs["provider_update"] == "false".
      • lint, unit-tests: may be "skipped" only if changes.outputs["go"] == "false".
      • build-cli: may be "skipped" only if changes.outputs["go"] == "false" OR plan.outputs["build"] == "{\"include\":[]}".
      • integration-tests-unprivileged: may be "skipped" only if changes.outputs["go"] == "false" OR plan.outputs["unprivileged"] == "{\"include\":[]}".
      • integration-tests: may be "skipped" only if changes.outputs["go"] == "false" OR plan.outputs["integration"] == "{\"include\":[]}".
      • prepare-microsandbox-provider, integration-tests-microsandbox: may be "skipped" only if changes.outputs["go"] == "false" OR plan.outputs["microsandbox"] == "{\"include\":[]}".
      • Any other job not in alwaysRun: falls back to changes.outputs["go"] == "false" (preserving backward compatibility with existing synthetic job names in unit tests).
    • Preserve fail-closed behavior: if changes (or plan for matrix-gated jobs) is missing, failed, or has an invalid output value, any "skipped" result on that downstream job is rejected with ::error::<job> ended with result=skipped unexpectedly.
  2. Unit Tests in hack/ci/success_test.go:
    • Verify all existing tests pass and add tests covering:
      • Skipping integration-tests-microsandbox and prepare-microsandbox-provider when changes.outputs.go == "true" and plan.outputs.microsandbox == '{"include":[]}' succeeds.
      • Skipping integration-tests-microsandbox when plan.outputs.microsandbox is non-empty fails with ::error::integration-tests-microsandbox ended with result=skipped unexpectedly.
      • Skipping agent-skill, licenses, github-credential-host-tests, provider-update-windows when their respective changes output is "false" succeeds, and fails when "true".

Task 4: Wire .github/workflows/pr-ci.yml

Files to Modify

  • .github/workflows/pr-ci.yml

Implementation Instructions

  1. Update changes Job (.github/workflows/pr-ci.yml:23-50):
    • Add outputs: go, agent_skill, licenses, gh_credentials, provider_update, mode, changed_files.
    • On github.event_name != 'pull_request' (push to main or workflow_dispatch), set go=true, agent_skill=true, licenses=true, gh_credentials=true, provider_update=true, mode=full.
    • On pull_request, run dorny/paths-filter@v4 with list-files: json and filters for go, agent_skill, licenses, gh_credentials, provider_update, and all: ['**'], and set mode=pr.
  2. Update plan Job (.github/workflows/pr-ci.yml:51-75):
    • Add needs: [changes].
    • Add build: ${{ steps.matrix.outputs.build }} to outputs.
    • Write needs.changes.outputs.changed_files (JSON array from dorny/paths-filter) to a newline-delimited temp file and pass --mode="${{ needs.changes.outputs.mode }}" --changed-files="$RUNNER_TEMP/changed-files.txt" to go run ./hack/ci matrix.
  3. Update precommit, agent-skill, lint, licenses, github-credential-host-tests, provider-update-windows:
    • precommit: run SKIP=golangci-lint prek run --all-files.
    • agent-skill: add needs: [changes] and if: needs.changes.outputs.agent_skill == 'true'.
    • lint: add needs: [changes] and if: needs.changes.outputs.go == 'true'.
    • licenses: change needs: [changes, precommit, lint] to needs: [changes] and if: needs.changes.outputs.licenses == 'true'.
    • github-credential-host-tests: change if: to needs.changes.outputs.gh_credentials == 'true'.
    • provider-update-windows: change if: to needs.changes.outputs.provider_update == 'true'.
  4. Update build-cli (.github/workflows/pr-ci.yml:222-380):
    • Change needs: [changes, precommit, lint] to needs: [changes, plan].
    • Change if: to needs.changes.outputs.go == 'true' && needs.plan.outputs.build != '{"include":[]}'.
    • Set strategy: to fail-fast: false and matrix: ${{ fromJSON(needs.plan.outputs.build) }}.
    • Gate the MicroSandbox build/upload steps (L320–L343) with if: runner.os == 'Linux' && matrix.build_microsandbox.
  5. Update E2E Jobs & prepare-microsandbox-provider (.github/workflows/pr-ci.yml:396-907):
    • integration-tests-unprivileged: if: needs.changes.outputs.go == 'true' && needs.plan.outputs.unprivileged != '{"include":[]}'
    • prepare-microsandbox-provider: needs: [changes, plan, build-cli], if: needs.changes.outputs.go == 'true' && needs.plan.outputs.microsandbox != '{"include":[]}'
    • integration-tests-microsandbox: if: needs.changes.outputs.go == 'true' && needs.plan.outputs.microsandbox != '{"include":[]}'
    • integration-tests: if: needs.changes.outputs.go == 'true' && needs.plan.outputs.integration != '{"include":[]}'
  6. Update ci-success (.github/workflows/pr-ci.yml:908-941):
    • Update command to: go run ./hack/ci success --always-run changes,plan,precommit

Verification & Acceptance Criteria

  1. Unit & Coverage Tests (hack/ci & hack/microsandbox):
    go test -race -v ./hack/ci/...
    go test -race -v ./hack/microsandbox -run TestMicrosandboxPins
  2. CLI Planner Smoke Verification:
    # 1. Full mode emits all 72 suites and 3 build runners
    go run ./hack/ci matrix --mode=full --secret-available=true
    
    # 2. PR mode with isolated package change (e.g., pkg/mcp/mcp.go) emits only `mcp` E2E suite and Linux build
    printf "pkg/mcp/mcp.go\n" > /tmp/changed.txt
    go run ./hack/ci matrix --mode=pr --changed-files=/tmp/changed.txt --secret-available=true
    
    # 3. PR mode with core package change (e.g., cmd/up.go) emits the 13 pr_smoke suites
    printf "cmd/up.go\n" > /tmp/changed.txt
    go run ./hack/ci matrix --mode=pr --changed-files=/tmp/changed.txt --secret-available=true
    
    # 4. PR mode with docs-only change emits empty E2E and build matrices
    printf "sites/docs-devsy-sh/content/docs/index.mdx\n" > /tmp/changed.txt
    go run ./hack/ci matrix --mode=pr --changed-files=/tmp/changed.txt --secret-available=true
  3. Linter & Pre-commit Validation:
    mise run cli:lint
    prek run --all-files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions