diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 0ea3629..acf8608 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -7,11 +7,21 @@ Describe the concrete trigger and resulting behavior. ## Evidence - Failing test before implementation (or explain a documentation-only change): -- Passing commands/results: +- Focused passing commands/results while iterating: +- Stable candidate hosted CI run/checks (exact head SHA): - Relevant fault/security/resource cases: - Independent review and model: - Tests not run and why: +## Incremental review ledger + +- Candidate source SHA and files/risk surface: +- Findings batched before the candidate push: +- Previously resolved findings carried forward with original URL, source SHA and resolution evidence: +- Final delta sign-off against the exact candidate SHA: +- New full review needed because of changed risk/interface (or explain why not): +- Security second pass (when applicable): + ## Operational effect State migrations, cleanup ownership, rollout/rollback and remaining limitations. Do not include raw credentials or private job logs. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2ed6056..ae8d96c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,8 +2,31 @@ Start with an issue in the implementation Project. Read [the plan](docs/PLAN.md), [agent execution](docs/EXECUTION.md) and [TDD strategy](docs/TEST-STRATEGY.md). -Keep changes scoped to an observable issue goal. Include a reproducing failing test for behavior changes, relevant checks, independent review of security/concurrency changes, and accurate limitations. Documentation-only changes do not require artificial tests. +Keep changes scoped to an observable issue goal. The writer follows red -> minimal +green -> refactor, then runs focused unit/negative checks while iterating; +documentation-only changes record why no artificial test is needed. Use +`make fast` only with explicit `FAST_MODULE`, `FAST_PACKAGE` and `FAST_TEST` +selectors; it fails closed for missing or no-match selectors and is not the full +gate. Batch source, documentation and finding-ledger fixes before pushing one +stable review candidate rather than launching a review for every local commit. -The public baseline is documented in [CI.md](docs/CI.md). Run `make check` before submitting a change; it uses the pinned Go toolchain and reports absent unit or fuzz suites explicitly. +Independent reviewers use the immutable candidate source and exact-source CI +evidence, adding delta/risk probes instead of repeating the complete suite. A +second independent security/recovery pass remains required when the changed +boundary warrants it. The coordinator audits the contract, ledger and evidence; +the coordinator is not a third full-suite tester. Carry resolved findings forward +with their original URL, source SHA and resolution evidence, and sign off the +final delta against the exact candidate SHA. + +The public baseline is documented in [CI.md](docs/CI.md). `make check` remains the +complete public validation suite and hosted PR CI remains the required stable +candidate gate; it is not an automatic per-commit requirement. The final merge +gate still requires exact-head GitHub Codex review, required CI for that same SHA, +stale/outdated finding resolution or rebuttal, and any applicable security second +pass. Main's postmerge integration run does not replace those premerge checks. + +Release, macOS, soak and other trusted/live checks run before the applicable +release or live qualification only, with an immutable reviewed commit and +explicit maintainer authorization. No mandatory security gate is deferred. Do not run unreviewed contributions on privileged self-hosted runners. Do not submit credentials or private workflow logs. Dependencies and assets must have documented licenses compatible with distribution. diff --git a/Makefile b/Makefile index fe8d84e..e634960 100644 --- a/Makefile +++ b/Makefile @@ -6,16 +6,21 @@ GOFLAGS ?= override GOTOOLCHAIN := go1.26.8 export GOTOOLCHAIN export GOFLAGS +FAST_MODULE := $(value FAST_MODULE) +FAST_PACKAGE := $(value FAST_PACKAGE) +FAST_TEST := $(value FAST_TEST) +export FAST_MODULE FAST_PACKAGE FAST_TEST FUZZTIME ?= 1s GOVULNCHECK_VERSION ?= v1.7.0 -.PHONY: help build check toolchain fmt fmt-check vet test test-race fuzz-smoke deps licenses vuln experiments +.PHONY: help build check fast toolchain fmt fmt-check vet test test-race fuzz-smoke deps licenses vuln experiments help: @printf '%s\n' \ 'gh-runnerd development commands:' \ ' make fmt format Go sources in place' \ ' make check run the complete public validation suite' \ + ' make fast run one explicit FAST_MODULE/FAST_PACKAGE/FAST_TEST selector (not the full gate)' \ ' make build compile all packages' \ ' make experiments test reviewed offline gate modules when present' \ ' make vuln run pinned govulncheck (network access may be needed)' @@ -25,6 +30,9 @@ build: check: toolchain fmt-check build vet test test-race fuzz-smoke deps licenses experiments vuln +fast: + GO="$(GO)" bash scripts/fast-check.sh + toolchain: GO="$(GO)" bash scripts/check-toolchain.sh diff --git a/docs/CI.md b/docs/CI.md index 6631d56..0ac65de 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -22,12 +22,52 @@ disabled. The aggregator uses `always()` so a failed, cancelled, or skipped required job is observed, then fails unless all three dependency results are exactly `success`; it does not check out source or run a test itself. +The pull-request workflow is the premerge source gate. The identical workflow on +`main` is postmerge integration evidence; a green postmerge run cannot substitute +for the exact-head premerge gate. During local editing, use focused checks and the +opt-in `make fast` selector below. After source, documentation and finding-ledger +changes are batched into one stable candidate, hosted CI supplies the complete +matrix; reviewers and the coordinator use that exact-source evidence rather than +rerunning the full suite independently. Release, macOS, soak and other trusted +profiles run only before an applicable release/live qualification, with explicit +maintainer authorization, and are not implied by this hosted check. + +### Issue #73 audit evidence + +This audit was captured from `origin/main` at +`cf67d4aeb511116fee0de31a4ac38409f28fa29f` and the live PR #72 record on +2026-09-10. It records measured observations, not a forecast of savings: + +| Surface | Measured observation | Policy consequence | +| --- | --- | --- | +| Main CI | `ci.yml` has three parallel test-bearing jobs (`root`, `offline`, `vuln`) plus the required `Go checks` aggregator; each remains capped at 15 minutes, with pinned actions and `cache: false`. | Preserve the current complete coverage, job/check names, cache policy and timeouts; no path classifier or cache shortcut was added. | +| Makefile | Existing `check` prerequisites remain `toolchain`, formatting, build, vet, unit/race, fuzz, dependency, license, offline-experiment and vulnerability checks. New `fast` is an opt-in target and is not a `check` prerequisite. | Keep `make check` complete and unchanged as the local public gate; focused iteration cannot silently weaken it. | +| [PR #72 review history](https://github.com/1XP-AI/gh-runnerd/pull/72) | Audit snapshot through immutable PR #72 head `116beda04dc2bf69280cdefc4de4ef2fef397ef3`, captured 2026-09-10. Review records and public checkpoints do not measure actor-side full-suite run counts; the checkpoint comments identify focused/offline or focused race regressions. | Batch findings, source and docs before one candidate push; reviewers perform delta/risk probes against shared CI evidence, and the coordinator audits rather than acting as a third tester. | +| [PR #72 hosted critical path](https://github.com/1XP-AI/gh-runnerd/actions/runs/34419651240) | `gh run view 34419651240 --json jobs` measured workflow start `00:04:06Z`, required jobs finishing by `00:15:14Z`, and aggregator completion at `00:15:18Z`; `00:15:19Z` is a workflow metadata update, not completion. Start-to-aggregator completion was 11m12s. Root ran 11m06s, offline 8m49s, vulnerability 33s, aggregator 2s. | No workflow critical-path speedup is claimed or changed; full CI remains the stable candidate gate. | +| Focused local command | Warmed direct baseline: `env GOTOOLCHAIN=go1.26.8 GOWORK=off go test -count=1 -run '^TestFixedTarget$' ./internal/scheduler/capacity` → `real 0.32s`. New entry point: `env FAST_MODULE=. FAST_PACKAGE=./internal/scheduler/capacity FAST_TEST='^TestFixedTarget$' make fast` → `real 0.34s`; both passed. | This one local pair demonstrates bounded behavior only; it does not claim a speedup or predict CI duration. | + +```console +$ gh run view 34419651240 --repo 1XP-AI/gh-runnerd --json headSha,startedAt,updatedAt,jobs +# head 116beda04dc2bf69280cdefc4de4ef2fef397ef3; start 00:04:06Z; aggregator 00:15:18Z; metadata update 00:15:19Z; +# jobs: root 00:04:08Z-00:15:14Z, offline 00:04:09Z-00:12:58Z, +# vuln 00:04:10Z-00:04:43Z, checks 00:15:16Z-00:15:18Z +``` + +The audit commands and exact observed values are kept in this single summary +instead of accumulating per-step logs. Finding ledgers must retain original URLs, +source SHAs and resolution evidence without copying credentials or private job +logs. + Run the same checks locally with: ```console make check ``` +`make check` remains the complete local public gate, but it is not a per-commit +requirement; use it on demand when the environment supports it and rely on hosted +CI for the stable candidate gate. + Individual commands are available when iterating: | Command | Check | @@ -43,6 +83,45 @@ Individual commands are available when iterating: | `make licenses` | Compare the exact runtime module/version/replacement graph with its inventory and require a top-level license file. | | `make experiments` | Require both established G01/G02 modules, run their static-partitioned default race/vet suites, then exercise the two explicitly reviewed G01 CLI packages with `g01_live,g01_worker` tags and the reviewed `g01_pair_fixture` livecanary collection/listener and terminal partitions with tagged vet. | | `make vuln` | Run the exact `golang.org/x/vuln/cmd/govulncheck@v1.7.0` tool. | +| `make fast FAST_MODULE=... FAST_PACKAGE=... FAST_TEST=...` | Run one explicit test selector in one selected module/package. All three selectors are required and invalid/no-match selectors fail closed; this is focused evidence only, never the complete gate. | + +For example: + +```console +FAST_MODULE=. FAST_PACKAGE=./internal/scheduler/capacity FAST_TEST='^TestFixedTarget$' make fast +``` + +The script resolves the selected module and every package matched by the Go +pattern, preserving wildcard forms such as `./.../capacity`, and rejects +selectors whose physical paths leave the current repository or selected module, +as well as absolute or lexical `..` paths, missing `go.mod`, missing selectors, +package patterns that match no package, and selectors that match no compiled +test. The focused invocation explicitly bounds Go's test execution flags: +inherited `-bench` and `-fuzz` selectors are cleared so they cannot expand work +beyond `FAST_TEST`; `-list`, `-skip`, and build-only `-c` are cleared so they +cannot suppress it; explicit `-run`/`-count` values retain the requested +selector and `-cpu=1` bounds each selected test to one execution. Inherited +`-exec` is cleared so a wrapper cannot bypass the test binary. The command forces +Go's `-test.v=test2json` framed output while disabling the outer `-json` mode, and +requires paired framed per-test `=== RUN` and terminal `--- PASS` lines whose test +name is a complete match under Go's slash-separated component and top-level +alternation semantics; possible-parent events do not count. Framed `skip` or +`fail` outcomes, including a later contradictory terminal after the first +terminal in one `RUN` lifecycle, package summaries, and ordinary unframed +`TestMain` output do not count. Each new framed `RUN` starts a new lifecycle for +that name, so a same-named test in a later package may qualify after an earlier +package skipped it. Dry-run or unsupported build modes fail closed when no +passing framed test result is observed. The frame is a Go test protocol boundary +against ordinary output, not authentication against a deliberately spoofing test +process; this remains a trusted local helper, not a hostile-code sandbox. Useful build +flags such as `-race` and `-mod=readonly` remain inherited; this is not a blanket +`GOFLAGS` removal. If selectors are passed as Make command-line variables instead +of environment assignments, escape literal `$` as `$$` so Make preserves the +regexp anchor. + +`FAST_TEST` is passed as one argument and may contain whitespace in a Go +subtest name; `FAST_MODULE` and `FAST_PACKAGE` remain whitespace-free path +selectors. No hardware, live GitHub, Docker or daemon suite is part of this public check. Those profiles remain explicit future or maintainer-controlled runs; they are not silently converted into passing tests here. G04 introduces the first application behavior contracts and should add meaningful unit and fuzz targets before claiming those forms of coverage. diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index cc4b4a4..45dd32c 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -24,6 +24,9 @@ runtime graph contains only the local module; nested experiments remain separate Go standard library: `go1.26.8`, BSD-3-Clause, the language/runtime baseline; [official release history](https://go.dev/doc/devel/release). +The selector matcher helper adapted from the Go standard library is documented +in the [Go testing matcher adaptation notice](third-party/go-testing-matcher.md). + ## Public CI actions Action refs are immutable commit pins. The version labels are recorded for human review; the workflow uses the full SHA. diff --git a/docs/EXECUTION.md b/docs/EXECUTION.md index ecbfbd2..8a341bc 100644 --- a/docs/EXECUTION.md +++ b/docs/EXECUTION.md @@ -13,6 +13,37 @@ The repository default is Luna max, including contract-setting evidence gates su Verify each issue's contract and dependencies before dependent implementation begins. Escalate discovered architecture, secret handling, process isolation or concurrency changes to an additional independent Luna max review; do not stretch a small issue into an unreviewed redesign. Parallelize only independent issues with non-overlapping file ownership; no simultaneous edits to shared protocol/state definitions. +## Incremental validation and review + +Validation is incremental during editing and complete at the stable merge +candidate. Do not require or automatically run the whole suite for every local +commit. The writer records a meaningful red case for behavior changes, applies a +minimal green fix, and runs the explicit focused unit/negative checks relevant to +the changed surface; documentation-only changes record why no artificial test is +needed. The opt-in `make fast` entry point requires an explicit module, package, +and test selector, fails closed for missing/invalid selectors, and never stands in +for the complete gate. + +| Role/tier | Required work and evidence | +|---|---| +| Writer / edit | Red -> minimal green -> refactor; focused checks and `git diff --check` while iterating. Batch source, docs and finding-ledger changes before one review candidate push. | +| Independent reviewer(s) | Review the immutable candidate source and shared exact-source CI evidence; run only delta/risk probes. Add the independent Luna max security/recovery pass when the changed boundary warrants it. Do not duplicate the full suite by default. | +| Coordinator | Audit the contract, changed surface, finding ledger and exact-source CI/review records; coordinate resolution. The coordinator is not a third full-suite tester. | +| Candidate / premerge | The pushed PR head receives the complete hosted CI matrix once stable. Preserve required job/check names, complete coverage, exact pull-request-head checkout, cache policy and timeouts. | +| Main / postmerge | The same workflow on `main` is integration evidence after merge; it does not replace the premerge candidate gate or authorize a merge. | +| Release/live qualification | Run release, macOS, soak and other trusted/live profiles only before the applicable release or live qualification, on a reviewed immutable commit with explicit maintainer authorization. Do not defer a mandatory security gate. | + +Use one finding ledger per candidate: carry each previously resolved finding with +its immutable source SHA, original finding URL and resolution evidence, then record +the final delta sign-off against the exact candidate SHA. A new internal full review +is needed only when the changed diff crosses a new risk or interface boundary; +focused delta review remains required for ordinary fixes. GitHub Codex review and +the required CI gate are separate and stricter: immediately before merge, Codex +must have reviewed the exact final HEAD, stale/outdated findings must be read and +resolved or rebutted, required CI must pass for that same SHA, and a security +second pass remains required where applicable. Do not request repetitive Codex +reviews mid-edit; after a fix push, request and await the fresh exact-head review. + ## Per-issue goal workflow 1. Select a Ready issue from the live GitHub Project whose dependencies are Done; read the plan, relevant ADR and current repository instructions. Do not treat `backlog.json` `status` as live Ready. diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 961e1b3..4028172 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -223,6 +223,31 @@ For every new issue, after checking for an explicit current user override: one integrator owns shared protocol/state definitions; - a Project field never launches work by itself. +## Incremental validation and review handoff + +Use focused validation while the candidate is changing and the complete gate once +the candidate is stable. The writer owns meaningful red -> minimal green -> +refactor evidence and focused unit/negative checks; an explicit `make fast` +module/package/test selector may help with local iteration, but it is fail-closed +and never represents `make check` or CI success. Do not make a whole-suite run an +automatic requirement for every local commit. + +| Role | Handoff contract | +|---|---| +| Writer | Batch all source, documentation and finding-ledger changes before one candidate push; include exact commands/results and `git diff --check`. | +| Independent reviewers | Inspect the immutable candidate and shared exact-source hosted CI evidence; run delta/risk probes only. Add the independent Luna max security/recovery pass for applicable boundaries. | +| Coordinator | Audit the issue contract, changed surface, ledger and exact-head evidence; coordinate fixes. Do not act as a third full-suite tester. | +| CI / merge | Hosted PR CI is the complete premerge source gate. GitHub Codex must review the exact final HEAD, including stale/outdated findings, and required CI must pass for that SHA. | +| Main / release | `main` CI is postmerge integration evidence. Release, macOS, soak and trusted/live checks are required only for their applicable qualification, on a reviewed immutable commit with explicit maintainer authorization; no mandatory security gate is deferred. | + +Carry resolved findings into each candidate ledger with the original finding URL, +immutable source SHA and resolution evidence, then record final delta sign-off for +the exact candidate SHA. New internal full review is limited to a changed risk or +interface boundary; ordinary fixes receive delta review. After a fix push, obtain +fresh exact-head Codex review and CI; do not request repetitive Codex reviews +mid-edit. The full candidate gate remains separate from internal review, and the +postmerge `main` run never substitutes for premerge evidence. + Suggested handoff prompt to give the next agent: > Work on ISSUE_URL in `1XP-AI/gh-runnerd`. Use the issue's implementer (default `gpt-5.6-luna` with `max` unless an explicit current user override is recorded) and one active goal exactly equal to the issue's Goal statement; do not invent a token budget. Independent review is Luna max even when the implementer is overridden. Read `AGENTS.md`, `docs/EXECUTION.md`, the plan, the linked ADRs and the current Project item. Verify dependencies first. Create the goal and isolated branch/worktree, then set the item to In progress. Follow meaningful red test -> minimal green implementation -> refactor -> boundary/failure tests. Preserve no-secrets, no-busy-kill, owned-cleanup, stable-idempotency and trusted-native invariants. Do not change live runners, Docker context, App/Keychain/launchd state or GitHub credentials without explicit maintainer authorization. Open one focused PR with exact commands/results, red evidence, gaps and rollback notes. Obtain independent Luna max review, then the exact-head GitHub Codex review before merge. Update the Project, issue and goal only when their actual state changes. Do not write Luna into the Project Agent field over an inspected override that has no Agent option; preserve the existing Agent value. @@ -417,14 +442,17 @@ Preserve these invariants: ### 5. Review, PR and exact-head gate -Run the repository checks appropriate to the issue and create one focused PR. Public +Run focused checks appropriate to the changed surface while editing, then batch +source, docs and finding-ledger changes into one stable review candidate. Public CI must not require live App credentials, local runner access or the unreleased -manager. Trusted hardware/live tests require a reviewed immutable commit and +manager. The hosted PR workflow supplies the complete candidate gate; local +`make check` is available on demand but is not an automatic per-commit +requirement. Trusted hardware/live tests require a reviewed immutable commit and explicit maintainer-triggered execution on the scoped runner group. ```sh git diff --check -make check # when the issue's files and environment support it +make check # optional local confidence; hosted PR CI is the full candidate gate git add path/to/changed/files git commit -m "..." git push -u origin "$BRANCH" @@ -597,6 +625,10 @@ Before handing work onward, confirm: values were not rewritten. - [ ] A meaningful red case, minimal green fix and relevant boundary tests are recorded, with actual commands/results and remaining gaps. +- [ ] Source, documentation and finding-ledger changes were batched before the + candidate push; focused checks are not represented as full-gate evidence. +- [ ] Previously resolved findings retain original URLs, immutable source SHAs and + resolution evidence, with final delta sign-off on the exact candidate SHA. - [ ] No secrets, personal paths, raw SDK errors, live tokens or unreviewed runner operations entered files, issues, logs or artifacts. - [ ] Busy work was never killed and cleanup is ownership-bound. diff --git a/docs/third-party/go-testing-matcher.md b/docs/third-party/go-testing-matcher.md new file mode 100644 index 0000000..3ab4594 --- /dev/null +++ b/docs/third-party/go-testing-matcher.md @@ -0,0 +1,54 @@ +# Go testing matcher adaptation notice + +The following functions in `scripts/fast-check-events/main.go` are adapted from +the Go 1.26.8 standard-library source file `src/testing/match.go`: + +- `splitSelector` adapts the upstream `splitRegexp` component-boundary state + machine. +- `rewrite` and `isSpace` adapt the corresponding upstream functions. + +The adaptation uses the project's selector and framed-output parser interfaces +and keeps the project-specific complete-match behavior. This is source +provenance documentation; the functions are not represented as independent +authorship. + +The upstream source identifies its notice as follows: + +```text +// Copyright 2015 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. +``` + +The applicable Go license text from the pinned Go 1.26.8 toolchain is reproduced +verbatim below. + +```text +Copyright 2009 The Go Authors. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google LLC nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` diff --git a/scripts/fast-check-events/main.go b/scripts/fast-check-events/main.go new file mode 100644 index 0000000..1e17d28 --- /dev/null +++ b/scripts/fast-check-events/main.go @@ -0,0 +1,246 @@ +package main + +import ( + "bufio" + "errors" + "flag" + "fmt" + "io" + "os" + "regexp" + "strconv" + "strings" +) + +type compiledSelector [][]*regexp.Regexp + +const test2JSONMarker = byte(0x16) + +func main() { + var pattern string + flag.StringVar(&pattern, "selector", "", "Go test selector") + flag.Parse() + if pattern == "" || flag.NArg() != 0 { + fmt.Fprintln(os.Stderr, "fast-check-events requires one non-empty --selector") + os.Exit(2) + } + + selector, err := compileSelector(pattern) + if err != nil { + fmt.Fprintf(os.Stderr, "invalid Go test selector: %v\n", err) + os.Exit(2) + } + found, err := hasCompleteMatch(os.Stdin, selector) + if err != nil { + fmt.Fprintf(os.Stderr, "could not read Go test events: %v\n", err) + os.Exit(2) + } + if !found { + os.Exit(1) + } +} + +func compileSelector(pattern string) (compiledSelector, error) { + parts := splitSelector(pattern) + compiled := make(compiledSelector, len(parts)) + for alternative, sequence := range parts { + compiled[alternative] = make([]*regexp.Regexp, len(sequence)) + for element, part := range sequence { + expression, err := regexp.Compile(rewrite(part)) + if err != nil { + return nil, fmt.Errorf("alternative %d element %d: %w", alternative, element, err) + } + compiled[alternative][element] = expression + } + } + return compiled, nil +} + +func splitSelector(pattern string) [][]string { + alternatives := make([][]string, 0, strings.Count(pattern, "|")+1) + sequence := make([]string, 0, strings.Count(pattern, "/")) + start := 0 + characterClassDepth := 0 + parenthesisDepth := 0 + for i := 0; i < len(pattern); i++ { + switch pattern[i] { + case '[': + characterClassDepth++ + case ']': + characterClassDepth-- + if characterClassDepth < 0 { + characterClassDepth = 0 + } + case '(': + if characterClassDepth == 0 { + parenthesisDepth++ + } + case ')': + if characterClassDepth == 0 { + parenthesisDepth-- + } + case '\\': + i++ + case '/': + if characterClassDepth == 0 && parenthesisDepth == 0 { + sequence = append(sequence, pattern[start:i]) + start = i + 1 + } + case '|': + if characterClassDepth == 0 && parenthesisDepth == 0 { + sequence = append(sequence, pattern[start:i]) + alternatives = append(alternatives, sequence) + sequence = make([]string, 0, strings.Count(pattern[i+1:], "/")) + start = i + 1 + } + } + } + sequence = append(sequence, pattern[start:]) + alternatives = append(alternatives, sequence) + return alternatives +} + +func (s compiledSelector) matches(name string) bool { + elements := strings.Split(name, "/") + for _, alternative := range s { + matched := true + for i, expression := range alternative { + if i >= len(elements) { + break + } + if !expression.MatchString(elements[i]) { + matched = false + break + } + } + if matched && len(elements) >= len(alternative) { + return true + } + } + return false +} + +func hasCompleteMatch(input io.Reader, selector compiledSelector) (bool, error) { + reader := bufio.NewReader(input) + results := make(map[string]testResult) + passedAny := false + for { + line, err := reader.ReadString('\n') + parseFramedTestEvents(line, func(action, name string) { + if name == "" || !selector.matches(name) { + return + } + result := results[name] + switch action { + case "run": + result = testResult{ran: true} + case "pass", "skip", "fail": + if result.ran && !result.terminal { + result.terminal = true + result.passed = action == "pass" + passedAny = passedAny || result.passed + } + } + results[name] = result + }) + if err != nil { + if errors.Is(err, io.EOF) { + return passedAny, nil + } + return false, err + } + } +} + +type testResult struct { + ran bool + terminal bool + passed bool +} + +func parseFramedTestEvents(line string, visit func(action, name string)) { + for { + marker := strings.IndexByte(line, test2JSONMarker) + if marker < 0 { + return + } + line = line[marker:] + next := strings.IndexByte(line[1:], test2JSONMarker) + if next >= 0 { + next++ + } + framed := line + if next >= 0 { + framed = line[:next] + line = line[next:] + } else { + line = "" + } + if action, name, ok := parseFramedTestEvent(framed); ok { + visit(action, name) + } + } +} + +func parseFramedTestEvent(line string) (action, name string, ok bool) { + marker := strings.IndexByte(line, test2JSONMarker) + if marker < 0 { + return "", "", false + } + line = strings.TrimSuffix(line[marker+1:], "\n") + line = strings.TrimSuffix(line, "\r") + switch { + case strings.HasPrefix(line, "=== RUN "): + return "run", strings.TrimSpace(line[len("=== RUN "):]), true + case strings.HasPrefix(line, "--- PASS: "): + return "pass", parseFramedTestName(line[len("--- PASS: "):]), true + case strings.HasPrefix(line, "--- SKIP: "): + return "skip", parseFramedTestName(line[len("--- SKIP: "):]), true + case strings.HasPrefix(line, "--- FAIL: "): + return "fail", parseFramedTestName(line[len("--- FAIL: "):]), true + default: + return "", "", false + } +} + +func parseFramedTestName(line string) string { + name := strings.TrimSpace(line) + if i := strings.Index(name, " ("); i >= 0 && strings.HasSuffix(name, "s)") { + name = name[:i] + } + return name +} + +func rewrite(s string) string { + var rewritten strings.Builder + for _, r := range s { + switch { + case isSpace(r): + rewritten.WriteByte('_') + case !strconv.IsPrint(r): + quoted := strconv.QuoteRune(r) + rewritten.WriteString(quoted[1 : len(quoted)-1]) + default: + rewritten.WriteRune(r) + } + } + return rewritten.String() +} + +func isSpace(r rune) bool { + if r < 0x2000 { + switch r { + case '\t', '\n', '\v', '\f', '\r', ' ', 0x85, 0xA0, 0x1680: + return true + } + } else { + if r <= 0x200a { + return true + } + switch r { + case 0x2028, 0x2029, 0x202f, 0x205f, 0x3000: + return true + } + } + return false +} diff --git a/scripts/fast-check-events/main_test.go b/scripts/fast-check-events/main_test.go new file mode 100644 index 0000000..40fad5b --- /dev/null +++ b/scripts/fast-check-events/main_test.go @@ -0,0 +1,215 @@ +package main + +import ( + "errors" + "reflect" + "strings" + "testing" +) + +func TestSplitSelectorPreservesGoMatcherBoundaries(t *testing.T) { + for _, tc := range []struct { + pattern string + want [][]string + }{ + {pattern: "A/B", want: [][]string{{"A", "B"}}}, + {pattern: "[/]/[(]", want: [][]string{{"[/]", "[(]"}}}, + {pattern: "([)/][(])", want: [][]string{{"([)/][(])"}}}, + {pattern: "A/B|C/D", want: [][]string{{"A", "B"}, {"C", "D"}}}, + {pattern: `A\/B`, want: [][]string{{`A\/B`}}}, + } { + t.Run(tc.pattern, func(t *testing.T) { + if got := splitSelector(tc.pattern); !reflect.DeepEqual(got, tc.want) { + t.Fatalf("splitSelector(%q) = %#v, want %#v", tc.pattern, got, tc.want) + } + }) + } +} + +func TestSelectorMatchesCompleteNames(t *testing.T) { + for _, tc := range []struct { + selector string + name string + want bool + }{ + {selector: "^TestParent$/^Actual$", name: "TestParent", want: false}, + {selector: "^TestParent$/^Actual$", name: "TestParent/Actual", want: true}, + {selector: "^TestParent$/^Actual$", name: "TestParent/Actual/Grandchild", want: true}, + {selector: "^TestParent$/^Actual$", name: "TestParent/Missing", want: false}, + {selector: "^TestParent$/^Actual[/]?$", name: "TestParent/Actual", want: true}, + {selector: "^TestParent$/^Actual(/)?$", name: "TestParent/Actual", want: true}, + {selector: "^TestParent$/^(Actual|Other)$", name: "TestParent/Other", want: true}, + {selector: "^TestParent$/^Actual$|^TestOther$/^Child$", name: "TestOther/Child", want: true}, + {selector: "^TestParent$/^Actual$|^TestOther$/^Child$", name: "TestParent/Missing", want: false}, + } { + t.Run(tc.selector+"/"+tc.name, func(t *testing.T) { + selector, err := compileSelector(tc.selector) + if err != nil { + t.Fatal(err) + } + if got := selector.matches(tc.name); got != tc.want { + t.Fatalf("selector %q name %q matched %t, want %t", tc.selector, tc.name, got, tc.want) + } + }) + } +} + +func TestHasCompleteMatchIgnoresAncestors(t *testing.T) { + selector, err := compileSelector("^TestParent$/^Missing$") + if err != nil { + t.Fatal(err) + } + input := strings.NewReader("\x16=== RUN TestParent\n\x16--- PASS: TestParent (0.00s)\n") + found, err := hasCompleteMatch(input, selector) + if err != nil { + t.Fatal(err) + } + if found { + t.Fatal("ancestor run event counted as a complete selector match") + } +} + +func TestHasCompleteMatchFindsChild(t *testing.T) { + selector, err := compileSelector("^TestParent$/^Actual$") + if err != nil { + t.Fatal(err) + } + input := strings.NewReader("\x16=== RUN TestParent\n\x16=== RUN TestParent/Actual\n\x16--- PASS: TestParent/Actual (0.00s)\n") + found, err := hasCompleteMatch(input, selector) + if err != nil { + t.Fatal(err) + } + if !found { + t.Fatal("complete selector run event was not recognized") + } +} + +func TestHasCompleteMatchHandlesFramingBoundaries(t *testing.T) { + selector, err := compileSelector("^TestSelected$") + if err != nil { + t.Fatal(err) + } + input := strings.NewReader("ordinary output\x16=== RUN TestSelected\x16--- PASS: TestSelected (0.00s)\n") + found, err := hasCompleteMatch(input, selector) + if err != nil { + t.Fatal(err) + } + if !found { + t.Fatal("framed test events after ordinary output were not recognized") + } +} + +func TestHasCompleteMatchRequiresFramedPass(t *testing.T) { + selector, err := compileSelector("^TestSelected$") + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name string + input string + want bool + }{ + { + name: "synthetic run has no terminal evidence", + input: "\x16=== RUN TestSelected\n", + want: false, + }, + { + name: "unframed run and pass are ignored", + input: "=== RUN TestSelected\n--- PASS: TestSelected (0.00s)\n", + want: false, + }, + { + name: "unmatched framed run and pass are ignored", + input: "\x16=== RUN TestOther\n\x16--- PASS: TestOther (0.00s)\n", + want: false, + }, + { + name: "skip is not a passing terminal", + input: "\x16=== RUN TestSelected\n\x16--- SKIP: TestSelected (0.00s)\n", + want: false, + }, + { + name: "first fail terminal blocks a later pass in the same run", + input: "\x16=== RUN TestSelected\n\x16--- FAIL: TestSelected (0.00s)\n\x16--- PASS: TestSelected (0.00s)\n", + want: false, + }, + { + name: "first skip terminal blocks a later pass in the same run", + input: "\x16=== RUN TestSelected\n\x16--- SKIP: TestSelected (0.00s)\n\x16--- PASS: TestSelected (0.00s)\n", + want: false, + }, + { + name: "skip in an earlier package and pass in a later package qualify", + input: "\x16=== RUN TestSelected\n\x16--- SKIP: TestSelected (0.00s)\n\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n", + want: true, + }, + { + name: "pass in an earlier package and skip in a later package qualify", + input: "\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n\x16=== RUN TestSelected\n\x16--- SKIP: TestSelected (0.00s)\n", + want: true, + }, + { + name: "pass in an earlier package and fail in a later package retain match evidence", + input: "\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n\x16=== RUN TestSelected\n\x16--- FAIL: TestSelected (0.00s)\n", + want: true, + }, + { + name: "fail in an earlier package and pass in a later package qualify", + input: "\x16=== RUN TestSelected\n\x16--- FAIL: TestSelected (0.00s)\n\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n", + want: true, + }, + { + name: "pass without run is ignored", + input: "\x16--- PASS: TestSelected (0.00s)\n", + want: false, + }, + { + name: "framed run and pass qualify", + input: "\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n", + want: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + found, err := hasCompleteMatch(strings.NewReader(tc.input), selector) + if err != nil { + t.Fatal(err) + } + if found != tc.want { + t.Fatalf("hasCompleteMatch() = %t, want %t", found, tc.want) + } + }) + } +} + +func TestHasCompleteMatchPropagatesReaderError(t *testing.T) { + selector, err := compileSelector("^TestSelected$") + if err != nil { + t.Fatal(err) + } + wantErr := errors.New("reader failed") + input := readerWithError{ + data: "\x16=== RUN TestSelected\n\x16--- PASS: TestSelected (0.00s)\n", + err: wantErr, + } + found, err := hasCompleteMatch(input, selector) + if found { + t.Fatal("reader error was hidden by a passing event") + } + if !errors.Is(err, wantErr) { + t.Fatalf("hasCompleteMatch() error = %v, want %v", err, wantErr) + } +} + +type readerWithError struct { + data string + err error +} + +func (r readerWithError) Read(p []byte) (int, error) { + if r.data == "" { + return 0, r.err + } + n := copy(p, r.data) + return n, r.err +} diff --git a/scripts/fast-check.sh b/scripts/fast-check.sh new file mode 100755 index 0000000..06b1fc7 --- /dev/null +++ b/scripts/fast-check.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash + +set -euo pipefail + +go_cmd="${GO:-go}" +exact_toolchain="go1.26.8" + +# Keep this standalone entry point on the same pinned toolchain as Make and +# hosted CI. It is an iteration aid, not a replacement for the complete gate. +export GOTOOLCHAIN="${exact_toolchain}" +export GOWORK=off + +fail() { + printf 'fast check failed: %s\n' "$1" >&2 + exit 2 +} + +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd -P)" || fail 'fast-check.sh directory does not exist' +event_checker="${script_dir}/fast-check-events" + +require_selector() { + local name="$1" + local value="$2" + + [[ -n "${value}" ]] || fail "${name} is required; specify FAST_MODULE, FAST_PACKAGE, and FAST_TEST" + if [[ "${name}" != FAST_TEST && "${value}" == *[[:space:]]* ]]; then + fail "${name} must not contain whitespace" + fi + [[ "${value}" != -* ]] || fail "${name} must not begin with '-'" +} + +fast_module="${FAST_MODULE:-}" +fast_package="${FAST_PACKAGE:-}" +fast_test="${FAST_TEST:-}" +require_selector FAST_MODULE "${fast_module}" +require_selector FAST_PACKAGE "${fast_package}" +require_selector FAST_TEST "${fast_test}" + +case "${fast_module}" in + .) ;; + ./*) [[ "${fast_module}" != "./" ]] || fail 'FAST_MODULE must name a module directory' + ;; + *) fail 'FAST_MODULE must be . or a relative path beginning with ./' ;; +esac + +case "${fast_package}" in + .) ;; + ./*) [[ "${fast_package}" != "./" ]] || fail 'FAST_PACKAGE must name a package pattern' + ;; + *) fail 'FAST_PACKAGE must be . or a relative package pattern beginning with ./' ;; +esac + +# Resolve the selected module and every package matched by the Go pattern +# physically before running tests. The `...` package wildcard is valid in any +# path segment; only an actual `..` path segment is refused lexically. +module_path="${fast_module#./}" +package_parts=() +IFS='/' read -r -a package_parts <<< "${fast_package}" +module_parts=() +IFS='/' read -r -a module_parts <<< "${module_path}" +for part in "${module_parts[@]}" "${package_parts[@]}"; do + [[ "${part}" != ".." ]] || fail 'selectors must not contain a .. path segment' +done + +repo_root="$(pwd -P)" +case "${go_cmd}" in + */*) + case "${go_cmd}" in + /*) ;; + *) go_cmd="${repo_root}/${go_cmd}" ;; + esac + ;; +esac +module_root="$(cd "${fast_module}" 2>/dev/null && pwd -P)" || fail "module directory does not exist: ${fast_module}" +[[ -f "${module_root}/go.mod" ]] || fail "selected module has no go.mod: ${fast_module}" +if [[ "${module_root}" != "${repo_root}" && "${module_root}" != "${repo_root}"/* ]]; then + fail 'FAST_MODULE must resolve inside the current repository' +fi + +if ! package_dirs="$(cd "${module_root}" && GOFLAGS="${GOFLAGS:-} -deps=false" "${go_cmd}" list -json=false -f '{{.Dir}}' "${fast_package}")"; then + fail "package pattern could not be resolved: ${fast_package}" +fi +package_found=0 +while IFS= read -r package_dir; do + [[ -n "${package_dir}" ]] || continue + package_found=1 + package_root="$(cd "${package_dir}" 2>/dev/null && pwd -P)" || fail "package directory does not exist: ${fast_package}" + if [[ "${package_root}" != "${module_root}" && "${package_root}" != "${module_root}"/* ]]; then + fail 'FAST_PACKAGE must resolve inside selected module' + fi +done <<< "${package_dirs}" +((package_found == 1)) || fail "FAST_PACKAGE matched no packages: ${fast_package}" + +printf 'fast check: module=%s package=%s test=%s (focused selector only; not make check)\n' \ + "${fast_module}" "${fast_package}" "${fast_test}" +( + cd "${module_root}" + set +e + test_output="$("${go_cmd}" test -v -json=false -list= -bench= -fuzz= -skip= -c=false -count=1 -cpu=1 -exec= -run "${fast_test}" "${fast_package}" -args -test.v=test2json 2>&1)" + status=$? + set -e + printf '%s\n' "${test_output}" + if ((status != 0)); then + printf 'fast check failed: selected test command exited %d\n' "${status}" >&2 + exit "${status}" + fi + if ! printf '%s\n' "${test_output}" | (cd "${event_checker}" && GOFLAGS='' "${go_cmd}" run . --selector "${fast_test}"); then + printf 'fast check failed: FAST_TEST matched no compiled test in FAST_PACKAGE\n' >&2 + exit 2 + fi +) + +printf 'fast check passed: selected test only; complete public validation remains make check/CI\n' diff --git a/scripts/tooling_test.go b/scripts/tooling_test.go index fc19097..da309d5 100644 --- a/scripts/tooling_test.go +++ b/scripts/tooling_test.go @@ -41,7 +41,14 @@ func toolingRun(t *testing.T, root string, extra []string, args ...string) (stri defer cancel() cmd := exec.CommandContext(ctx, args[0], args[1:]...) cmd.Dir = root - cmd.Env = append(os.Environ(), "GOTOOLCHAIN=go1.26.8", "GOFLAGS=", "GOWORK=off") + env := make([]string, 0, len(os.Environ())+3+len(extra)) + for _, entry := range os.Environ() { + if strings.HasPrefix(entry, "FAST_MODULE=") || strings.HasPrefix(entry, "FAST_PACKAGE=") || strings.HasPrefix(entry, "FAST_TEST=") { + continue + } + env = append(env, entry) + } + cmd.Env = append(env, "GOTOOLCHAIN=go1.26.8", "GOFLAGS=", "GOWORK=off") cmd.Env = append(cmd.Env, extra...) data, err := cmd.CombinedOutput() return string(data), err @@ -57,7 +64,9 @@ func toolingGoWrapper(t *testing.T, root string) (string, string, string) { wrapperPath := filepath.Join(root, "logging-go") toolingFile(t, root, "logging-go", `#!/bin/sh set -eu -printf '%s\t%s\n' "${GOTOOLCHAIN:-}" "$*" >> "$TOOLING_GO_LOG" +if [ "${1:-}" != run ]; then + printf '%s\t%s\n' "${GOTOOLCHAIN:-}" "$*" >> "$TOOLING_GO_LOG" +fi exec "$TOOLING_REAL_GO" "$@" `, 0700) return wrapperPath, logPath, realGo @@ -66,7 +75,7 @@ exec "$TOOLING_REAL_GO" "$@" func toolingFixture(t *testing.T) string { t.Helper() root := t.TempDir() - for _, name := range []string{"Makefile", "go.mod", "LICENSE", "docs/DEPENDENCIES.md", "scripts/check-toolchain.sh", "scripts/check-licenses.sh", "scripts/check-offline-experiments.sh", "scripts/gofmt.sh", "scripts/fuzz-smoke.sh"} { + for _, name := range []string{"Makefile", "go.mod", "LICENSE", "docs/DEPENDENCIES.md", "scripts/check-toolchain.sh", "scripts/check-licenses.sh", "scripts/check-offline-experiments.sh", "scripts/gofmt.sh", "scripts/fuzz-smoke.sh", "scripts/fast-check.sh", "scripts/fast-check-events/main.go"} { data, err := os.ReadFile(filepath.Join("..", name)) if err != nil { t.Fatal(err) @@ -133,6 +142,820 @@ func TestToolingPinsNewerSystemGo(t *testing.T) { } } +func assertFastCheckRequiresExplicitSelectors(t *testing.T, root, fastCheck string) { + t.Helper() + for _, tc := range []struct { + name string + env []string + want string + }{ + {name: "missing module", env: []string{"FAST_PACKAGE=./cmd/gh-runnerd", "FAST_TEST=^TestFastSelected$"}, want: "FAST_MODULE"}, + {name: "missing package", env: []string{"FAST_MODULE=.", "FAST_TEST=^TestFastSelected$"}, want: "FAST_PACKAGE"}, + {name: "missing test", env: []string{"FAST_MODULE=.", "FAST_PACKAGE=./cmd/gh-runnerd"}, want: "FAST_TEST"}, + } { + t.Run(tc.name, func(t *testing.T) { + out, err := toolingRun(t, root, tc.env, "bash", fastCheck) + if err == nil || !strings.Contains(out, tc.want) { + t.Fatalf("fast check accepted %s: err=%v output=%s", tc.name, err, out) + } + }) + } +} + +func TestFastCheckHandlesJSONGOFLAGS(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSelected(t *testing.T) {} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-json -race"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestFastSelected$$", + "fast", + ) + if err != nil { + t.Fatalf("make fast did not normalize JSON output while preserving other GOFLAGS: %s", out) + } +} + +func TestFastCheckPreservesEnvironmentSelectorDollar(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestB(t *testing.T) {} +`, 0600) + out, err := toolingRun(t, root, []string{ + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestA$.*", + }, "make", "fast") + if err == nil || !strings.Contains(out, "FAST_TEST matched no compiled test") { + t.Fatalf("environment selector was changed before matching: err=%v output=%s", err, out) + } +} + +func TestFastCheckExecutesSelectedTestWithListGOFLAGS(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSelected(t *testing.T) { + t.Fatal("selected-test-ran") +} +`, 0600) + for _, tc := range []struct { + name, test, want string + }{ + {name: "selected test runs", test: "^TestFastSelected$", want: "selected-test-ran"}, + {name: "no matching test remains an error", test: "^NoSuchTest$", want: "fast check failed"}, + } { + t.Run(tc.name, func(t *testing.T) { + out, err := toolingRun(t, root, []string{"GOFLAGS=-list=."}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST="+strings.ReplaceAll(tc.test, "$", "$$"), + "fast", + ) + if err == nil || !strings.Contains(out, tc.want) { + t.Fatalf("GOFLAGS=-list=. accepted %s: err=%v output=%s", tc.name, err, out) + } + }) + } +} + +func TestFastCheckRequiresCompleteSubtestSelector(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastCheckParent(t *testing.T) { + t.Run("ActualChild", func(t *testing.T) {}) + t.Run("Actual", func(t *testing.T) {}) + t.Run("Other", func(t *testing.T) {}) + t.Run("hello world", func(t *testing.T) {}) +} +`, 0600) + for _, tc := range []struct { + name, selector, wantOutput string + wantErr bool + }{ + { + name: "missing child", + selector: "^TestFastCheckParent$/^MissingChild$", + wantErr: true, + wantOutput: "FAST_TEST matched no compiled test", + }, + { + name: "actual child", + selector: "^TestFastCheckParent$/^ActualChild$", + }, + { + name: "subtest whitespace", + selector: "^TestFastCheckParent$/^hello world$", + }, + { + name: "slash in character class", + selector: "^TestFastCheckParent$/^Actual[/]?$", + }, + { + name: "slash in parentheses", + selector: "^TestFastCheckParent$/^Actual(/)?$", + }, + { + name: "grouped alternation", + selector: "^TestFastCheckParent$/^(Actual|Other)$", + }, + { + name: "top-level alternation", + selector: "^TestFastCheckParent$/^ActualChild$|^TestFastCheckParent$/^Other$", + }, + } { + t.Run(tc.name, func(t *testing.T) { + out, err := toolingRun(t, root, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST="+strings.ReplaceAll(tc.selector, "$", "$$"), + "fast", + ) + if (err != nil) != tc.wantErr { + t.Fatalf("selector %q returned err=%v, want error=%t; output=%s", tc.selector, err, tc.wantErr, out) + } + if tc.wantOutput != "" && !strings.Contains(out, tc.wantOutput) { + t.Fatalf("selector %q output lacks %q: %s", tc.selector, tc.wantOutput, out) + } + }) + } +} + +func TestFastCheckRejectsSyntheticTestMainRun(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import ( + "fmt" + "os" + "testing" +) + +func TestMain(m *testing.M) { + fmt.Println("=== RUN NoSuchTest") + os.Exit(m.Run()) +} + +func TestFastActual(t *testing.T) {} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-v"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^NoSuchTest$$", + "fast", + ) + if err == nil || !strings.Contains(out, "FAST_TEST matched no compiled test") { + t.Fatalf("synthetic TestMain run was treated as selected-test evidence: err=%v output=%s", err, out) + } +} + +func TestFastCheckRejectsSpoofedTestMainEvents(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import ( + "fmt" + "os" + "testing" +) + +func TestMain(m *testing.M) { + fmt.Println("=== RUN NoSuchTest") + fmt.Println("--- PASS: NoSuchTest (0.00s)") + os.Exit(m.Run()) +} + +func TestFastActual(t *testing.T) {} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-json"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^NoSuchTest$$", + "fast", + ) + if err == nil || !strings.Contains(out, "FAST_TEST matched no compiled test") { + t.Fatalf("spoofed TestMain run/pass events were treated as selected-test evidence: err=%v output=%s", err, out) + } +} + +func TestFastCheckRejectsSkippedSelectedTest(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSkipped(t *testing.T) { + t.Skip("bounded fixture skip") +} +`, 0600) + out, err := toolingRun(t, root, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestFastSkipped$$", + "fast", + ) + if err == nil || !strings.Contains(out, "FAST_TEST matched no compiled test") { + t.Fatalf("skipped test was treated as passing evidence: err=%v output=%s", err, out) + } +} + +func TestFastCheckAcceptsPassingSameNameAcrossPackages(t *testing.T) { + for _, tc := range []struct { + name, firstDir, firstPackage, firstBody, secondDir, secondPackage, secondBody, wantOutput string + wantErr bool + }{ + { + name: "skip then pass", + firstDir: "a-skip", + firstPackage: "skipfixture", + firstBody: "\tt.Skip(\"bounded fixture skip\")\n", + secondDir: "b-pass", + secondPackage: "passfixture", + secondBody: "", + }, + { + name: "pass then skip", + firstDir: "a-pass", + firstPackage: "passfixture", + firstBody: "", + secondDir: "b-skip", + secondPackage: "skipfixture", + secondBody: "\tt.Skip(\"bounded fixture skip\")\n", + }, + { + name: "pass then fail", + firstDir: "a-pass", + firstPackage: "passfixture", + firstBody: "", + secondDir: "b-fail", + secondPackage: "failfixture", + secondBody: "\tt.Fatal(\"bounded fixture failure\")\n", + wantOutput: "fast check failed: selected test command exited", + wantErr: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "internal/fast-check-multi/"+tc.firstDir+"/fast_check_test.go", `package `+tc.firstPackage+` + +import "testing" + +func TestFastSameName(t *testing.T) { +`+tc.firstBody+`} +`, 0600) + toolingFile(t, root, "internal/fast-check-multi/"+tc.secondDir+"/fast_check_test.go", `package `+tc.secondPackage+` + +import "testing" + +func TestFastSameName(t *testing.T) { +`+tc.secondBody+`} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-p=1"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./internal/fast-check-multi/...", + "FAST_TEST=^TestFastSameName$$", + "fast", + ) + if (err != nil) != tc.wantErr { + t.Fatalf("same-name package order %s returned err=%v, want error=%t: %s", tc.name, err, tc.wantErr, out) + } + if tc.wantOutput != "" && !strings.Contains(out, tc.wantOutput) { + t.Fatalf("same-name package order %s output lacks %q: %s", tc.name, tc.wantOutput, out) + } + }) + } +} + +func TestFastCheckNeutralizesGoTestExecutionFlags(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import ( + "os" + "testing" +) + +func fastCheckRecord(t testing.TB, marker string) { + t.Helper() + path := os.Getenv("TOOLING_FAST_SENTINEL_LOG") + if path == "" { + t.Fatal("TOOLING_FAST_SENTINEL_LOG is not set") + } + file, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteString(marker + "\n"); err != nil { + t.Fatal(err) + } +} + +func TestFastCheckSelected(t *testing.T) { + fastCheckRecord(t, "selected-test") +} + +func BenchmarkFastCheckUnexpected(b *testing.B) { + fastCheckRecord(b, "unexpected-benchmark") + b.Fatal("unexpected benchmark ran") +} + +func BenchmarkFastCheckFalseSuccess(b *testing.B) { + fastCheckRecord(b, "false-success-benchmark") +} + +func FuzzFastCheckUnexpected(f *testing.F) { + f.Add("fixture-seed") + f.Fuzz(func(t *testing.T, _ string) { + fastCheckRecord(t, "unexpected-fuzz") + t.Fatal("unexpected fuzz ran") + }) +} + +func FuzzFastCheckFalseSuccess(f *testing.F) { + f.Add("fixture-seed") + f.Fuzz(func(t *testing.T, _ string) { + fastCheckRecord(t, "false-success-fuzz") + }) +} +`, 0600) + sentinelLog := filepath.Join(root, "fast-check-sentinel.log") + for _, tc := range []struct { + name, goflags, test, wantOutput string + wantErr bool + wantSentinel string + }{ + { + name: "benchmark cannot expand selected test", + goflags: "-bench=. -benchtime=1x", + test: "^TestFastCheckSelected$", + wantSentinel: "selected-test\n", + }, + { + name: "inherited run and count cannot replace bounded selector", + goflags: "-run=^NoSuchTest$ -count=5", + test: "^TestFastCheckSelected$", + wantSentinel: "selected-test\n", + }, + { + name: "fuzz cannot expand selected test", + goflags: "-fuzz=^FuzzFastCheckUnexpected$ -fuzztime=1x", + test: "^TestFastCheckSelected$", + wantSentinel: "selected-test\n", + }, + { + name: "skip and build-only cannot suppress selected test", + goflags: "-skip=^TestFastCheckSelected$ -c -count=5", + test: "^TestFastCheckSelected$", + wantSentinel: "selected-test\n", + }, + { + name: "benchmark cannot satisfy missing test", + goflags: "-bench=^BenchmarkFastCheckFalseSuccess$ -benchtime=1x", + test: "^NoSuchTest$", + wantErr: true, + wantOutput: "FAST_TEST matched no compiled test", + }, + { + name: "fuzz cannot satisfy missing test", + goflags: "-fuzz=^FuzzFastCheckFalseSuccess$ -fuzztime=1x", + test: "^NoSuchTest$", + wantErr: true, + wantOutput: "FAST_TEST matched no compiled test", + }, + } { + t.Run(tc.name, func(t *testing.T) { + if err := os.WriteFile(sentinelLog, nil, 0600); err != nil { + t.Fatal(err) + } + env := []string{ + "GOFLAGS=" + tc.goflags, + "TOOLING_FAST_SENTINEL_LOG=" + sentinelLog, + } + out, err := toolingRun(t, root, env, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST="+strings.ReplaceAll(tc.test, "$", "$$"), + "fast", + ) + if (err != nil) != tc.wantErr { + t.Fatalf("GOFLAGS=%q test=%q returned err=%v, want error=%t; output=%s", tc.goflags, tc.test, err, tc.wantErr, out) + } + if tc.wantOutput != "" && !strings.Contains(out, tc.wantOutput) { + t.Fatalf("GOFLAGS=%q test=%q output lacks %q: %s", tc.goflags, tc.test, tc.wantOutput, out) + } + gotSentinel := toolingReadFile(t, sentinelLog) + if tc.wantSentinel != "" && gotSentinel != tc.wantSentinel { + t.Fatalf("GOFLAGS=%q test=%q sentinel log = %q, want %q; output=%s", tc.goflags, tc.test, gotSentinel, tc.wantSentinel, out) + } + if tc.wantSentinel == "" && gotSentinel != "" { + t.Fatalf("GOFLAGS=%q test=%q ran an execution-expanding target: sentinel log = %q; output=%s", tc.goflags, tc.test, gotSentinel, out) + } + }) + } +} + +func TestFastCheckNeutralizesExecAndCPUControls(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import ( + "os" + "testing" +) + +func TestFastCheckSelected(t *testing.T) { + path := os.Getenv("TOOLING_FAST_SENTINEL_LOG") + if path == "" { + t.Fatal("TOOLING_FAST_SENTINEL_LOG is not set") + } + file, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600) + if err != nil { + t.Fatal(err) + } + defer file.Close() + if _, err := file.WriteString("selected-test\n"); err != nil { + t.Fatal(err) + } + t.Fatal("selected-test-ran") +} +`, 0600) + sentinelLog := filepath.Join(root, "fast-check-sentinel.log") + for _, tc := range []struct { + name, goflags, wantSentinel string + wantErr bool + }{ + { + name: "exec cannot bypass selected test", + goflags: "-exec=true", + wantErr: true, + wantSentinel: "selected-test\n", + }, + { + name: "cpu list cannot repeat selected test", + goflags: "-cpu=1,2", + wantErr: true, + wantSentinel: "selected-test\n", + }, + { + name: "dry run cannot satisfy execution evidence", + goflags: "-n", + wantErr: true, + wantSentinel: "", + }, + } { + t.Run(tc.name, func(t *testing.T) { + if err := os.WriteFile(sentinelLog, nil, 0600); err != nil { + t.Fatal(err) + } + out, err := toolingRun(t, root, []string{ + "GOFLAGS=" + tc.goflags, + "TOOLING_FAST_SENTINEL_LOG=" + sentinelLog, + }, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestFastCheckSelected$$", + "fast", + ) + if (err != nil) != tc.wantErr { + t.Fatalf("GOFLAGS=%q returned err=%v, want error=%t; output=%s", tc.goflags, err, tc.wantErr, out) + } + if got := toolingReadFile(t, sentinelLog); got != tc.wantSentinel { + t.Fatalf("GOFLAGS=%q sentinel log = %q, want %q; output=%s", tc.goflags, got, tc.wantSentinel, out) + } + }) + } +} + +func TestFastCheckIgnoresTestMainOutputForMatchEvidence(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import ( + "fmt" + "os" + "testing" +) + +func TestMain(m *testing.M) { + fmt.Println("ok\tfrom TestMain") + os.Exit(m.Run()) +} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-v"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^NoSuchTest$$", + "fast", + ) + if err == nil || !strings.Contains(out, "FAST_TEST matched no compiled test") { + t.Fatalf("TestMain output was treated as selected-test evidence: err=%v output=%s", err, out) + } +} + +func TestFastCheckPreservesRaceGOFLAGS(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +var fastCheckRaceValue int + +func TestFastCheckRaceTarget(t *testing.T) { + start := make(chan struct{}) + done := make(chan struct{}, 2) + for i := 0; i < 2; i++ { + go func() { + <-start + for j := 0; j < 1000; j++ { + fastCheckRaceValue = j + } + done <- struct{}{} + }() + } + close(start) + <-done + <-done +} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-race"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestFastCheckRaceTarget$$", + "fast", + ) + if err == nil || !strings.Contains(out, "DATA RACE") { + t.Fatalf("GOFLAGS=-race was not preserved by fast check: err=%v output=%s", err, out) + } +} + +func TestFastCheckResolvesPackageSymlinks(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "linked/selected/fast_check_test.go", `package selected + +import "testing" + +func TestFastSymlinkSelected(t *testing.T) {} +`, 0600) + toolingFile(t, root, "linked/other/fast_check_test.go", `package other + +import "testing" + +func TestFastSymlinkNotSelected(t *testing.T) { + t.Fatal("wrong test ran") +} +`, 0600) + toolingFile(t, root, "linked/selected/capacity/fast_check_test.go", `package capacity + +import "testing" + +func TestFastWildcardSelected(t *testing.T) {} +`, 0600) + if err := os.Symlink("../linked", filepath.Join(root, "scripts", "package-link")); err != nil { + t.Fatal(err) + } + out, err := toolingRun(t, root, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./scripts/package-link/...", + "FAST_TEST=^TestFastSymlinkSelected$$", + "fast", + ) + if err != nil { + t.Fatalf("valid in-module package symlink failed: %s", out) + } + out, err = toolingRun(t, root, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./.../capacity", + "FAST_TEST=^TestFastWildcardSelected$$", + "fast", + ) + if err != nil { + t.Fatalf("valid general package wildcard failed: %s", out) + } + + outside := t.TempDir() + toolingFile(t, outside, "escape_test.go", `package external + +import "testing" + +func TestFastSymlinkEscape(t *testing.T) {} +`, 0600) + if err := os.Symlink(outside, filepath.Join(root, "scripts", "outside-link")); err != nil { + t.Fatal(err) + } + out, err = toolingRun(t, root, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./scripts/outside-link", + "FAST_TEST=^TestFastSymlinkEscape$$", + "fast", + ) + if err == nil || !strings.Contains(out, "FAST_PACKAGE must resolve inside selected module") { + t.Fatalf("package symlink escaping module was accepted: err=%v output=%s", err, out) + } +} + +func TestFastCheckRunsOnlyTheExplicitSelector(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSelected(t *testing.T) {} + +func TestFastNotSelected(t *testing.T) { + t.Fatal("fast check ran an unselected test") +} +`, 0600) + toolingFile(t, root, "internal/no_tests/empty.go", "package no_tests\n", 0600) + toolingFile(t, root, "experiments/g01-scaleset/fast_check_test.go", `package fixture + +import "testing" + +func TestNestedFastSelected(t *testing.T) {} +`, 0600) + + wrapper, logPath, realGo := toolingGoWrapper(t, root) + env := []string{"GO=" + wrapper, "TOOLING_REAL_GO=" + realGo, "TOOLING_GO_LOG=" + logPath} + for _, tc := range []struct { + name, module, packagePath, test, invocation string + }{ + { + name: "root module", + module: ".", + packagePath: "./...", + test: "^TestFastSelected$", + invocation: "go1.26.8\tlist -json=false -f {{.Dir}} ./...\ngo1.26.8\ttest -v -json=false -list= -bench= -fuzz= -skip= -c=false -count=1 -cpu=1 -exec= -run ^TestFastSelected$ ./... -args -test.v=test2json", + }, + { + name: "nested module", + module: "./experiments/g01-scaleset", + packagePath: ".", + test: "^TestNestedFastSelected$", + invocation: "go1.26.8\tlist -json=false -f {{.Dir}} .\ngo1.26.8\ttest -v -json=false -list= -bench= -fuzz= -skip= -c=false -count=1 -cpu=1 -exec= -run ^TestNestedFastSelected$ . -args -test.v=test2json", + }, + } { + t.Run(tc.name, func(t *testing.T) { + if err := os.WriteFile(logPath, nil, 0600); err != nil { + t.Fatal(err) + } + out, err := toolingRun(t, root, env, "make", + "FAST_MODULE="+tc.module, + "FAST_PACKAGE="+tc.packagePath, + "FAST_TEST="+strings.ReplaceAll(tc.test, "$", "$$"), + "fast", + ) + if err != nil { + t.Fatalf("focused selector failed: %s", out) + } + if !strings.Contains(out, "focused selector only") || !strings.Contains(out, "complete public validation remains make check/CI") { + t.Fatalf("focused selector output did not identify its bounded scope: %s", out) + } + log := strings.TrimSpace(toolingReadFile(t, logPath)) + if log != tc.invocation { + t.Fatalf("focused selector invocation = %q, want %q", log, tc.invocation) + } + }) + } +} + +func TestFastCheckResolvesRelativeGoOverride(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "experiments/g01-scaleset/fast_check_test.go", `package fixture + +import "testing" + +func TestNestedFastSelected(t *testing.T) {} +`, 0600) + wrapper, logPath, realGo := toolingGoWrapper(t, root) + data, err := os.ReadFile(wrapper) + if err != nil { + t.Fatal(err) + } + toolingFile(t, root, "tools/go", string(data), 0700) + out, err := toolingRun(t, root, []string{ + "GO=./tools/go", + "TOOLING_REAL_GO=" + realGo, + "TOOLING_GO_LOG=" + logPath, + }, "make", + "FAST_MODULE=./experiments/g01-scaleset", + "FAST_PACKAGE=.", + "FAST_TEST=^TestNestedFastSelected$$", + "fast", + ) + if err != nil { + t.Fatalf("relative Go override failed after changing directories: %s", out) + } +} + +func TestFastCheckDisablesDependencyExpansionForPackageResolution(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/main.go", `package main + +import "fmt" + +var _ = fmt.Sprint + +func main() {} +`, 0600) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSelected(t *testing.T) {} +`, 0600) + out, err := toolingRun(t, root, []string{"GOFLAGS=-deps"}, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./cmd/gh-runnerd", + "FAST_TEST=^TestFastSelected$$", + "fast", + ) + if err != nil { + t.Fatalf("dependency expansion escaped selected module during package resolution: %s", out) + } +} + +func TestFastCheckRequiresExplicitSelectors(t *testing.T) { + root := toolingFixture(t) + fastCheck, err := filepath.Abs("fast-check.sh") + if err != nil { + t.Fatal(err) + } + assertFastCheckRequiresExplicitSelectors(t, root, fastCheck) + + repoRoot, err := filepath.Abs("..") + if err != nil { + t.Fatal(err) + } + if out, err := toolingRun(t, repoRoot, nil, "make", + "FAST_MODULE=.", + "FAST_PACKAGE=./scripts", + "FAST_TEST=^TestFastCheckRequiresExplicitSelectorsUnderMake$$", + "fast", + ); err != nil { + t.Fatalf("make fast negative-selector regression failed: %s", out) + } +} + +func TestFastCheckRequiresExplicitSelectorsUnderMake(t *testing.T) { + root := toolingFixture(t) + fastCheck, err := filepath.Abs("fast-check.sh") + if err != nil { + t.Fatal(err) + } + assertFastCheckRequiresExplicitSelectors(t, root, fastCheck) +} + +func TestFastCheckRejectsInvalidSelectors(t *testing.T) { + root := toolingFixture(t) + toolingFile(t, root, "cmd/gh-runnerd/fast_check_test.go", `package main + +import "testing" + +func TestFastSelected(t *testing.T) {} +`, 0600) + wrapper, logPath, realGo := toolingGoWrapper(t, root) + env := []string{"GO=" + wrapper, "TOOLING_REAL_GO=" + realGo, "TOOLING_GO_LOG=" + logPath} + for _, tc := range []struct { + name, module, packagePath, test string + }{ + {name: "missing module directory", module: "./missing", packagePath: "./cmd/gh-runnerd", test: "^TestFastSelected$"}, + {name: "module without go.mod", module: "./cmd/gh-runnerd", packagePath: "./", test: "^TestFastSelected$"}, + {name: "escaping module path", module: "./../", packagePath: "./cmd/gh-runnerd", test: "^TestFastSelected$"}, + {name: "missing package directory", module: ".", packagePath: "./missing", test: "^TestFastSelected$"}, + {name: "escaping package path", module: ".", packagePath: "./../outside", test: "^TestFastSelected$"}, + {name: "no matching test", module: ".", packagePath: "./cmd/gh-runnerd", test: "^NoSuchTest$"}, + {name: "malformed test regexp", module: ".", packagePath: "./cmd/gh-runnerd", test: "["}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := os.WriteFile(logPath, nil, 0600); err != nil { + t.Fatal(err) + } + out, err := toolingRun(t, root, env, "make", + "FAST_MODULE="+tc.module, + "FAST_PACKAGE="+tc.packagePath, + "FAST_TEST="+strings.ReplaceAll(tc.test, "$", "$$"), + "fast", + ) + if err == nil || !strings.Contains(out, "fast check failed") { + t.Fatalf("invalid %s was accepted: err=%v output=%s", tc.name, err, out) + } + }) + } +} + func TestToolingCheckRequiresExecutableLink(t *testing.T) { root := toolingFixture(t) if out, err := toolingRun(t, root, nil, "make", "check"); err != nil {