Skip to content

test(ci): run unit tests and fix the suite's load-sensitive failures - #1308

Closed
skevetter wants to merge 5 commits into
fix/redaction-machine-readable-outputfrom
test/ci-unit-tests-and-suite-reliability
Closed

skevetter wants to merge 5 commits into
fix/redaction-machine-readable-outputfrom
test/ci-unit-tests-and-suite-reliability

Conversation

@skevetter

Copy link
Copy Markdown
Contributor

Stacked on #1307. Review that first; this branch is based on it.

Summary

No workflow ran go test. The only occurrence in .github/workflows compiled the e2e binary with go test -c, and the e2e suites need Docker plus a built CLI, so the 401 unit test files were never executed on a pull request.

Turning the suite on for the first time surfaced three kinds of problem, addressed here.

Changes

ci: run Go unit tests on pull requests — adds a unit-tests job mirroring task cli:test: race detector, coverage profile, and the same exclusion of /test and /e2e. Coverage is uploaded as an artifact.

The job is deliberately report-only, and one line from required. Making it required is the point of the job, but it would also mean a slow runner can block an unrelated pull. That tradeoff is worth deciding deliberately rather than by default.

fix(docker): report a missing image when the inspect poll times out — a real product defect, not test tuning. waitForLocalImage documents that it returns the last ErrImageNotFound once the poll timeout elapses, but that only held while the poll was not interrupted mid-command. When the deadline fired while a docker inspect was still running, the context killed that process and signal: killed was returned instead.

This matters beyond the message: EnsureImage branches on that error to decide that a devsy-built image must never be resolved by a registry pull. Returning the wrong error changes behavior under load. The fix treats a deadline-expired poll as a miss and lets the deadline decide; a cancelled parent context still propagates.

test: make two environment-dependent tests hermetic — TestFindDarwinDockerCLIRancherDesktopPath searched a candidate list beginning with /usr/local/bin/docker, so installing Docker made it find the real binary rather than the fixture. TestChangedFilesDetectsWorkingTree created a throwaway repo and committed to it, which fails for any contributor who sets commit.gpgsign globally — likely most of them, since this project requires signed commits.

fix(docker): widen the post-deadline diagnostic budget — WaitContainerRunning re-inspects without the deadline so a daemon error survives into the final message, but that retry had a 250ms budget inline. On a loaded machine the retry is itself killed and the diagnostic degrades to signal: killed.

Verification

With the poll fix, an untruncated uncached run of the full suite under the CI command returns exit 0 with zero failures, where before it failed three to five tests per run. The new TestEnsureImage_PollDeadlineDuringInspectReportsNotFound forces the deadline to land inside an inspect and fails with the original inspect image: signal: killed symptom if the fix is reverted.

Not verified

I could not confirm the job's result on an Ubuntu runner from a macOS machine. Three tests (TestEnsureImage_*, TestWaitContainerRunning_PreservesDeadlineExceededWithInspectError) still fail on this machine under a full parallel run, but pass in isolation on both main and this branch, and main passes them at the same load average, so the cause is contention rather than the code. The single remaining wall-clock assertion, TestWaitContainerRunningFailsFastOnExit, completes in about 0.2s against a 5s budget.

skevetter and others added 4 commits September 27, 2026 23:24
No workflow executed `go test`. The only occurrence compiled the e2e
binary with `go test -c`, and the e2e suites need Docker and a built
CLI, so the 401 unit test files were never run on a pull request. That
is how a redactor bug and eight failing tests in pkg/agent/delivery
survived unnoticed.

Add a unit-tests job mirroring `task cli:test`: race detector, coverage
profile, -p 4 to keep the wall-clock assertions from being exceeded on a
busy runner, and the same exclusion of /test and /e2e. It uploads the
coverage profile as an artifact.

The job is report-only, and one line away from required. The suite's
intermittent failures were a poll defect, fixed in this branch; the one
remaining wall-clock test, TestWaitContainerRunningFailsFastOnExit, has a
wide margin. Promoting it means a slow runner can block an unrelated
pull.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both fail on a developer machine for reasons unrelated to the code under
test.

TestFindDarwinDockerCLIRancherDesktopPath searched the full candidate
list, which begins with machine-wide locations such as
/usr/local/bin/docker. Installing Docker therefore made the search return
the real binary instead of the fixture. Check that the Rancher location
is among the candidates and look it up on its own.

TestChangedFilesDetectsWorkingTree created a throwaway repository and
committed to it. This project requires signed commits, so contributors
commonly set commit.gpgsign globally, and that setting is inherited by
the throwaway repo, where no signing key is available; the setup commit
exited 128. Disable signing in the fixture repository.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
waitForLocalImage documents that it returns the last ErrImageNotFound
once the poll timeout elapses, but that only held while the poll was not
interrupted mid-command. When the deadline fires while a docker inspect is
still in flight, the context kills that process and the "signal: killed"
failure is returned instead. Callers branch on that error to decide that a
built image must never be resolved by a pull, so the wrong error changes
behavior under load rather than only changing a message.

Treat a deadline-expired poll as a miss and let the deadline decide the
outcome. A cancelled parent context still propagates, so callers keep
seeing the cancellation they asked for.

This was the cause of the intermittent EnsureImage failures: with the fix,
the full suite passes where before it failed three to five tests per run.
The new test forces the deadline to land inside an inspect and fails with
the original "inspect image: signal: killed" symptom if the fix is
reverted.

Also fix a race in TestCommandDevContainer_CancelWrapsCtxErr, whose helper
goroutine gave up after one second and cancelled regardless, so on a slow
start it cancelled while the driver was still in the inspect call
preceding exec.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WaitContainerRunning re-inspects without the poll deadline so a daemon
error survives into the final message, but that retry had a 250ms budget
hard-coded inline. On a loaded machine the retry is itself killed and the
diagnostic degrades to "signal: killed", which is what
TestWaitContainerRunning_PreservesDeadlineExceededWithInspectError
asserts against.

Name the budget and raise it to 2s. It only runs on the slow path, where
the alternative is losing the error entirely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1792e1db-ceb2-43b5-8bf7-abee2149b1db

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

actionlint runs shellcheck over `run:` blocks, and SC2046 flagged the
`$(go list ...)` substitution in the unit test step. The word splitting
is the point there: the package list is meant to become arguments.

Verify the scope first: ./pkg/... ./cmd/... is not a substitute, since
the list also covers the root module, hack/ and providers/. Keep the
substitution and disable the check for that line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skevetter
skevetter marked this pull request as ready for review October 3, 2026 03:33
@mergify

mergify Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

This pull request does not currently match the merge queue conditions, so it cannot be queued from here. The box comes back if it matches again.

@skevetter skevetter closed this Oct 3, 2026
@skevetter
skevetter deleted the test/ci-unit-tests-and-suite-reliability branch November 7, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant