Skip to content

mutate: a test that fails with nothing broken fails the run - #118

Merged
aledbf merged 1 commit into
mainfrom
mutate-broken-tests-fail
Nov 18, 2026
Merged

aledbf merged 1 commit into
mainfrom
mutate-broken-tests-fail

Conversation

@aledbf

@aledbf aledbf commented Nov 18, 2026

Copy link
Copy Markdown
Member

On #117 the Gate job was green while its mutate step printed --- FAIL for two tests. mutate ran each test once in its copy of the module, said the failure, asked that test about no edit, and passed. A FAIL in a green job teaches everyone to read past FAILs.

The policy. RunAll still asks every edit, then returns an error naming each test that failed with nothing broken (Tally.Broken), and mutate exits 2. cmd/mutate/main.go is not touched: its existing error path already exits 2.

The two tests that failed so:

  • versions/gate TestThisRepositoryPassesTheGate runs git ls-files, and the copy had no repository. The copy is now a git repository of its own (git init + git add -A), never the checkout's .git, and a run's reset leaves it alone. The same holds for any repository's test that asks git what the module has.
  • internal/mutate TestARunsTemporaryDirectoryTakesASocketFromATestWithALongName passes under go test, but under mutate's own run its TMPDIR is a run's directory 17 bytes deeper, and the socket path was 110 of the 108 bytes allowed. The test now sets the short TMPDIR it means.

The site's mutate page says both: exit 2 for a test that fails with nothing broken, and the copy as a repository of its own.

For the other repositories: with this version, a test that fails only in mutate's copy turns their mutate step red instead of printing FAIL in a green job.

Checked: task test, task lint, task site, and task mutate (38 refused, 0 survived). A mutate run over #117's range (-base adeb9f5 -max 60) shows no test failing with nothing broken. Each new or changed test fails without its fix: the socket test with the CI run's TMPDIR, and the git test with not a git repository.

🤖 Generated with Claude Code

…copy answers git

A test that failed in mutate's copy of the module was said and left out, and the run passed: a
green check over a FAIL. It now asks every edit and then exits 2 naming the test.

The two that failed so here: versions/gate's TestThisRepositoryPassesTheGate asks git which files
the module holds, and the copy had no repository - it now is one of its own, every file it holds
staged, never the checkout's .git; and the socket test's path, under mutate's own run, was 17
bytes deeper than 108 allow - it now names the short TMPDIR it holds the tree to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aledbf
aledbf merged commit 612636a into main Nov 18, 2026
3 checks passed
@aledbf
aledbf deleted the mutate-broken-tests-fail branch November 18, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant