Skip to content

A visual change is Chromatic's to report, not this job's - #189

Merged
abernier merged 1 commit into
mainfrom
chromatic-exit-zero
Aug 13, 2026
Merged

A visual change is Chromatic's to report, not this job's#189
abernier merged 1 commit into
mainfrom
chromatic-exit-zero

Conversation

@abernier

Copy link
Copy Markdown
Member

#186 made a changed Chromatic build fail chromatic-job, so that a change nobody accepted could not read as green. Right rule, wrong place.

Chromatic posts its own UI Tests check on the commit. It is red while a build has unaccepted changes and it clears itself the moment someone accepts -- no re-run of anything. chromatic-job failing on top of that adds no signal and costs two:

  • a pull request that meant its change reads as broken rather than as waiting for review (Upgrade three 0.165.0 -> 0.181.2聽#166: 68 snapshots, all expected from three 0.165 -> 0.181);
  • the one failure worth seeing here -- an upload that did not work, an expired token, an archive that never arrived -- looks exactly like 68 snapshots pending review.

So --exit-zero-on-changes off main. This job answers "did the upload work?"; Chromatic answers "should this have changed?". --auto-accept-changes on main is unchanged.

Before merging: the GitHub app

There is no UI Tests check on this repository today -- the check-runs on #166's head are the Actions jobs and nothing else, so the Chromatic GitHub app is not installed:

$ gh api repos/pmndrs/examples/commits/<sha>/check-runs --jq '.check_runs[] | "\(.app.slug)\t\(.name)"'
github-actions  test-job (1)
github-actions  chromatic-job
...

Until it is installed (chromatic.com -> the project -> Manage -> Integrations -> GitHub), this flag removes the gate rather than moving it: a changed build reports nothing at all, which is how build 42 went green with a change waiting unreviewed.

馃 Generated with Claude Code

`chromatic-job` fails on a changed build today, which #186 introduced on
purpose: a change nobody accepted should not read as green. It is the right
rule enforced in the wrong place.

Chromatic already posts a UI Tests check on the commit, red until the build
is accepted, and it clears itself the moment someone accepts -- no re-run of
anything. This job failing on top of that adds nothing and costs two things.
A pull request that meant its change reads as broken rather than as waiting
for review. And the one failure worth seeing here -- an upload that did not
work, a token that expired, an archive that never arrived -- looks exactly
like 68 snapshots pending review.

So `--exit-zero-on-changes` off main: this job answers "did the upload
work?", Chromatic answers "should this have changed?".

This needs the Chromatic GitHub app installed on the repository -- it is
what publishes UI Tests. Without it a changed build now reports nothing at
all, which is how build 42 went green with a change waiting unreviewed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abernier
abernier merged commit 9b63e8f into main Aug 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant