Skip to content

Add Argos visual tests alongside Chromatic - #765

Merged
nedtwigg merged 4 commits into
mainfrom
argos-ci
Sep 23, 2026
Merged

nedtwigg merged 4 commits into
mainfrom
argos-ci

Conversation

@nedtwigg

Copy link
Copy Markdown
Member

Adds Argos next to Chromatic so we can compare the two on real PRs. Chromatic is unchanged.

What Argos covers (same as Chromatic)

  • Visual snapshots of every story: 275 stories, captured in Chromium and WebKit to match Chromatic's Chrome and Safari. Each browser uploads as its own Argos build, storybook-chromium and storybook-webkit. That's about 550 screenshots per run, and there's no TurboSnap equivalent, so every run captures everything.
  • Storybook publish: argos deploy. Pushes to main deploy with --prod.
  • Deterministic captures: the isChromatic() freeze block in lib/.storybook/preview.ts also runs under Argos, via the __ARGOS_SNAPSHOT__ define that only lib/vitest.argos.config.ts sets.

How it runs

Chromatic renders in its own cloud. Argos renders inside our CI job: the Storybook Vitest addon runs each story and its play in Playwright, then @argos-ci/storybook takes the screenshot. lib/vitest.argos.config.ts sets two things:

  • viewport 1200×900, Chromatic's capture width.
  • argos.fitToContent: false, so full-screen layouts aren't shrunk into a sliver.

A play exception now fails the job, whereas the Storybook 10 iframe used to swallow it. All 550 tests pass locally.

.github/workflows/argos.yml copies chromatic.yml's triggers, path globs, concurrency, draft gate and fork gate.

Secret and audit

  • ARGOS_TOKEN is a repo-level secret, set up like CHROMATIC_PROJECT_TOKEN.
  • It is added to .config/tend.yaml secrets.allowed, and Argos is added to ci-fix's watched workflows. tend-ci-fix.yaml is regenerated at 0.2.15 in its own commit.
  • docs/specs/security-ci.md changes in five places: the secrets table, "Reachable repo-level secrets", the inventory FAIL IF, the secrets.allowed FAIL IF, and the rationale. docs/specs/security.md also changes.
  • Word budgets are ratcheted.

Before this runs: gh secret set ARGOS_TOKEN --repo diffplug/dormouse.

Test plan

  • pnpm test:argos locally: 550/550 pass, and the screenshots were spot-checked (Wall, ThemePicker short viewport).
  • pnpm --filter dormouse-lib test, spec-lint, public-docs-lint
  • gh workflow run Argos --ref argos-ci goes green and produces two Argos builds plus a Storybook deploy.

🤖 Generated with Claude Code

nedtwigg and others added 2 commits September 23, 2026 09:36
Every Storybook story runs in Chromium and WebKit through the Storybook
Vitest addon and uploads to Argos as one build per browser; the built
Storybook is deployed to Argos too. Same triggers, draft gate, and fork gate
as Chromatic, which stays in place. The preview's snapshot freezes now also
apply under Argos, and ARGOS_TOKEN joins the audited repo-level secret
inventory.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 23, 2026

Copy link
Copy Markdown

Deploying mouseterm with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4c12c99
Status: ✅  Deploy successful!
Preview URL: https://cf330b47.mouseterm.pages.dev
Branch Preview URL: https://argos-ci.mouseterm.pages.dev

View logs

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback on work in progress, not a merge verdict — mark it ready when you want the full pass.

The job carries no timeout-minutes, and playwright install --with-deps shells out to an unbounded apt-get install. That is the mode ci.yml's "Install zsh" step already documents and defends against — "a degraded Azure mirror dribbles bytes rather than failing" — and here nothing bounds it at either level, so a dribbling mirror or a hung WebKit instance holds a runner for the 360-minute default. Every other job in this repo that does real in-job work carries a cap (ci.yml 13/16/20, hosted-* 5–25, security-audit 40). chromatic.yml has none, but its work happens in Chromatic's cloud; this job runs all 550 captures itself, which is exactly the difference the workflow header calls out. Suggestion inline.

Build Storybook and Deploy Storybook sit after the capture step, so the play-exception change cuts both ways: on main, one throwing play now also stops the standing --prod Storybook link from updating until the story is fixed. Under Chromatic the publish and the snapshot are one action and a throwing play never held the publish back, so this is new coupling rather than parity. Worth a deliberate call — if: ${{ !cancelled() }} on the two steps decouples them, at the price of deploying a Storybook with a story you know is broken.

The freezes' second trigger isn't reflected in the specs that describe them. docs/specs/layout.md attributes them to Chromatic alone in five places: the marching-ants bullet's "(Chromatic sets it in lib/.storybook/preview.ts)", the Snap gate's "!cfg.layout.animate (Chromatic)", the unfocus-saturate line's "Under Chromatic it snapshots already finished", the rename-warning's "0 under Chromatic, pinned in lib/.storybook/preview.ts", and the Renderer section's "is off under Chromatic". docs/specs/tiling-engine.md's ownership-split bullet says "which Chromatic's animate: false in lib/.storybook/preview.ts also sets", and docs/specs/alert.rationale.md says "It is the Chromatic freeze that pins the alarm". AGENTS.md: "When code covered by a spec changes, change the spec." Each is a pointer rather than a rule, so the fix is a word or two apiece, but left alone they tell the next editor that Chromatic is the only thing driving a guard that now has two arms — which is the reader most likely to delete the __ARGOS_SNAPSHOT__ clause as redundant.

lib/.storybook/vitest.setup.ts drops beforeAll(project.beforeAll) from the documented setup. Nothing breaks today, since preview.ts defines no beforeAll — but a later preview.beforeAll would run in Storybook and silently not run under Argos, and the symptom would be screenshots of a differently-initialized app rather than a red run. Suggestion inline.

Checked and clear
  • node scripts/spec-lint.mjs is clean at the ratcheted budgets (the two reported failures are the pre-existing standalone/sidecar/node_modules path errors from an uninstalled tree).
  • The secrets.allowed entry, the secret-inventory line, and the "Reachable repo-level secrets" edit satisfy the three FAIL IFs in docs/specs/security-ci.md that name CHROMATIC_PROJECT_TOKEN; argos.yml declares contents: read and SHA-pins all three actions, so the two "GitHub Actions Policies" FAIL IFs hold too.
  • No Storybook version bump rides along — main's lockfile is already on storybook@10.6.0, which is what @storybook/addon-vitest@10.6.0's storybook: ^10.6.0 peer wants.
  • Splitting the tend regen into its own commit keeps workflow-audit.yaml's regen classifier applicable to it — that arm requires the commit leave .config/tend.yaml alone, which 2221f3b does.

Comment thread .github/workflows/argos.yml
Comment thread .github/workflows/argos.yml
Comment thread lib/.storybook/vitest.setup.ts Outdated
@nedtwigg
nedtwigg marked this pull request as ready for review September 23, 2026 16:59
# Conflicts:
#	scripts/spec-word-budgets.json
@argos-ci

argos-ci Bot commented Sep 23, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Argos notifications ↗︎

Build Status Details Updated (UTC)
storybook-chromium (Inspect) 🔵 Orphan build 275 added Sep 23, 2026, 5:08 PM
storybook-webkit (Inspect) 🔵 Orphan build 275 added Sep 23, 2026, 5:08 PM
Deployment Status Branch Updated (UTC)
preview (Open) Ready argos-ci Sep 23, 2026, 5:08 PM

- Drop `.storybook/vitest.setup.ts`: since Storybook 10.3 the Vitest addon
  applies the preview's annotations, `beforeAll` included, on its own.
- Name the freeze gate `visualSnapshot` and point the specs at visual
  snapshots rather than at Chromatic alone.
- Workflow: a 30-minute cap, publish the Storybook even when a capture
  fails, drop the full-history checkout Argos does not need, and keep the
  Chromatic cross-reference in one header comment.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@nedtwigg
nedtwigg merged commit c57bb0c into main Sep 23, 2026
18 checks passed
@nedtwigg
nedtwigg deleted the argos-ci branch September 23, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants