Chore/retire screenshot suite

.github/workflows/ci.yml

@@ -1,10 +1,17 @@
 name: CI
 
-# Push is the trigger that matters: this repo is reviewed off a branch rather
-# than through pull requests. The pull_request trigger costs one line and means
-# the checks are already wired if that ever changes.
+# Pull requests only. An unfiltered `push:` alongside this ran the whole job
+# twice for every commit on a PR branch -- once for the push, once for the
+# synchronize -- which was the bulk of the CI spend.
+#
+# The trade-off: nothing runs on the merge commit itself, so the tests never
+# see exactly what lands on master. Netlify covers the worst of that, since it
+# builds master on merge and a broken build shows up as a failed deploy -- but
+# smoke and e2e do not run there. If that gap ever bites, the fix is two lines:
+#
+#   push:
+#     branches: [master]
 on:
-  push:
   pull_request:
 
 concurrency:
@@ -67,23 +74,6 @@ jobs:
         env:
           BASE_URL: http://localhost:9100/tech-blog/
 
-      # Runs in the Playwright container, not on the runner. Screenshot
-      # comparison is only meaningful if the baselines were rasterised in the
-      # same environment: `monospace` is whatever font the OS supplies, and it
-      # resolves three different ways across a dev machine (DejaVu Sans Mono),
-      # this runner, and this image (WenQuanYi Zen Hei Mono). Body text hid the
-      # problem because Open Sans is a webfont; the code blocks did not.
-      # tests/README.md documents regenerating baselines the same way.
-      - name: Visual regression
-        run: |
-          docker run --rm --network host \
-            -v "$PWD/tests":/tests -w /tests \
-            --user "$(id -u):$(id -g)" -e HOME=/tmp \
-            -e BASE_URL=http://localhost:9100/tech-blog/ \
-            -e CI=true \
-            mcr.microsoft.com/playwright:v1.62.1-noble \
-            npx playwright test --project=desktop --project=tablet --project=mobile
-
       - name: Upload failure artifacts
         if: failure()
         uses: actions/upload-artifact@v4

package.json

@@ -21,7 +21,6 @@
     "test": "npm --prefix tests test",
     "test:smoke": "npm --prefix tests run test:smoke",
     "test:e2e": "npm --prefix tests run test:e2e",
-    "test:visual": "npm --prefix tests run test:visual",
     "favIcon16": "sharp -i ./src/images/app-icons/icon.png -o ./static/icons/favicon-16x16.png resize 16",
     "favIcon32": "sharp -i ./src/images/app-icons/icon.png -o ./static/icons/favicon-32x32.png resize 32",
     "favIcon96": "sharp -i ./src/images/app-icons/icon.png -o ./static/icons/favicon-96x96.png resize 96",

tests/BASELINE.md

@@ -9,7 +9,7 @@ This is the reference the Gatsby 5 migration is checked against.
 |---|---|
 | Live URL | https://www.boldare.com/tech-blog/ |
 | `sitemap.xml` URLs | **148** — 27 posts, 1 page, 118 tag pages, plus `/`, `/search/`, `/tags/` |
-| Visual baselines | 18 PNGs — 6 pages × 3 viewports (375 / 768 / 1440) |
+| Visual baselines | 18 PNGs — 6 pages × 3 viewports (375 / 768 / 1440). Retired after the upgrade verified 18/18; see tests/README.md |
 | e2e journeys | 4, all passing against production |
 
 ## CMS editorial workflow
@@ -49,3 +49,15 @@ steadier, not flakier.
   was under the old phase-1 plan, since the target is now an explicit `NODE_VERSION = "24"`
   rather than "record whatever works today".
 - **GTM container ID** for the analytics swap — needed before C2 can finish.
+
+## Outcome
+
+Verified against the final deploy preview on 2026-09-03:
+
+- 18/18 screenshots matched pre-upgrade production, across 6 pages and 3 viewports.
+- 9/9 journeys and widget metrics passed against the deployed site.
+- 27 of 28 content pages render structurally identical to production. The
+  exception is the table of contents on `Dev-and-prod-ready-Docker-setup-for-SPA-app`,
+  which renders tight where production renders loose: the source has no blank
+  lines between items, so CommonMark says tight and remark 9 had been lenient.
+  Left as-is deliberately.

tests/README.md

@@ -11,48 +11,31 @@ npm ci
 npx playwright install chromium
 ```
 
-## The three suites
+## The two suites
 
 | Suite | What it checks | Target |
 |---|---|---|
-| `npm run test:smoke` | Crawls built HTML: every page has title/description/canonical/h1, no broken internal links, no missing images, sitemap + rss present | `../public` on disk |
-| `npm run test:visual` | Screenshot diff at 375 / 768 / 1440px | `BASE_URL` |
-| `npm run test:e2e` | Reader journeys, Algolia search, `/admin/` boots | `BASE_URL` |
+| `npm run test:smoke` | Crawls built HTML: every page has title/description/canonical/h1, no broken internal links, no missing images, sitemap + rss present. Also checks the Algolia query and transformer, which no build without credentials ever exercises | `../public` on disk |
+| `npm run test:e2e` | Reader journeys, Algolia search, `/admin/` boots, and the widget metrics MUI 9 changed | `BASE_URL` |
 
-See `BASELINE.md` for what the pre-upgrade state actually was.
-
-## Visual baselines must be made in the container
+## There used to be a screenshot suite
 
-`monospace` is whatever font the operating system supplies, and it resolves three
-different ways across a dev machine (DejaVu Sans Mono), a GitHub runner, and the
-Playwright image (WenQuanYi Zen Hei Mono). Body text hides this, because Open Sans is a
-webfont and is downloaded identically everywhere — code blocks and inline `<code>` do
-not, and different glyph widths reflow the line around them. Baselines rasterised on a
-host therefore fail in CI for reasons that have nothing to do with the site.
+Six pages x three viewports, diffed against baselines captured from production
+before the Gatsby 5 upgrade. It answered one question — "does the upgraded site
+still look like the site as it was?" — confirmed 18/18 against the final deploy
+preview, and was removed once that question was settled. Keeping it would have
+meant comparing forever against a site that no longer exists.
 
-So capture and comparison both run in `mcr.microsoft.com/playwright`, which is what the
-CI job uses too. **Never run `--update-snapshots` outside it** — `npm run
-test:visual:update` on a host will produce baselines that only work on that host.
+Its durable half lives on in `e2e/widget-metrics.spec.js`. Screenshot diffing
+turned out to be poor at exactly the things that broke here: small chrome is a
+rounding error on a full-page image, and `toHaveScreenshot`'s own per-pixel
+`threshold` (0.2 by default) makes a whole-page shade change invisible. Asserting
+computed values instead is deterministic, needs no container, and caught what the
+pixels missed.
 
-```bash
-# regenerate baselines FROM PRODUCTION. The point of the suite is comparing against the
-# pre-upgrade site, so this target stays production even after the upgrade ships.
-docker run --rm --network host -v "$PWD":/tests -w /tests \
-  --user "$(id -u):$(id -g)" -e HOME=/tmp \
-  -e BASE_URL=https://www.boldare.com/tech-blog/ \
-  mcr.microsoft.com/playwright:v1.62.1-noble \
-  npx playwright test --project=desktop --project=tablet --project=mobile --update-snapshots
+See `BASELINE.md` for what the pre-upgrade state actually was.
 
-# compare a local build against them
-cd .. && npm run build && npx gatsby serve --prefix-paths -p 9100 &
-cd tests && docker run --rm --network host -v "$PWD":/tests -w /tests \
-  --user "$(id -u):$(id -g)" -e HOME=/tmp \
-  -e BASE_URL=http://localhost:9100/tech-blog/ \
-  mcr.microsoft.com/playwright:v1.62.1-noble \
-  npx playwright test --project=desktop --project=tablet --project=mobile
-```
 
-`smoke` and `e2e` have no such constraint and run fine on the host.
 
 `smoke` needs no server — it reads `public/` straight off disk, so it works the moment
 `gatsby build` finishes.

tests/e2e/widget-metrics.spec.js

@@ -5,11 +5,16 @@ const { test, expect } = require("@playwright/test");
 // still came in under the pixel-ratio threshold, because four small buttons are
 // a rounding error on a full-page image. These assert the numbers directly.
 //
-// Every expected value here is what production (MUI v3) renders, measured
-// against https://www.boldare.com/tech-blog/. MUI 9 changed each of them:
-// IconButton padding 12 -> 8, SvgIcon's fixed 24px -> a pxToRem value inflated
-// by this theme's typography.fontSize, and Chip's solid fill -> a translucent
-// overlay. They are pinned back in src/styles/theme.js.
+// Every expected value is the design as it stood before the Gatsby 5 upgrade,
+// measured against production while it still ran Gatsby 2. (Once this ships,
+// production becomes this code, so "matches production" stops being a
+// meaningful check -- these numbers are the record of what it should be.)
+//
+// MUI 9 changed each of them: IconButton padding 12 -> 8, SvgIcon's fixed 24px
+// -> a pxToRem value inflated by this theme's typography.fontSize, and Chip's
+// solid fill -> a translucent overlay. They are pinned in src/styles/theme.js.
+//
+// This file is also what replaced the screenshot suite; see tests/README.md.
 
 const POST = "weekly-ai-bites-last-manual-gap-qa-testing/";
 

tests/package.json

@@ -6,10 +6,7 @@
   "scripts": {
     "test": "playwright test",
     "test:smoke": "playwright test --project=smoke",
-    "test:e2e": "playwright test --project=e2e",
-    "test:visual": "playwright test --project=desktop --project=tablet --project=mobile",
-    "test:visual:update": "docker run --rm --network host -v \"$PWD\":/tests -w /tests --user \"$(id -u):$(id -g)\" -e HOME=/tmp -e BASE_URL=${BASE_URL:-https://www.boldare.com/tech-blog/} mcr.microsoft.com/playwright:v1.62.1-noble npx playwright test --project=desktop --project=tablet --project=mobile --update-snapshots",
-    "test:visual:docker": "docker run --rm --network host -v \"$PWD\":/tests -w /tests --user \"$(id -u):$(id -g)\" -e HOME=/tmp -e BASE_URL=${BASE_URL:-https://www.boldare.com/tech-blog/} mcr.microsoft.com/playwright:v1.62.1-noble npx playwright test --project=desktop --project=tablet --project=mobile"
+    "test:e2e": "playwright test --project=e2e"
   },
   "devDependencies": {
     "@playwright/test": "1.62.1"

tests/playwright.config.js

@@ -9,8 +9,6 @@ const chrome = devices["Desktop Chrome"];
 
 module.exports = defineConfig({
   testDir: ".",
-  // Visual diffs against a live site carry some irreducible noise (lazy-loaded
-  // covers, webfont timing, the Facebook SDK). One retry locally, two in CI.
   retries: process.env.CI ? 2 : 1,
   workers: process.env.CI ? 2 : undefined,
   reporter: process.env.CI ? [["github"], ["list"]] : [["list"]],
@@ -19,25 +17,11 @@ module.exports = defineConfig({
     trace: "retain-on-failure",
     screenshot: "only-on-failure",
   },
-  expect: {
-    toHaveScreenshot: {
-      // Anti-aliasing and subpixel text rendering differ between machines, so
-      // some tolerance is needed. 0.02 was too loose: it silently passed a real
-      // regression where every post thumbnail rendered 90x51 instead of 90x90,
-      // because the images are a small share of a full-page screenshot.
-      maxDiffPixelRatio: 0.004,
-      animations: "disabled",
-    },
-  },
   projects: [
     // Reads public/ off disk — a browser is used only to parse the HTML, so one
     // viewport is enough. Running this per-viewport would just triple the work.
     { name: "smoke", testMatch: /smoke\//, use: { ...chrome } },
 
     { name: "e2e", testMatch: /e2e\//, use: { ...chrome, viewport: { width: 1440, height: 900 } } },
-
-    { name: "desktop", testMatch: /visual\//, use: { ...chrome, viewport: { width: 1440, height: 900 } } },
-    { name: "tablet", testMatch: /visual\//, use: { ...chrome, viewport: { width: 768, height: 1024 } } },
-    { name: "mobile", testMatch: /visual\//, use: { ...chrome, viewport: { width: 375, height: 812 } } },
   ],
 });