Skip to content

test: refresh visual baselines after the View Source change (#453) - #457

Merged
mmcky merged 3 commits into
mainfrom
test/refresh-baselines-after-453
Oct 9, 2026
Merged

mmcky merged 3 commits into
mainfrom
test/refresh-baselines-after-453

Conversation

@mmcky

@mmcky mmcky commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Refreshes the Playwright baselines so they match what the fixtures site renders today. They were regenerated by /update-snapshots, the first run with #456's --update-snapshots=all fix, against fixtures 0048d10, and the bot's re-run confirmed they reproduce.

What changed (39 files)

I diffed every old and new baseline pixel by pixel and looked at crops of each kind of change. Nothing is unexplained:

Change Files Cause
Toolbar loses the GitHub icon; the other icons shift right by one slot toolbar.png and 27 full-page shots (both devices, dark-mode.png included) #453. The fixtures set no repository_url, so the View Source button is no longer rendered. Each file's change is the same: desktop x 959–1250 / y 15–34, mobile x 265–370 / y 15–34
Byline: "QuantEcon" becomes "Thomas J. Sargent and John Stachurski", and "Translated by Jane Doe and John Roe" appears intro.png, intro-sidebar.png (both devices); also the header of every full-page shot. On desktop math.png and admonitions.png it makes the page 3 px taller Stale since #431 re-pinned the fixtures to the byline demo. It was within tolerance, so the old changed-mode bot never captured it
Glyph-edge anti-aliasing only, with identical colours desktop bold-text-dark.png, italic-text-dark.png Chromium rendering drift since these were last generated in #390 (May)
PNG bytes only, with pixel-identical images mobile bold-text, italic-text, their dark variants, code-block Re-encoded by the current Playwright. Harmless, and they now match what future runs write

CI

On the empty commit pushed after the bot's commit, everything passes: the visual job had 103 passed, 3 skipped, and 0 flaky.

Unrelated flakiness still there

Math-heavy pages still sometimes render at a different height, and a size mismatch fails toHaveScreenshot whatever the tolerance. This existed before this PR and the refresh doesn't fix it:

  • main's CI for the ci: make /update-snapshots regenerate all baselines again #456 merge (f43b75d) failed visual on desktop math: baseline 3006 px, received 2939–2942 px on every retry.
  • This PR's first run (empty starter commit, same code as main) failed mobile long-page (5911 vs 5901–5902 px) and mobile "math equation rendering".
  • The bot's render of mobile long-page was 5911 px, matching the old baseline, so the height varies from run to run rather than having changed.

So one clean run here doesn't show the flakiness is gone. Tracked in #463, which traces it to MathJax sizing math before the web font has loaded.

No CHANGELOG entry, since a baseline refresh isn't user-facing.

🤖 Generated with Claude Code

Starter commit for the snapshot refresh; the regenerated baselines are
added by /update-snapshots once #456 is on main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

@github-actions
github-actions Bot temporarily deployed to pull request October 9, 2026 02:13 Inactive
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@11d4b12). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #457   +/-   ##
=======================================
  Coverage        ?   59.48%           
=======================================
  Files           ?        2           
  Lines           ?      548           
  Branches        ?        0           
=======================================
  Hits            ?      326           
  Misses          ?      222           
  Partials        ?        0           
Flag Coverage Δ
pytests 59.48% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🎭 Visual Regression Test Results

passed  103 passed
skipped  3 skipped

Details

stats  106 tests across 1 suite
duration  3 minutes, 52 seconds
commit  6094c26

Skipped tests

desktop-chrome › theme.spec.ts › Visual Regression Tests › prob-matrix - full page screenshot
mobile-chrome › theme.spec.ts › Visual Regression Tests › prob-matrix - full page screenshot
mobile-chrome › theme.spec.ts › Theme Features › f-string interpolation styling

@mmcky

mmcky commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

/update-snapshots

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

✅ Regenerated visual snapshots and committed 39 changed file(s) to this PR.

Built against fixtures QuantEcon/quantecon-book-theme-fixtures@0048d104ed1afa587b824d130e5210dc969713c5.

✅ Re-ran the suite against the new baselines — they reproduce.

📦 Download snapshot-update-diff artifact for the regenerated images; this PR's Files changed tab shows each one before and after.

⚠️ The commit was pushed with GITHUB_TOKEN, which does not trigger workflows — push an empty commit to re-run CI against the new baselines. (Set a SNAPSHOT_BOT_TOKEN secret to make this automatic.)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to pull request October 9, 2026 02:36 Inactive
@mmcky
mmcky marked this pull request as ready for review October 9, 2026 02:41
Copilot AI balanced review requested due to automatic review settings October 9, 2026 02:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

@mmcky
mmcky merged commit 554546d into main Oct 9, 2026
12 checks passed
@mmcky
mmcky deleted the test/refresh-baselines-after-453 branch October 9, 2026 04:06
@mmcky mmcky mentioned this pull request Oct 9, 2026

This branch was previously deployed

1 inactive deployment
pull request — 6094c261 Deployed Oct 9, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants