Fix flaky communities visual test (was failing every PR) - #13
Merged
Conversation
The `communities search filters cards` test was the only visual check
failing on every PR. Two root causes:
1. It never stubbed Supabase (unlike every other spec, which goes through
auditPage → stubSupabase), so it rendered LIVE prod data. As live
communities changed, the full-page screenshot drifted and the baseline
went stale — the failure reproduces on unchanged main.
2. It waited a fixed 500 ms after typing, racing the 200 ms-debounced
re-render.
Fix: stub Supabase with the fixed fixture (2 Bengaluru + 1 Mumbai
community) before navigating, and replace the sleep with
`expect('.community-card').toHaveCount(2)`, which auto-retries until the
filtered re-render settles. Also hide dynamic regions before the shot,
matching auditPage.
Baseline regenerated in the pinned Playwright image (fixture render).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the previous commit. The fuzzy search box (Fuse) returns all 3 fixture communities for the query "bengaluru", so a search-box count assertion can't be both meaningful and stable. Switch the test to the city dropdown, which filters by exact match (c.city === selected), giving a deterministic narrow from 3 → 2 cards. - Assert toHaveCount(3) then toHaveCount(2) around selectOption — a real wait-for-filtered-count with no fixed sleep. - Rename screenshot to communities-filter-bengaluru.png; remove the old communities-search-bengaluru-*.png baselines. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Linux baselines for the deterministic city-filter screenshot, generated in the pinned Playwright Docker image via the Visual Regression workflow (update_snapshots). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
communities search filters cardswas the only visual check failing on every PR (and it fails on unchangedmaintoo — I reproduced it by re-runningmain's last green run in today's environment).Two root causes:
auditPage → stubSupabaseand renders a fixed fixture. This one navigated directly, so it rendered live prod data — as communities changed in production, the full-page screenshot drifted and the baseline went stale.While fixing it I also found the fuzzy search box (Fuse) returns all 3 fixture communities for the query "bengaluru", so a search-box count assertion can't be both meaningful and stable.
Fix
toHaveCount(3)→selectOption('#cityFilter', 'Bengaluru')→toHaveCount(2). The city dropdown filters by exact match (c.city === selected), so it's fully deterministic (no fuzzy nondeterminism).auditPage).communities-filter-bengaluru.png; removed the oldcommunities-search-bengaluru-*.pngbaselines and added the regenerated Linux baselines (pinned Playwright image, per CONTRIBUTING §6).Verification
The branch's
update_snapshotsrun passed the3 → 2assertions cleanly. The PR's normal CI run should now be fully green — restoring the visual check as a trustworthy gate instead of a permanent red.🤖 Generated with Claude Code