Skip to content

Add cypress-audit Lighthouse checks for dashboard and form preview - #3359

Open
vivi-the-going-merry[bot] wants to merge 2 commits into
masterfrom
fix/issue-6683-cypress-audit
Open

vivi-the-going-merry[bot] wants to merge 2 commits into
masterfrom
fix/issue-6683-cypress-audit

Conversation

@vivi-the-going-merry

@vivi-the-going-merry vivi-the-going-merry Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Adds cypress-audit to the Cypress suite, alongside the existing cypress-axe/cypress-html-validate checks.

  • Registers lighthouse/prepareAudit in cypress.config.js and cypress-audit/commands in the support file.
  • Adds one spec per page for the first pass, matching the existing admin-a11y.cy.js / form-preview-a11y.cy.js split: the dashboard (admin-lighthouse-audit.cy.js) and the front-end form preview (form-preview-lighthouse-audit.cy.js).
  • Thresholds start permissive (0 for every category) on purpose — this run is the baseline measurement. The lighthouse task logs each page's raw category scores to the CI log; thresholds will be tightened to that measured baseline in a follow-up commit before this is ready for review.

package-lock.json's diff is npm-generated fallout from adding lighthouse (a large transitive tree) — not hand-edited.

Verified: npm ci --legacy-peer-deps --include=dev installs cleanly from the updated lockfile; eslint passes on all changed files. Behavioral verification (real Lighthouse scores) comes from this PR's own CI run with run e2e tests — no local browser/wp-env available in this environment.

Fixes Strategy11/formidable-pro#6683

Registers cypress-audit's lighthouse task/prepareAudit alongside the
existing axe/html-validate plugins and adds one spec per page
(admin dashboard, front-end form preview), matching the existing
admin-a11y.cy.js / form-preview-a11y.cy.js split. Thresholds start
permissive - the lighthouse task logs raw category scores to the CI
log so real thresholds can be set from a measured baseline.
@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: e756c40d-4183-4dbd-875f-380a6e549d98

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@vivi-the-going-merry vivi-the-going-merry Bot added run analysis run e2e tests Run the Cypress end-to-end suite on this PR labels Sep 17, 2026
@deepsource-io

deepsource-io Bot commented Sep 17, 2026

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in 1b64c62...4ffffed on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade   Security  

Reliability  

Complexity  

Hygiene  

Code Review Summary

Analyzer Status Updated (UTC) Details
PHP Sep 17, 2026 5:51p.m. Review ↗
JavaScript Sep 17, 2026 5:51p.m. Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

form-preview-lighthouse-audit.cy.js hit the same admin-ajax preview
URL as form-preview-a11y.cy.js but without cy.login() first, so
WordPress returned 403 and cy.visit() failed before the audit ran.

Separately, cy.lighthouse() silently skips ("Electron is not
supported") under Cypress's default Electron browser - Lighthouse
needs Chrome DevTools Protocol access, which Electron doesn't expose.
That's why the dashboard spec "passed" in 4s with no score output:
the audit never actually ran. Run the suite under --browser chrome
(present on the ubuntu-latest runner) instead.
@vivi-the-going-merry

Copy link
Copy Markdown
Contributor Author

Root-caused this run's 4/4 Cypress-shard failures — not a flake, and not caused by the browser switch itself acting alone.

Control check: the commit before --browser chrome landed (bd0be27) only failed 1 of 4 shards, and that failure was the new form-preview-lighthouse-audit.cy.js spec itself (the 403 this next commit set out to fix). After switching e2e:githubrun to --browser chrome, all 4 shards failed, each on a different, pre-existing, unrelated spec (GlobalSettings/globalSettingsUpdates.cy.js timing out on a toast close button, a cy.visit() load failure, a "2 elements matched" click, a "content not found" assertion) — none of them touch lighthouse or this PR's diff.

Mechanism: e2e:githubrun is the one script the whole sharded suite runs, so switching it to --browser chrome moved every spec in every shard off Electron, not just the two new lighthouse specs. Real Chrome under this CI runner is evidently unstable enough for the existing wp-admin timing-sensitive specs to start failing broadly. Separately, before:browser:launch in cypress.config.js calls prepareAudit(launchOptions) unconditionally — so every spec's browser launch now carries Lighthouse's CDP flags too, regardless of browser.

Fix shape (not yet implemented — ran out of runway this tick): don't change the browser or the launch hook for the whole suite. Scope Chrome + prepareAudit to just the two lighthouse specs — e.g. pull admin-lighthouse-audit.cy.js/form-preview-lighthouse-audit.cy.js out of split-specs.sh's sharded set and run them via their own npm script/CI step with --browser chrome, leaving e2e:githubrun (and the other 4 shards) on Electron.

Stylelint red on this PR is unrelated and pre-existing — css/formidableforms.css isn't generated on master either (that job shows skipped there); it's the same gap tracked in #6684, not something this diff caused.

Picking this back up next run to implement the split.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run analysis run e2e tests Run the Cypress end-to-end suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants