Skip to content

Commit 435cf72

Browse files
T4rk1nclaude
andcommitted
Replace Percy with nitpix for visual testing
Visual snapshots are now captured locally by dash.testing (real browser screenshots into NITPIX_SNAPSHOT_DIR, one PNG per width) and diffed in CI by plotly/nitpix: baselines live on the nitpix-baselines orphan branch, PRs get a review comment with before/after/diff images and a nitpix/visual commit status, and changes are approved by commenting "/nitpix approve" — no test-matrix re-run needed. - dash/testing/browser.py: percy_snapshot() keeps its name and signature (public API) but saves PNGs locally instead of calling the Percy SDK. convert_canvases is now a no-op: real screenshots capture canvases. - testing.yml: drop PERCY_* env, percy-exec wrapper and percy-finalize; snapshot-producing jobs (test-main, html-test, dcc-test, 3.12 leg only) upload nitpix-snapshots-* artifacts instead. - nitpix.yml (new): workflow_run diff/report job in base-repo context, so fork and Dependabot PRs are fully supported — the old percy/dash placeholder-status hack in post-test-status.yml is removed. - nitpix-approve.yml (new): handles /nitpix approve comments. - table-visual-test still uses percy-storybook (separate token/mechanism); converting it is a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 4e0fb90 commit 435cf72

7 files changed

Lines changed: 161 additions & 239 deletions

File tree

.github/actions/percy-exec/action.yml

Lines changed: 0 additions & 50 deletions
This file was deleted.
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: nitpix approve
2+
3+
# Handles `/nitpix approve` comments on PRs: records the approved snapshot
4+
# hashes on the nitpix-baselines branch and flips the `nitpix/visual` commit
5+
# status to success — without re-running the test matrix. Only users with
6+
# write access can approve.
7+
8+
on:
9+
issue_comment:
10+
types: [created]
11+
12+
permissions:
13+
contents: write
14+
pull-requests: write
15+
statuses: write
16+
17+
jobs:
18+
approve:
19+
name: Approve visual changes
20+
runs-on: ubuntu-latest
21+
if: github.event.issue.pull_request && startsWith(github.event.comment.body, '/nitpix')
22+
steps:
23+
- uses: plotly/nitpix/approve@main # TODO: pin to @v1 once released
24+
with:
25+
baseline-branch: nitpix-baselines

.github/workflows/nitpix.yml

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
name: nitpix
2+
3+
# Visual diffing (replaces Percy). Runs in the base-repo context via
4+
# workflow_run so it has a full-privilege token even for fork and Dependabot
5+
# PRs (which get a read-only token in "Dash Testing" itself).
6+
#
7+
# - PR-triggered test runs: diff the uploaded nitpix-snapshots-* artifacts
8+
# against baselines/<base-branch>/ on the nitpix-baselines orphan branch,
9+
# post/update the review comment and the `nitpix/visual` commit status.
10+
# Approve visual changes by commenting `/nitpix approve` on the PR
11+
# (see nitpix-approve.yml).
12+
# - push-triggered test runs (dev/master): promote that run's snapshots to
13+
# the branch's baselines.
14+
#
15+
# SECURITY: never check out or execute PR code in this workflow — it runs
16+
# with write permissions for all PRs. nitpix only handles PNG artifacts.
17+
18+
on:
19+
workflow_run:
20+
workflows: ["Dash Testing"]
21+
types: [completed]
22+
23+
permissions:
24+
contents: write # push to the nitpix-baselines branch
25+
pull-requests: write # the review comment
26+
statuses: write # the nitpix/visual check
27+
28+
concurrency:
29+
group: nitpix-${{ github.event.workflow_run.head_sha }}
30+
cancel-in-progress: true
31+
32+
jobs:
33+
visual-review:
34+
name: Visual Review
35+
runs-on: ubuntu-latest
36+
if: |
37+
github.event.workflow_run.conclusion == 'success' &&
38+
(github.event.workflow_run.event == 'pull_request' ||
39+
github.event.workflow_run.event == 'push')
40+
steps:
41+
- uses: plotly/nitpix@main # TODO: pin to @v1 once released
42+
with:
43+
baseline-branch: nitpix-baselines

.github/workflows/post-test-status.yml

Lines changed: 3 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -65,43 +65,9 @@ jobs:
6565
}
6666
}
6767
68-
- name: Post Percy status for fork/Dependabot PRs
69-
# Percy needs PERCY_TOKEN to run, which is unavailable to fork PRs (and to
70-
# Dependabot unless added to Dependabot secrets). Without a build, the
71-
# Percy GitHub App never posts the required `percy/dash` status and the PR
72-
# is blocked. Post a success status here so the check resolves — but only
73-
# if Percy hasn't already posted one, so a real Percy result is never
74-
# clobbered (e.g. when Dependabot does have the token).
75-
if: >
76-
github.event.workflow_run.head_repository.full_name != github.repository ||
77-
github.event.workflow_run.actor.login == 'dependabot[bot]'
78-
uses: actions/github-script@v7
79-
with:
80-
script: |
81-
const { owner, repo } = context.repo;
82-
const sha = context.payload.workflow_run.head_sha;
83-
const statusContext = 'percy/dash';
84-
85-
const { data: { statuses } } = await github.rest.repos.getCombinedStatus({
86-
owner,
87-
repo,
88-
ref: sha,
89-
});
90-
91-
if (statuses.some(s => s.context === statusContext)) {
92-
console.log(`'${statusContext}' already posted — leaving it untouched.`);
93-
return;
94-
}
95-
96-
await github.rest.repos.createCommitStatus({
97-
owner,
98-
repo,
99-
sha,
100-
state: 'success',
101-
context: statusContext,
102-
description: 'Skipped — Percy unavailable for fork/Dependabot PRs',
103-
});
104-
console.log(`Posted skipped status for ${statusContext}`);
68+
# NOTE: the "Post Percy status for fork/Dependabot PRs" workaround that
69+
# lived here was removed with the move to nitpix (nitpix.yml), which
70+
# handles fork/Dependabot PRs natively via workflow_run.
10571

10672
test-report:
10773
name: Consolidated Test Report (Fork PR)

.github/workflows/testing.yml

Lines changed: 42 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -327,7 +327,7 @@ jobs:
327327
timeout-minutes: 15
328328
run: |
329329
cd bgtests
330-
pytest --headless --nopercyfinalize --junitxml=test-reports/junit_background.xml tests/background_callback -v -s
330+
pytest --headless --junitxml=test-reports/junit_background.xml tests/background_callback -v -s
331331
332332
- name: Cleanup background processes
333333
if: always()
@@ -340,7 +340,7 @@ jobs:
340340
timeout-minutes: 15
341341
run: |
342342
cd bgtests
343-
pytest --headless --nopercyfinalize --junitxml=test-reports/junit_async.xml tests/async_tests -v -s
343+
pytest --headless --junitxml=test-reports/junit_async.xml tests/async_tests -v -s
344344
345345
- name: Upload test results
346346
if: always()
@@ -422,7 +422,7 @@ jobs:
422422
cp -r tests bgtests/tests
423423
cd bgtests
424424
touch __init__.py
425-
pytest --headless --nopercyfinalize tests/backend_tests -v -s
425+
pytest --headless tests/backend_tests -v -s
426426
427427
table-unit:
428428
name: Table Unit/Lint Tests (Python ${{ matrix.python-version }})
@@ -536,7 +536,7 @@ jobs:
536536
- name: Run Table Server Tests
537537
run: |
538538
cd components/dash-table
539-
pytest --nopercyfinalize --headless --junitxml=test-reports/junit_table.xml --splits 3 --group ${{ matrix.test-group }}
539+
pytest --headless --junitxml=test-reports/junit_table.xml --splits 3 --group ${{ matrix.test-group }}
540540
541541
- name: Upload test results
542542
if: always()
@@ -605,7 +605,7 @@ jobs:
605605
cp -r tests wstests/tests
606606
cd wstests
607607
touch __init__.py
608-
pytest --headless --nopercyfinalize tests/websocket -v -s
608+
pytest --headless tests/websocket -v -s
609609
610610
test-main:
611611
name: Main Dash Tests (Python ${{ matrix.python-version }}, Group ${{ matrix.test-group }})
@@ -619,17 +619,10 @@ jobs:
619619
test-group: ["1", "2", "3"]
620620

621621
env:
622-
PERCY_TOKEN: ${{ matrix.python-version == '3.12' && secrets.PERCY_TOKEN || '' }}
623-
PERCY_ENABLE: ${{ matrix.python-version == '3.12' && '1' || '0' }}
624-
PERCY_PARALLEL_TOTAL: -1
625-
# Pin the build identity so every shard joins the same parallel build and
626-
# Percy links it to the PR. Auto-detection otherwise uses the ephemeral
627-
# merge SHA (refs/pull/N/merge), so the build never shows up on the PR.
628-
PERCY_PARALLEL_NONCE: ${{ github.run_id }}-${{ github.run_attempt }}
629-
PERCY_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
630-
PERCY_BRANCH: ${{ github.head_ref || github.ref_name }}
631-
PERCY_PULL_REQUEST: ${{ github.event.pull_request.number }}
632-
PERCY_TARGET_BRANCH: ${{ github.base_ref }}
622+
# Visual snapshots (nitpix) are captured on a single matrix leg so
623+
# parallel legs don't produce duplicate snapshot names. When unset,
624+
# percy_snapshot() is a no-op.
625+
NITPIX_SNAPSHOT_DIR: ${{ matrix.python-version == '3.12' && '/tmp/nitpix-snapshots' || '' }}
633626

634627
steps:
635628
- name: Checkout repository
@@ -680,16 +673,14 @@ jobs:
680673
run: npm run setup-tests.py
681674

682675
- name: Run main integration tests
683-
if: matrix.python-version != '3.12'
684-
run: pytest --headless --nopercyfinalize --junitxml=test-reports/junit_intg.xml tests/integration --splits 3 --group ${{ matrix.test-group }}
676+
run: pytest --headless --junitxml=test-reports/junit_intg.xml tests/integration --splits 3 --group ${{ matrix.test-group }}
685677

686-
- name: Run main integration tests with Percy
687-
if: matrix.python-version == '3.12'
688-
uses: ./.github/actions/percy-exec
678+
- name: Upload visual snapshots
679+
if: always() && matrix.python-version == '3.12'
680+
uses: plotly/nitpix/upload@main # TODO: pin to @v1 once released
689681
with:
690-
command: pytest --headless --nopercyfinalize --junitxml=test-reports/junit_intg.xml tests/integration --splits 3 --group ${{ matrix.test-group }}
691-
percy-token: ${{ secrets.PERCY_TOKEN }}
692-
browser-executable: ${{ steps.setup-chrome.outputs.chrome-path }}
682+
name: main-${{ matrix.test-group }}
683+
path: /tmp/nitpix-snapshots
693684

694685
- name: Upload test results
695686
if: always()
@@ -722,17 +713,9 @@ jobs:
722713
python-version: ["3.9", "3.12"]
723714

724715
env:
725-
PERCY_TOKEN: ${{ matrix.python-version == '3.12' && secrets.PERCY_TOKEN || '' }}
726-
PERCY_ENABLE: ${{ matrix.python-version == '3.12' && '1' || '0' }}
727-
PERCY_PARALLEL_TOTAL: -1
728-
# Pin the build identity so every shard joins the same parallel build and
729-
# Percy links it to the PR. Auto-detection otherwise uses the ephemeral
730-
# merge SHA (refs/pull/N/merge), so the build never shows up on the PR.
731-
PERCY_PARALLEL_NONCE: ${{ github.run_id }}-${{ github.run_attempt }}
732-
PERCY_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
733-
PERCY_BRANCH: ${{ github.head_ref || github.ref_name }}
734-
PERCY_PULL_REQUEST: ${{ github.event.pull_request.number }}
735-
PERCY_TARGET_BRANCH: ${{ github.base_ref }}
716+
# Visual snapshots (nitpix) are captured on a single matrix leg so
717+
# parallel legs don't produce duplicate snapshot names.
718+
NITPIX_SNAPSHOT_DIR: ${{ matrix.python-version == '3.12' && '/tmp/nitpix-snapshots' || '' }}
736719

737720
steps:
738721
- name: Checkout repository
@@ -784,18 +767,15 @@ jobs:
784767
run: npm ci
785768

786769
- name: Run HTML components tests
787-
if: matrix.python-version != '3.12'
788770
working-directory: components/dash-html-components
789-
run: pytest --headless --nopercyfinalize --junitxml=test-reports/junit_html.xml
771+
run: pytest --headless --junitxml=test-reports/junit_html.xml
790772

791-
- name: Run HTML components tests with Percy
792-
if: matrix.python-version == '3.12'
793-
uses: ./.github/actions/percy-exec
773+
- name: Upload visual snapshots
774+
if: always() && matrix.python-version == '3.12'
775+
uses: plotly/nitpix/upload@main # TODO: pin to @v1 once released
794776
with:
795-
command: pytest --headless --nopercyfinalize --junitxml=test-reports/junit_html.xml
796-
percy-token: ${{ secrets.PERCY_TOKEN }}
797-
browser-executable: ${{ steps.setup-chrome.outputs.chrome-path }}
798-
working-directory: components/dash-html-components
777+
name: html
778+
path: /tmp/nitpix-snapshots
799779

800780
- name: Upload test results
801781
if: always()
@@ -918,17 +898,9 @@ jobs:
918898
test-group: ["1", "2", "3"]
919899

920900
env:
921-
PERCY_TOKEN: ${{ matrix.python-version == '3.12' && secrets.PERCY_TOKEN || '' }}
922-
PERCY_ENABLE: ${{ matrix.python-version == '3.12' && '1' || '0' }}
923-
PERCY_PARALLEL_TOTAL: -1
924-
# Pin the build identity so every shard joins the same parallel build and
925-
# Percy links it to the PR. Auto-detection otherwise uses the ephemeral
926-
# merge SHA (refs/pull/N/merge), so the build never shows up on the PR.
927-
PERCY_PARALLEL_NONCE: ${{ github.run_id }}-${{ github.run_attempt }}
928-
PERCY_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
929-
PERCY_BRANCH: ${{ github.head_ref || github.ref_name }}
930-
PERCY_PULL_REQUEST: ${{ github.event.pull_request.number }}
931-
PERCY_TARGET_BRANCH: ${{ github.base_ref }}
901+
# Visual snapshots (nitpix) are captured on a single matrix leg so
902+
# parallel legs don't produce duplicate snapshot names.
903+
NITPIX_SNAPSHOT_DIR: ${{ matrix.python-version == '3.12' && '/tmp/nitpix-snapshots' || '' }}
932904

933905
steps:
934906
- name: Checkout repository
@@ -980,7 +952,14 @@ jobs:
980952
cd components/dash-core-components
981953
npm ci
982954
npm run test:jest
983-
pytest --headless --nopercyfinalize --junitxml=test-reports/junit_intg.xml --junitprefix="components.dash-core-components" tests/integration --splits 3 --group ${{ matrix.test-group }}
955+
pytest --headless --junitxml=test-reports/junit_intg.xml --junitprefix="components.dash-core-components" tests/integration --splits 3 --group ${{ matrix.test-group }}
956+
957+
- name: Upload visual snapshots
958+
if: always() && matrix.python-version == '3.12'
959+
uses: plotly/nitpix/upload@main # TODO: pin to @v1 once released
960+
with:
961+
name: dcc-${{ matrix.test-group }}
962+
path: /tmp/nitpix-snapshots
984963

985964
- name: Upload test results
986965
if: always()
@@ -1084,35 +1063,12 @@ jobs:
10841063
retention-days: 7
10851064
if-no-files-found: ignore
10861065

1087-
percy-finalize:
1088-
name: Finalize Percy Snapshots
1089-
needs: [test-main, dcc-test, html-test]
1090-
runs-on: ubuntu-latest
1091-
if: always()
1092-
env:
1093-
PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}
1094-
# Must match the snapshot jobs so finalize targets the same parallel build
1095-
# and the result is attached to the PR head commit, not the merge SHA.
1096-
PERCY_PARALLEL_NONCE: ${{ github.run_id }}-${{ github.run_attempt }}
1097-
PERCY_COMMIT: ${{ github.event.pull_request.head.sha || github.sha }}
1098-
PERCY_BRANCH: ${{ github.head_ref || github.ref_name }}
1099-
PERCY_PULL_REQUEST: ${{ github.event.pull_request.number }}
1100-
PERCY_TARGET_BRANCH: ${{ github.base_ref }}
1101-
steps:
1102-
- name: Finalize Main Percy Build
1103-
if: |
1104-
env.PERCY_TOKEN != '' && (
1105-
needs.test-main.result != 'skipped' ||
1106-
needs.dcc-test.result != 'skipped' ||
1107-
needs.html-test.result != 'skipped'
1108-
)
1109-
run: |
1110-
npm install -g @percy/cli
1111-
npx percy build:finalize
1112-
1113-
- name: Skip Percy finalize (fork PR)
1114-
if: env.PERCY_TOKEN == ''
1115-
run: echo "::notice::Skipping Percy finalize (no token available - likely a fork PR)"
1066+
# Visual diffing (nitpix) runs in a separate workflow (nitpix.yml) triggered
1067+
# by workflow_run: it downloads the nitpix-snapshots-* artifacts uploaded by
1068+
# the jobs above, diffs them against the baselines on the nitpix-baselines
1069+
# branch, and posts the PR comment + nitpix/visual commit status. Running it
1070+
# over there (base-repo context) is what makes it work for fork PRs and
1071+
# Dependabot, which get a read-only token in this workflow.
11161072

11171073
test-report:
11181074
name: Consolidated Test Report
@@ -1181,7 +1137,7 @@ jobs:
11811137
11821138
artifacts:
11831139
name: Store Build Artifacts
1184-
needs: [build, percy-finalize]
1140+
needs: [build, test-main]
11851141
runs-on: ubuntu-latest
11861142
if: |
11871143
github.event_name == 'push' &&

0 commit comments

Comments
 (0)