Skip to content

ci(workbench): reland parallel smoke with one user per worker - #842

Merged
ian-flores merged 4 commits into
mainfrom
wb-smoke-parallel-644-reland
Nov 19, 2026
Merged

ian-flores merged 4 commits into
mainfrom
wb-smoke-parallel-644-reland

Conversation

@ian-flores

@ian-flores ian-flores commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Relands #840 (reverted in #841 after an early merge), plus the fix for its last open review thread. Closes #644.

Smoke job (2026.11.0, PR tier) main this branch
Workbench 393 s 254 s
Connect 138 s 221 s
Package Manager 76 s 71 s

Reapplies #840 (reverted in #841 after an early merge).
xdist dispatches whole groups, queued by each group's first test, so sign-out sharing workbench_auth with the early login scenario ran at the start of a shared-auth run and ended the session other workers were using. Give it a group of its own so its order(95) queues it last.
Copilot AI balanced review requested due to automatic review settings November 18, 2026 23:53
@ian-flores
ian-flores marked this pull request as ready for review November 18, 2026 23:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The sign-out group can run before other workers finish, invalidating their shared session mid-test; it needs a separate serial phase and a multi-worker execution test.

1 open issue
What changed in this PR

Restores parallel Workbench smoke testing in VIP, relanding #840 after its revert and adding a dedicated sign-out group.

Changes:

  • Creates one Workbench account per worker, isolates Git checkout directories, and increases the login retry delay.
  • Ensures xdist sees VIP’s groups and assigns sign-out its own group.
  • Pins pytest for xdist compatibility and adds regression coverage for worker isolation and grouping.
File Description
uv.lock Locks pytest to 9.1.1.
src/​vip/​plugin/​selection.py Assigns groups before xdist reads them; leaves non-product tests ungrouped.
src/​vip_tests/​workbench/​test_git_ops.py Uses worker-specific Git checkout directories.
src/​vip_tests/​workbench/​naming.py Adds opt-in per-worker usernames.
src/​vip_tests/​workbench/​login.py Extends login retries beyond the default throttle window.
src/​vip_tests/​workbench/​conftest.py Enables per-worker grouping and usernames; separates sign-out’s group.
src/​vip_tests/​workbench/​cleanup.py Targets the worker’s account during token-based cleanup.
selftests/​test_workbench_parallel.py Covers usernames, retry timing, and applied xdist groups.
selftests/​test_workbench_ordering.py Checks that sign-out is collected in its own final group.
selftests/​test_workbench_git_ops.py Tests worker-specific checkout paths and directory creation.
selftests/​test_workbench_cleanup.py Checks cleanup uses the worker-specific account.
pyproject.toml Pins pytest to the xdist-compatible version.
AGENTS.md Documents per-worker accounts and grouping requirements.
.github/​workflows/​workbench-smoke.yml Provisions worker accounts and enables parallel smoke execution.
.github/​dependabot.yml Holds pytest updates below 9.2 pending xdist compatibility.

Review effort: Balanced

Comment thread src/vip_tests/workbench/conftest.py Resolved
Workbench sign-out revokes every cookie the account holds, and xdist runs the
last group while other workers finish theirs. Each worker now drops a marker at
session end; under shared auth the sign-out step waits for them (unproven on
timeout). A real -n 2 selftest fails without the wait.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

On shared hosts that permit following other users’ /tmp symlinks, the predictable temporary path enables an attack that can make rm -rf delete the test account’s existing checkout outside VIP.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread src/vip_tests/workbench/test_git_ops.py Resolved
The per-worker /tmp/vip_git_<worker> path is predictable, so another user could
pre-create it as a symlink and aim the clone's rm -rf at the test account's
files. The command now stops unless the directory is a real one the test user
owns, and makes it 0700 first.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Shared-account sign-out depends on xdist teardown timing, but the separate-user smoke run and synthetic barrier selftest do not validate a full parallel --headless-auth run against real Workbench, including cleanup and cache restoration.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@ian-flores
ian-flores merged commit 1a05f96 into main Nov 19, 2026
51 checks passed
@ian-flores
ian-flores deleted the wb-smoke-parallel-644-reland branch November 19, 2026 17:58
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.2
Preview removed because the pull request was closed.
2026-11-19 17:58 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Let the Workbench smoke suite run in parallel again

2 participants