Skip to content

ci(workbench): run the smoke suite in parallel, one user per worker - #840

Merged
ian-flores merged 6 commits into
mainfrom
wb-smoke-parallel-644
Nov 18, 2026
Merged

ian-flores merged 6 commits into
mainfrom
wb-smoke-parallel-644

Conversation

@ian-flores

Copy link
Copy Markdown
Collaborator

Closes #644.

The Workbench smoke suite ran serially (-n 0) because parallel workers failed to log in with "Temporary server error". The cause is Workbench's per-user sign-in throttle: rserver refuses a second sign-in by the same user within auth-sign-in-throttle-seconds (5 s by default). Maria Semple suggested giving each worker its own user, and that fixes it.

  • The workflow creates <user>-gw<N> for each xdist worker and runs with a matching -n.
  • VIP_TEST_USERNAME_PER_WORKER=1 makes each worker sign in as its own user. It's off by default, so a customer run keeps its single account.
  • Each worker clones into /tmp/vip_git_<worker>/. The shared /tmp/<repo> let workers delete each other's clones.
  • The login retry now waits 6 s, longer than the throttle window.
Full suite, 3 versions Serial (main nightly) This branch
Smoke job time ~12 min ~5 min
Results 22 pass, 8 skip same, test by test

Workbench refuses a second sign-in by the same user within auth-sign-in-throttle-seconds (5 s by default) and reports it as "Temporary server error", which is why parallel workers sharing one account failed. The smoke workflow now creates <user>-gw<N> per xdist worker and sets VIP_TEST_USERNAME_PER_WORKER=1 so each worker signs in as its own account. The login retry now waits 6 s so a retry clears the throttle window.

Closes #644
Every IDE scenario cloned into /tmp/<repo> and removed it first, so parallel workers removed or cloned over each other's checkout; as different users the rm failed with Permission denied. Clone into /tmp/vip_git_<worker>/<repo> instead.
Copilot AI balanced review requested due to automatic review settings November 18, 2026 22:58
@ian-flores
ian-flores marked this pull request as ready for review November 18, 2026 22:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The password-auth smoke run assigns all Workbench scenarios to one loadgroup, preventing parallel execution and leaving per-worker IDE/module grouping and a collection regression test necessary.

1 open issue
What changed in this PR

This PR aims to shorten VIP’s Workbench smoke runs by giving parallel test workers separate accounts, avoiding per-user login throttling and shared Git checkout conflicts.

Changes:

  • Provisions per-worker accounts and passes a matching worker count to the smoke run.
  • Adds opt-in worker usernames, isolated Git checkout directories, and a longer login retry delay.
  • Adds regression tests and documents the per-worker setup.
File Description
src/​vip_tests/​workbench/​test_git_ops.py Uses worker-specific Git checkout directories.
src/​vip_tests/​workbench/​naming.py Adds opt-in per-worker username derivation.
src/​vip_tests/​workbench/​login.py Raises the default login retry delay to six seconds.
src/​vip_tests/​workbench/​conftest.py Supplies worker-specific usernames through the Workbench fixture.
src/​vip_tests/​workbench/​cleanup.py Targets the worker’s account during token-based cleanup.
selftests/​test_workbench_parallel.py Tests username derivation and the retry-delay default.
selftests/​test_workbench_git_ops.py Tests worker directory isolation and creation.
selftests/​test_workbench_cleanup.py Tests cleanup targeting the worker-specific account.
AGENTS.md Documents the opt-in setting and CI account provisioning.
.github/​workflows/​workbench-smoke.yml Creates worker accounts and configures the matching test worker count.

Review effort: Balanced

Comment thread .github/workflows/workbench-smoke.yml Resolved
pytest-xdist reads xdist_group markers in its own collection hook, which ran before VIP's, so no group VIP assigned ever took effect and --dist loadgroup spread every test freely. Run VIP's hooks tryfirst, keep groups only for product directories, and group Workbench scenarios by IDE and module when VIP_TEST_USERNAME_PER_WORKER is set.
pytest 9.2 reads node ids from a field pytest-xdist 3.8.0 does not write its loadgroup suffix to, so every xdist_group was ignored (pytest-dev/pytest-xdist#1407). Pin 9.1.1 and keep Dependabot off 9.2 until a newer xdist release.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical concurrency issue groups login and sign-out together, allowing sign-out to revoke sessions still used by parallel workers despite the ordering marker.

1 open issue
1 resolved since last review

Review effort: Balanced

return f"workbench_{stem}"


@pytest.hookimpl(tryfirst=True)
@ian-flores
ian-flores merged commit 0c7e44d into main Nov 18, 2026
52 checks passed
@ian-flores
ian-flores deleted the wb-smoke-parallel-644 branch November 18, 2026 23:36
@github-actions

Copy link
Copy Markdown
Contributor
PR Preview Action v1.8.2
Preview removed because the pull request was closed.
2026-11-18 23:36 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Let the Workbench smoke suite run in parallel again

2 participants