Repository navigation
ci(workbench): run the smoke suite in parallel, one user per worker - #840
Merged
Merged
Conversation
Workbench refuses a second sign-in by the same user within auth-sign-in-throttle-seconds (5 s by default) and reports it as "Temporary server error", which is why parallel workers sharing one account failed. The smoke workflow now creates <user>-gw<N> per xdist worker and sets VIP_TEST_USERNAME_PER_WORKER=1 so each worker signs in as its own account. The login retry now waits 6 s so a retry clears the throttle window. Closes #644
Every IDE scenario cloned into /tmp/<repo> and removed it first, so parallel workers removed or cloned over each other's checkout; as different users the rm failed with Permission denied. Clone into /tmp/vip_git_<worker>/<repo> instead.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The password-auth smoke run assigns all Workbench scenarios to one loadgroup, preventing parallel execution and leaving per-worker IDE/module grouping and a collection regression test necessary.
1 open issue
What changed in this PR
This PR aims to shorten VIP’s Workbench smoke runs by giving parallel test workers separate accounts, avoiding per-user login throttling and shared Git checkout conflicts.
Changes:
- Provisions per-worker accounts and passes a matching worker count to the smoke run.
- Adds opt-in worker usernames, isolated Git checkout directories, and a longer login retry delay.
- Adds regression tests and documents the per-worker setup.
| File | Description |
|---|---|
src/vip_tests/workbench/test_git_ops.py |
Uses worker-specific Git checkout directories. |
src/vip_tests/workbench/naming.py |
Adds opt-in per-worker username derivation. |
src/vip_tests/workbench/login.py |
Raises the default login retry delay to six seconds. |
src/vip_tests/workbench/conftest.py |
Supplies worker-specific usernames through the Workbench fixture. |
src/vip_tests/workbench/cleanup.py |
Targets the worker’s account during token-based cleanup. |
selftests/test_workbench_parallel.py |
Tests username derivation and the retry-delay default. |
selftests/test_workbench_git_ops.py |
Tests worker directory isolation and creation. |
selftests/test_workbench_cleanup.py |
Tests cleanup targeting the worker-specific account. |
AGENTS.md |
Documents the opt-in setting and CI account provisioning. |
.github/workflows/workbench-smoke.yml |
Creates worker accounts and configures the matching test worker count. |
Review effort: Balanced
pytest-xdist reads xdist_group markers in its own collection hook, which ran before VIP's, so no group VIP assigned ever took effect and --dist loadgroup spread every test freely. Run VIP's hooks tryfirst, keep groups only for product directories, and group Workbench scenarios by IDE and module when VIP_TEST_USERNAME_PER_WORKER is set.
pytest 9.2 reads node ids from a field pytest-xdist 3.8.0 does not write its loadgroup suffix to, so every xdist_group was ignored (pytest-dev/pytest-xdist#1407). Pin 9.1.1 and keep Dependabot off 9.2 until a newer xdist release.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A critical concurrency issue groups login and sign-out together, allowing sign-out to revoke sessions still used by parallel workers despite the ordering marker.
1 open issue
1 resolved since last review
Review effort: Balanced
| return f"workbench_{stem}" | ||
|
|
||
|
|
||
| @pytest.hookimpl(tryfirst=True) |
Contributor
|
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Closes #644.
The Workbench smoke suite ran serially (
-n 0) because parallel workers failed to log in with "Temporary server error". The cause is Workbench's per-user sign-in throttle: rserver refuses a second sign-in by the same user withinauth-sign-in-throttle-seconds(5 s by default). Maria Semple suggested giving each worker its own user, and that fixes it.<user>-gw<N>for each xdist worker and runs with a matching-n.VIP_TEST_USERNAME_PER_WORKER=1makes each worker sign in as its own user. It's off by default, so a customer run keeps its single account./tmp/vip_git_<worker>/. The shared/tmp/<repo>let workers delete each other's clones.