Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ updates:
- dependency-name: pytest-bdd
update-types:
- version-update:semver-major
# pytest 9.2 stores node ids where pytest-xdist 3.8.0 does not write the loadgroup suffix, so
# --dist loadgroup silently ignores every group (pytest-dev/pytest-xdist#1407). Drop this once a
# pytest-xdist release after 3.8.0 is pinned; TestXdistSeesVipGroups fails if it comes off early.
- dependency-name: pytest
versions:
- ">=9.2"
package-ecosystem: uv
schedule:
interval: weekly
Expand Down
30 changes: 26 additions & 4 deletions .github/workflows/workbench-smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -247,11 +247,31 @@ jobs:
env:
SUITE: ${{ inputs.suite || (github.event_name == 'schedule' && 'full' || 'gate') }}

# Workbench refuses a second sign-in by the same user within
# auth-sign-in-throttle-seconds (5 s by default) and answers it with
# "Temporary server error", so xdist workers sharing one account trip it.
# Give each worker its own account, named the way VIP derives it under
# VIP_TEST_USERNAME_PER_WORKER (<user>-gw<N>), with the same password. The
# password goes in on stdin so it never appears in a process list.
- name: Create per-worker test users
id: workers
env:
CID: ${{ steps.workbench.outputs.CONTAINER_ID }}
WB_USER: ${{ steps.workbench.outputs.WORKBENCH_USER }}
WB_PASSWORD: ${{ steps.workbench.outputs.WORKBENCH_PASSWORD }}
run: |
WORKERS=$(nproc)
for i in $(seq 0 $((WORKERS - 1))); do
docker exec "${CID}" useradd -m -s /bin/bash "${WB_USER}-gw${i}"
printf '%s:%s\n' "${WB_USER}-gw${i}" "${WB_PASSWORD}" | docker exec -i "${CID}" chpasswd
done
echo "Created ${WORKERS} per-worker users (${WB_USER}-gw0..$((WORKERS - 1)))"
echo "count=${WORKERS}" >> "$GITHUB_OUTPUT"

# Run the subset of tests that work against Docker Workbench
- name: Run Workbench smoke tests
# -n 0 disables xdist (overriding -n auto from pyproject.toml) because
# the Docker Workbench returns "Temporary server error" when multiple
# workers attempt concurrent logins.
# -n matches the per-worker users created above, so every worker has an
# account to sign in as.
#
# No `-k` filter: the plugin already deselects scenarios for products
# that are not enabled in vip.toml, so the previous `-k "workbench"`
Expand Down Expand Up @@ -317,14 +337,16 @@ jobs:
fi
# shellcheck disable=SC2086 # word splitting is how the file list is passed
uv run pytest $files \
-v -rs -n 0 \
-v -rs -n "${WORKERS}" \
Comment thread
ian-flores marked this conversation as resolved.
Resolved
--vip-config=vip.toml \
--junitxml=smoke-results.xml
env:
# PR/push get the gate set. The nightly schedule runs everything, so the
# slow suites are still exercised daily without taxing the merge path.
SUITE: ${{ inputs.suite || (github.event_name == 'schedule' && 'full' || 'gate') }}
VIP_TEST_PASSWORD: ${{ steps.workbench.outputs.WORKBENCH_PASSWORD }}
VIP_TEST_USERNAME_PER_WORKER: "1"
WORKERS: ${{ steps.workers.outputs.count }}

# Tolerant, unretried upload -- see connect-smoke.yml's "Upload test
# results" step for the full rationale, including why no safe retry
Expand Down
8 changes: 7 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ Key principles:
| `src/vip/install/plan.py` | Pure `build_install_plan` / `build_uninstall_plan` builders |
| `src/vip/install/runner.py` | Plan executor: dry-run formatting + execute (system packages, Playwright, manifest writes) |
| `src/vip_tests/conftest.py` | Directory-scoped warning filter (kept out of the global plugin deliberately) plus the three autouse Connect content-cleanup fixtures — see that file's docstring for why those stay directory-scoped instead of moving to `src/vip/fixtures.py` |
| `src/vip_tests/workbench/naming.py` | Worker-scoped Workbench session names: `current_worker_id`, `vip_session_prefix`, `capacity_session_prefix`, `k8s_session_prefix`, `unique_session_name` — see "Workbench session ownership" below |
| `src/vip_tests/workbench/naming.py` | Worker-scoped Workbench session names: `current_worker_id`, `vip_session_prefix`, `capacity_session_prefix`, `k8s_session_prefix`, `unique_session_name` — see "Workbench session ownership" below; also `worker_username`, the per-worker account behind `VIP_TEST_USERNAME_PER_WORKER` |
| `src/vip_tests/workbench/capacity.py` | Resource-profile detection and capping shared by `test_session_capacity.py` and `test_session_capacity_k8s.py` (`ResourceProfileDisabledError`) |
| `src/vip_tests/workbench/cleanup.py` | Cookie-based Workbench session cleanup, the API-token sweep used when there are no cookies (`--api-auth`), and the autouse cleanup fixtures. `@api_auth` scenarios get the token sweep and never request `page` |
| `src/vip_tests/workbench/sessions.py` | Session-state waits, failure/timeout messages, and the unproven-skip helpers for Workbench scenarios, plus `join_session` — the one way to click a session's join link (waits for Active, retries the click once with a warning, fails if the browser never enters the session). Don't hand-roll `a[title='join ...']` clicks in step files |
Expand Down Expand Up @@ -289,6 +289,7 @@ Configuration is in `vip.toml` (see `vip.toml.example` for the template). Secret
- `VIP_WORKBENCH_API_KEY` — Workbench API token (`Authorization: Bearer`). Under `--api-auth` it launches IDE sessions through the Workbench API and lets cleanup run without a browser; use a super-admin token (see `vip.toml.example`)
- `VIP_TEST_USERNAME`
- `VIP_TEST_PASSWORD`
- `VIP_TEST_USERNAME_PER_WORKER` — set to `1` to give each xdist worker its own Workbench account, `<username>-gw<N>`, sharing the configured password. Workbench refuses a repeat sign-in by one user within `auth-sign-in-throttle-seconds` (5 s by default), which parallel workers sharing an account hit as "Temporary server error". The accounts must already exist; `workbench-smoke.yml` creates them. Serial runs ignore it
- `VIP_TEST_TOTP_SECRET` — optional base32 TOTP seed used by `--headless-auth` to auto-fill MFA codes for a dedicated test service account. **Equivalent to bypassing 2FA — never use a personal account's seed.**

The plugin loads config via `--vip-config` or defaults to `./vip.toml`. If no config file exists, all product tests are skipped.
Expand Down Expand Up @@ -339,6 +340,7 @@ Every render also produces `_output/vip-report.pdf` from `report/vip-report.qmd`
- **Smoke workflows** (`connect-smoke.yml`, `workbench-smoke.yml`, `packagemanager-smoke.yml`, `mock-idp-e2e.yml`) -- run the product suites against real containers. On PR/push each tests a single latest version (change-gated via a `changes` paths-filter job); on `schedule` (nightly, staggered hourly) and `workflow_dispatch` a `set-matrix` job fans each out across the product version support window (current + 2 back). There are no pinned tags to bump: every `set-matrix` job calls `recent-production-versions.yml`, which resolves versions from the product's own GHCR tag list at run time. Each workflow's `*-status` aggregation job is scope-aware: it passes when the suite was legitimately out of scope (the PR's paths didn't match) but **fails** when the suite was in scope (`changes.relevant == 'true'`) yet did not succeed -- so a path-gated skip, an excluded actor, or a missing license secret cannot report a green required check without the suite having run. The Connect, Workbench, and Package Manager `*-status` jobs are required merge checks.
`connect-smoke.yml` also starts a Mailpit container and points Connect's SMTP settings at it (through with-connect's `env:` input, using `CONNECT_<SECTION>_<PROPERTY>` overrides), gives the bootstrap admin an email address, and enables `[email]`, so `test_email` performs a real send instead of skipping. A follow-up step fails the job if Mailpit received nothing.
`mock-idp-e2e.yml` is structured the same way so `Mock-IdP E2E Status` can be promoted to a required check via a separate admin action. It seeds a Keycloak realm and has no Entra lane: Entra ID is a Microsoft-hosted login app that can't be stood up in a container, so the `entra` strategy in `src/vip/idp.py` is validated against a real tenant instead of mock-IdP CI. That test account must also be excluded from Conditional Access device-compliance policies, since headless auth runs in an unmanaged browser Entra's CA would otherwise block.
`workbench-smoke.yml` runs the suite in parallel with one Workbench account per xdist worker: a step creates `<user>-gw0..N-1` in the container (N = the runner's `nproc`, also passed as `-n`) and the test step sets `VIP_TEST_USERNAME_PER_WORKER=1`. One shared account fails under xdist because Workbench throttles repeat sign-ins by the same user (`auth-sign-in-throttle-seconds`, 5 s by default). With the flag set, the Workbench conftest groups scenarios by IDE and by module, as it does under a shared auth session; without it, every Workbench scenario stays in one `workbench` xdist group, so a single account signs in from one worker.
`workbench-smoke.yml` additionally splits into two tiers via a `suite` value: PR/push runs `gate` (the fast subset), the nightly schedule runs `full` (every Workbench file), and `workflow_dispatch` can pick either. The split is based on a measured run rather than taste — `full` buys 8 more real passes for ~523s more test time, which is worth a nightly but not a merge gate. Skip reasons do **not** appear in the log even with `-rs`, because VIP's plugin owns the terminal reporter; read them from `<skipped message=...>` in the uploaded `smoke-results.xml`.
`workbench-smoke.yml` also has an `API-auth smoke test` job that runs `vip verify --api-auth` over `workbench/test_ide_launch_api.py` against the same kind of container with no browser installed. It appends `workbench-api-enabled=1` and `workbench-api-super-admin-enabled=1` to `rserver.conf`, restarts the container with `docker restart` (the image's supervisor stops the container if rserver alone exits), mints a super-admin token with `rstudio-server generate-api-token ... --token-env`, and hands it to VIP as `VIP_WORKBENCH_API_KEY` step env, never in `vip.toml`. It then reruns the RStudio scenario with a `user` token owned by the test user, the other documented launch path. A final step fails the job unless the RStudio scenario passed, rather than skipped, under both tokens. `Workbench Smoke Tests Status` aggregates it with the main job.
The three cross-cutting suites (`cross_product/test_resources`, `security/test_auth_policy`, `config_hygiene/test_secrets`) run in all three product workflows. `test_secrets` asserts no plaintext `api_key`/`password` in the generated `vip.toml`, so credentials must be passed to pytest as step `env` (`VIP_CONNECT_API_KEY`, `VIP_TEST_PASSWORD`) and never written into the config file.
Expand Down Expand Up @@ -426,6 +428,10 @@ refactor(connect)!: rename client constructor parameters
- Missing the colon and space after the type (e.g. `feat add feature` — must be `feat: add feature`).
- Using a PR title that is not conventional when the branch will be squash-merged.

## xdist groups

`--dist loadgroup` (the default in `pyproject.toml`) keeps every test with the same `xdist_group` on one worker. `plugin/selection.py` groups tests in a `connect/`, `workbench/` or `package_manager/` directory by that name and leaves everything else ungrouped; the Workbench conftest refines its group by IDE and module when concurrent logins are safe. pytest-xdist reads the groups in its own `pytest_collection_modifyitems` on each worker, so every hook that adds a group must be `@pytest.hookimpl(tryfirst=True)` -- a group added after xdist's hook is silently ignored. `selftests/test_workbench_parallel.py::TestXdistSeesVipGroups` runs real workers to guard this. pytest stays below 9.2 (pin in `pyproject.toml`, ignore in `.github/dependabot.yml`) until a pytest-xdist release after 3.8.0: pytest 9.2 moved node ids where xdist 3.8.0 writes the group, so every group is ignored (pytest-dev/pytest-xdist#1407).

## Pytest warning filters

Register warning filters in `src/vip/plugin/configure.py::pytest_configure` (via `config.addinivalue_line("filterwarnings", ...)`), not in `pyproject.toml`'s `[tool.pytest.ini_options]`. Filters in `pyproject.toml` only apply when pytest runs from this repo's rootdir -- users who install vip into another project pick up the plugin but not the config, so the warnings reappear there. Keeping the filters in the plugin means they travel with the installed package.
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ dependencies = [
"httpx>=0.27,<1",
"playwright==1.65.0", # exact-pinned: shapes vip run output; see docs/development.md
"pyotp~=2.9",
"pytest==9.2.0", # exact-pinned; at/above CVE-2025-71176 floor (9.0.3)
"pytest==9.1.1", # exact-pinned; at/above CVE-2025-71176 floor (9.0.3); <9.2 until pytest-xdist > 3.8.0 (pytest-dev/pytest-xdist#1407)
"pytest-bdd==9.0.1", # exact-pinned
"pytest-order==1.5.0", # exact-pinned
"pytest-playwright==0.10.0", # exact-pinned
Expand Down
15 changes: 15 additions & 0 deletions selftests/test_workbench_cleanup.py
Original file line number Diff line number Diff line change
Expand Up @@ -1201,3 +1201,18 @@ def test_token_sweep_logs_instead_of_raising_when_the_api_refuses(caplog):
with caplog.at_level(logging.WARNING, logger="vip_tests.workbench.cleanup"):
assert cleanup._quit_vip_sessions_via_token(client, _token_config(), owner="main") == 0
assert "could not sweep VIP Workbench sessions" in caplog.text


def test_token_sweep_targets_this_workers_own_account(monkeypatch):
from types import SimpleNamespace

from vip_tests.workbench import cleanup

monkeypatch.setenv("PYTEST_XDIST_WORKER", "gw1")
monkeypatch.setenv("VIP_TEST_USERNAME_PER_WORKER", "1")
users: list[str] = []
client = SimpleNamespace(
quit_vip_sessions_via_api=lambda user, *, owner: users.append(user) or 0
)
cleanup._quit_vip_sessions_via_token(client, _token_config(), owner="gw1")
assert users == ["alice-gw1"]
25 changes: 23 additions & 2 deletions selftests/test_workbench_git_ops.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,20 +40,41 @@ def fake_terminal_run(page, cmd, timeout=30_000, *, readback_lang="r"):
return ""

monkeypatch.setattr(git_ops, "terminal_run", fake_terminal_run)
monkeypatch.setenv("PYTEST_XDIST_WORKER", "gw3")

git_cfg = _FakeGitCfg("https://github.com/example/jumpstart_examples.git")
git_session_ctx: dict = {}

clone_dir = git_ops._do_clone(object(), git_session_ctx, git_cfg, readback_lang="r")

assert clone_dir == "/tmp/jumpstart_examples"
assert clone_dir == "/tmp/vip_git_gw3/jumpstart_examples"
assert git_session_ctx["clone_dir"] == clone_dir
rm_index = captured["cmd"].find("rm -rf /tmp/jumpstart_examples")
rm_index = captured["cmd"].find("rm -rf /tmp/vip_git_gw3/jumpstart_examples")
clone_index = captured["cmd"].find("git clone")
assert rm_index != -1, f"expected rm -rf of clone dir in command: {captured['cmd']!r}"
assert rm_index < clone_index, "rm -rf must run before git clone"


def test_do_clone_gives_each_worker_its_own_directory(monkeypatch):
# Parallel workers (often different users) must not delete or clone over each other's checkout.
cmds: list[str] = []
monkeypatch.setattr(
git_ops,
"terminal_run",
lambda page, cmd, timeout=30_000, *, readback_lang="r": cmds.append(cmd),
)
git_cfg = _FakeGitCfg("https://github.com/example/repo.git")

dirs = []
for worker in ("gw0", "gw1"):
monkeypatch.setenv("PYTEST_XDIST_WORKER", worker)
dirs.append(git_ops._do_clone(object(), {}, git_cfg, readback_lang="r"))

assert dirs == ["/tmp/vip_git_gw0/repo", "/tmp/vip_git_gw1/repo"]
# The per-worker parent does not exist on a fresh server, so it is created before cd.
assert cmds[0].index("mkdir -p /tmp/vip_git_gw0") < cmds[0].index("cd /tmp/vip_git_gw0")


def test_do_clone_uses_custom_workdir(monkeypatch):
captured: dict = {}

Expand Down
Loading
Loading