Description
Summary
PR #713 (fix(workbench): stop masking VS Code load failures as a skip) narrows open_vscode_session's VS Code load-wait in src/vip_tests/workbench/test_publish_to_connect.py from a bare except Exception (which turned every failure, including a genuine deployment defect, into a skip) to except PlaywrightTimeoutError, so any other exception now propagates as a real test failure.
This behavior change cannot be exercised in CI — there is no live Workbench deployment there, and the product-test suite is only collected (--collect-only), never run, on PR/push. It needs to be verified against a real deployment before the next live VIP run.
What to verify
Against a real Workbench + Connect deployment, run the workbench category (which includes test_publish_to_connect.py) via vip verify, and confirm both of these hold:
- The happy path still passes. When VS Code genuinely is not installed (or is slow enough to hit
TIMEOUT_IDE_LOAD), the scenario still recordsattest.not_applicable("VS Code did not load within timeout — the IDE may not be installed on this Workbench instance")and the run stays green. - A real failure now fails loudly. Force a non-timeout failure during the VS Code load wait (e.g. tear down the session mid-load, or otherwise cause a
PlaywrightErrorother than a timeout) and confirm the scenario now reports a hard test failure instead of a silent skip — the exact gap issue [Feature] Distinguish "not applicable" from "could not verify" and fail the run on unproven checks #616 exists to close.
Why this needs a human/live check
Every other line of this PR is covered by selftests (selftests/test_skip_triage.py) and --collect-only, but neither exercises the actual Playwright wait against a running Workbench session. Closing this issue is the record that someone ran the scenario against a live deployment and confirmed both outcomes above, rather than the fix shipping unverified.
Activity
[-]test_publish_to_connect.py: verify VS Code load-failure now fails loudly against a live deployment[/-][+][Test] Verify the VS Code load failure in test_publish_to_connect fails loudly on a live deployment[/+]