Skip to content

[Test] Verify the VS Code load failure in test_publish_to_connect fails loudly on a live deployment #718

Open

Description

@ian-flores

Summary

PR #713 (fix(workbench): stop masking VS Code load failures as a skip) narrows open_vscode_session's VS Code load-wait in src/vip_tests/workbench/test_publish_to_connect.py from a bare except Exception (which turned every failure, including a genuine deployment defect, into a skip) to except PlaywrightTimeoutError, so any other exception now propagates as a real test failure.

This behavior change cannot be exercised in CI — there is no live Workbench deployment there, and the product-test suite is only collected (--collect-only), never run, on PR/push. It needs to be verified against a real deployment before the next live VIP run.

What to verify

Against a real Workbench + Connect deployment, run the workbench category (which includes test_publish_to_connect.py) via vip verify, and confirm both of these hold:

  1. The happy path still passes. When VS Code genuinely is not installed (or is slow enough to hit TIMEOUT_IDE_LOAD), the scenario still records attest.not_applicable("VS Code did not load within timeout — the IDE may not be installed on this Workbench instance") and the run stays green.
  2. A real failure now fails loudly. Force a non-timeout failure during the VS Code load wait (e.g. tear down the session mid-load, or otherwise cause a PlaywrightError other than a timeout) and confirm the scenario now reports a hard test failure instead of a silent skip — the exact gap issue [Feature] Distinguish "not applicable" from "could not verify" and fail the run on unproven checks #616 exists to close.

Why this needs a human/live check

Every other line of this PR is covered by selftests (selftests/test_skip_triage.py) and --collect-only, but neither exercises the actual Playwright wait against a running Workbench session. Closing this issue is the record that someone ran the scenario against a live deployment and confirmed both outcomes above, rather than the fix shipping unverified.

Related: #616, PR #713.

Activity

  1. changed the title [-]test_publish_to_connect.py: verify VS Code load-failure now fails loudly against a live deployment[/-] [+][Test] Verify the VS Code load failure in test_publish_to_connect fails loudly on a live deployment[/+] on Nov 4, 2026
  2. added theissue type on Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions