Add real setup CLI deployment and AI configuration end-to-end tests

Author: msukkari

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - Added privacy-scoped setup wizard funnel telemetry with deployment identity handoff and Node 24 support. [#1653](https://github.com/sourcebot-dev/sourcebot/pull/1653)
+- Added isolated live setup CLI deployment tests covering public code hosts, local clones, AI configuration, search, and restart identity; repaired cross-platform test-runner cleanup. [#1653](https://github.com/sourcebot-dev/sourcebot/pull/1653)
 
 ## [5.1.13] - 2026-09-12
 

packages/setupWizard/telemetry.html

@@ -1743,6 +1743,7 @@ <h3>Reo coexistence</h3>
 <p>The release notes and telemetry documentation should state that setup-wizard PostHog telemetry is always on when its code executes, list the categories collected, explain that PostHog creates a Person profile keyed only by a random pseudonymous session/deployment ID, explain the new-deployment ID handoff, and distinguish PostHog from Reo’s separate tracking and variable. They must not describe the project ingestion token as a secret.</p>
 <h2>Testing plan</h2>
 <h3>Implementation verification status</h3>
+<p><strong>Live deployment suite:</strong> <code>tests/e2e/liveDeployment.mjs</code> builds and installs the npm tarball, drives the real interactive CLI, and starts the released Sourcebot <code>v5.1.13</code> image with real PostgreSQL, Redis, migrations, backend, and web processes. It exercises public GitHub repository/organization/user scopes, GitLab projects, Gitea repositories, Gerrit projects, generic Git URLs, three local-clone layouts, automatic startup/interruption, and AI configuration. Each deployment uses an isolated Compose project and loopback port; only the test Compose networking/image settings and deployment telemetry opt-out are changed. Generated configuration, environment values, local mounts, and install-ID handoff remain intact. See <code>tests/e2e/liveDeployment.md</code> for reproduction and limits. A final clean rerun of all thirteen live scenarios is in progress; this does not replace the still-required full scenario/branch-coverage accounting.</p>
 <p><strong>Live AI validation scope (user clarification):</strong> verify that the packaged wizard writes the selected provider/model configuration and credential references correctly, that the generated environment values reach the resulting Sourcebot container unchanged, and that Sourcebot starts successfully. Do not invoke Ask or make model-inference requests. A Pro license is not required for this test scope. Use available development credentials without printing them; where credentials are unavailable, distinguish configuration/startup coverage with synthetic values from actual credential validation. Continue real repository discovery, indexing, search, browser access, and restart tests independently.</p>
 <p>The feature is not release-complete until the full completion gate below is satisfied. Executable tests cover packed-package collectors, terminal paths, real SDK payloads, Docker branches, and actual container identity continuity. CI definitions are not evidence that Windows or other remote jobs have run.</p>
 <p>Implementation-discovered changes for the behavioral baseline allowlist: status 130 for Ctrl+C; cancellation-aware readiness cleanup; explicit final process exit after bounded telemetry shutdown; recoverable autocomplete fallback on network/decode failure with an 8-second timeout; and fixed, credential-free SDK shutdown-timeout diagnostics where PostHog itself emits them. Do not patch SDK internals or silence the global console. Empty search/catalog results are not failures; actual infrastructure errors can emit recoverable diagnostics while retaining manual fallback.</p>

packages/setupWizard/tests/e2e/liveDeployment.md

@@ -0,0 +1,52 @@
+# Live deployment verification
+
+Run `liveDeployment.mjs` with Node 24 on macOS or Linux with Docker available.
+It builds and installs the actual npm tarball in a disposable directory, drives
+the interactive CLI through a PTY, starts the real Sourcebot/Postgres/Redis stack,
+onboards a synthetic owner through Chromium, verifies indexing and authenticated
+search, then verifies health, search, and install-ID continuity after restart.
+
+Pull `docker.sourcebot.dev/sourcebot-dev/sourcebot:v5.1.13`, `postgres:16`, and
+`redis:8` first. The default scenario is `github_repo`. For the public/configuration
+matrix, set:
+
+```sh
+SETUP_TEST_LIVE_CASES=github_repo,github_org,github_user,gitlab_project,gitea_repo,gerrit_project,remote_git,local_root,local_wildcard,local_nested,auto_start,ai_all \
+SETUP_TEST_LIVE_OUTPUT=/absolute/path/outside/the/repository \
+node packages/setupWizard/tests/e2e/liveDeployment.mjs
+```
+
+The optional `anthropic` scenario reads `ANTHROPIC_API_KEY` from development env
+files under `SETUP_TEST_CREDENTIAL_DIR`. Never pass the credential as a command
+argument. `ai_all` configures all twelve providers with synthetic values. Neither
+scenario calls Ask or an inference endpoint; neither proves credential validity
+or requires a Pro license. The tests check generated model configuration,
+credential references, container environment propagation, and successful startup.
+
+Every deployment uses a unique Compose project, random loopback HTTP port, and
+unpublished database/Redis ports. The Compose download is intercepted only to
+apply these isolation settings, pin the image, and disable deployment telemetry.
+The wizard's generated config, `.env`, identity, and local-repository override
+remain intact. Wizard PostHog requests use the real SDK and are routed to the
+local TLS collector; live public autocomplete/model-catalog requests remain real.
+This suite does not forward to production or dev PostHog. The separate dev smoke
+test provides actual ingestion/query evidence.
+
+Public fixtures are bounded: one explicit repository/project per host, the
+small `chalk` organization, and the `octocat` user. Organization/user tests obtain
+the expected repository count before starting and require all repositories to
+be discovered and indexed. Multi-local tests clone two distinct origins because
+Sourcebot deduplicates multiple local clones of the same origin.
+
+Each scenario records success/failure and removes its containers, volumes,
+network, generated secrets, and temporary files in cleanup. Only redacted reports
+and browser screenshots remain when an output directory is supplied. Task-owned
+image downloads should be removed by the operator afterward without removing
+images that existed before the run. Do not commit generated artifacts.
+
+This live matrix supplements, not replaces, the deterministic collector, fault,
+transport, lifecycle, and platform suites. It does not establish live access to
+private/GHE/GitLab self-managed, Azure DevOps, or Bitbucket deployments, and does
+not claim every possible code-host scope or credential mode has live coverage.
+Unavailable integrations must remain explicitly unverified in the completion
+report rather than being inferred from fixture-based coverage.