feat(tools): standalone tvc-deploy helper + manual deploy workflow

[prasanna-anchorage]

Summary

A standalone Rust helper + manual workflow to perform dev/staging parser_app TVC deploys, plus a post-deploy smoke check.

tools/tvc-deploy/ — standalone helper (xshell + anyhow + lexopt; own workspace, no visualsign deps)

• gen-operator-key --out <path>: mint a qos_p256 operator key (seed → 0600 file, only the public key printed).
• deploy …: image-derived digest gate — extract /parser_app from the image, sha256 it, and assert it equals --expected-digest before anything is submitted (ties the deployed digest to the actual binary) — then assemble tvc-deploy.json (gRPC health), tvc deploy create → approve → poll-to-healthy → set-live.

.github/workflows/tvc-deploy.yml — manual + label-gated deploy

• workflow_dispatch (explicit inputs) and a tvc-deploy-test-labeled pull_request path (test app via vars.TVC_TEST_*). Pins tvc --version 0.7.0 --locked; minimal permissions.
• Smoke step (post set-live): runs the published turnkey-client container parse --dev-path against /visualsign-dev and asserts the V0+ALT fixture renders (regression guard for "Cannot render V0"), reusing TVC_API_KEY_*; outage abort-guard. Requires the turnkey-client image on GHCR.

Validated: helper proven end-to-end on a throwaway dev app and via a green labeled CI run against the test app 80b6f48f; smoke logic validated live (35,177 chars rendered).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a standalone Rust helper and a manually-triggered GitHub Actions workflow to perform dev/staging TVC deployments of parser_app, including manifest digest gating and health polling, without pulling in the main src/ Rust workspace.

Changes:

• Introduces tools/tvc-deploy/ as an independent Rust crate to generate operator keys and orchestrate TVC deploy flows via the tvc CLI.
• Adds a workflow_dispatch GitHub Actions workflow to build/run the helper with input parameters and scrub the operator seed afterward.
• Vendors a lockfile and minimal .gitignore to keep the helper self-contained and fast to build.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tools/tvc-deploy/src/main.rs	Implements the deploy helper (flag parsing, config generation, digest gate, approve, poll health, set-live).
tools/tvc-deploy/Cargo.toml	Declares the standalone crate and its minimal dependencies, with its own workspace root.
tools/tvc-deploy/Cargo.lock	Locks the helper’s dependency graph for reproducible builds.
tools/tvc-deploy/.gitignore	Ignores the helper’s local target/ directory.
.github/workflows/tvc-deploy.yml	Adds a manual CI workflow to install tvc, build the helper, run deploy, and scrub the seed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[prasanna-anchorage]

Addressed the review feedback in 6c4aef8d:

• CodeQL (missing permissions): added a minimal permissions: { contents: read } to the workflow.
• cargo install reproducibility (+ pin request): now cargo install tvc --version 0.6.2 --locked.
• Operator-seed temp file could leak on early error: moved config assembly/write inside the cleanup closure, so the seed + config temp files are always removed regardless of where the flow fails.
• path() lossy non-UTF-8 → empty arg: removed entirely — the helper was reorganized onto xshell (cmd!), which takes Path/PathBuf directly, so there's no to_str() fallback path anymore.

Also reorganized the helper onto the idiomatic shell-orchestration stack (xshell + anyhow + lexopt, all small/fast-compiling) and replaced the fragile approve --dry-run manifest parse with an image-derived digest gate (extract /parser_app, sha256, assert == --expected-digest) — stronger, ties the digest to the actual binary.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

[pepe-anchor]

must: The pull_request: types: [labeled] trigger runs tvc-deploy deploy, which unconditionally calls tvc deploy approve ... --dangerous-skip-interactive right after create, i.e. a full create->approve->set-live with no human checkpoint, fed by TVC_CI_OPERATOR_SEED/TVC_API_KEY_* secrets, triggered by anyone able to label a same-repo PR tvc-deploy-test. This already happened for real on this PRs own branch (feat/tvc-deploy-helper): GitHub Actions run 28013160686 shows checkout, build, Deploy and Scrub all green from the PRs own head commit. Because pull_request (unlike pull_request_target) resolves both the checked-out code AND the workflow YAML from the PR head, a same-repo collaborator can edit main.rs or this workflow file in the same PR to exfiltrate the operator seed / API keys, and there is no environment: gate or required-reviewer step anywhere in the file to stop it. This directly contradicts the requesters own stated design goal in the linked Slack thread ("give a role to Github thats only allowed to initiate a deploy, it can never approve, only humans should approve") and runs ahead of the phased plan (PRS-516/519/520/521), whose Enabler ticket (PRS-521, still Backlog) is supposed to provision scoped CI keys + a Turnkey policy before any auto-approve happens from CI.

[prasanna-anchorage]

Agreed, and this is the exact concern #404 is designed around: splitting deploy into separate initiate / approve / promote subcommands (behind a TvcOps trait, with unit tests) so CI can be scoped to initiate-only and a human/operator key does the approve. That said, #404 currently only has the main.rs split + design docs landed — the workflow-level fix (retiring this labeled-PR auto-approve path in favor of the planned Release/Promote workflows, Phase 4) hasn't shipped, and PRS-521 (scoped CI keys + Turnkey policy) is still Backlog. So this stays an open risk on the actual tvc-deploy-test label path in this PR until that phase lands. Tracking there rather than duplicating the fix here — leaving this thread open until Phase 4 ships or we explicitly decide to gate/disable the labeled-PR path in the interim.

[pepe-anchor]

must: The outage-vs-regression classifier greps stderr for generic terms including bare timeout, context deadline, and \bEOF\b, and treats any match as a pre-existing outage (exit 0, reported as a pass/skip). A genuinely broken deployment, e.g. an enclave that hangs on the gRPC health check or crashes mid-response, is very likely to surface as exactly context deadline exceeded or an EOF from the clients perspective. Since this step runs immediately after set-live specifically to catch a broken deploy, a deploy broken in a way that manifests as a timeout or dropped connection is misclassified as an unrelated outage and CI reports smoke PASS, defeating the checks purpose at the moment it matters most.

[prasanna-anchorage]

Fixed in c1fb6be: the outage classifier no longer treats bare timeout/EOF/context deadline as a benign outage-skip. It now only skips on unambiguous never-connected errors (connection refused/reset, no such host, dial tcp, tls handshake, network unreachable, etc.); timeout/EOF/context-deadline on an otherwise-reachable endpoint now falls through to an explicit FAIL (exit 1) instead of SKIP, since those are exactly what a hung or crashed enclave would surface as right after set-live.

[pepe-anchor]

should: set_live lowercases combined stdout+stderr from tvc app set-live-deploy and treats a bare substring match on "already" (line 259) as success, regardless of the actual failure mode. If tvc ever emits an unrelated error containing "already" (e.g. a retry-exhaustion message), this would silently report success on a failed deploy in a fully automated, no-human-in-the-loop path. No current evidence this string appears in a real failure message today, so this is a latent risk rather than a proven break.

[prasanna-anchorage]

Fixed in c1fb6be — set_live now requires both "already" and "live" in the combined stdout+stderr before treating it as success, so an unrelated failure that happens to contain "already" (e.g. a retry-exhaustion message) can no longer be misreported as a live deploy.

[pepe-anchor]

should: For the PR-label path, github.ref is per-PR (refs/pull/<n>/merge), so two different PRs both labeled tvc-deploy-test at the same time land in different concurrency groups and can race each others create/approve/set-live against the same vars.TVC_TEST_APP_ID. Same issue between two workflow_dispatch runs targeting the same app_id from different branches. Blast radius is limited to the dev/test app today, not prod.

Suggested change

	cancel-in-progress: false
	Key the group on the resolved app id instead, e.g. `tvc-deploy-${{ inputs.app_id || vars.TVC_TEST_APP_ID }}`.

[prasanna-anchorage]

Fixed in c1fb6be — took the suggested fix, concurrency group is now keyed on inputs.app_id || vars.TVC_TEST_APP_ID instead of github.ref.

[pepe-anchor]

should: These are pure functions scraping freeform tvc CLI output (e.g. Healthy / Desired Replicas: X/Y) with zero test coverage, and the workflow only runs cargo build --release, never cargo test. If tvc ever reformats that text, deployment_health silently stops matching and poll_health spins until timeout with an "unknown" status, with nothing catching the regression before a real deploy run.

[prasanna-anchorage]

Fixed in c1fb6be — added a #[cfg(test)] module covering deployment_health, validate_digest, and parse_after (8 unit tests), and added a cargo test step to the workflow before the build step (previously it only ran cargo build), so a regression in the freeform-output parsing now fails CI instead of silently spinning to an "unknown" status.

[shahan-khatchadourian-anchorage]

Nice, thorough hardening across the review history here — the digest gate, seed scrubbing, and the recent set-live/concurrency/test-coverage fixes all look solid, and I independently re-verified cargo build/cargo test/cargo clippy all pass clean for tools/tvc-deploy.

Ran a second pass looking for anything not already covered in this thread. The one structural gap (no human-approval checkpoint on the label-triggered path) is already tracked as a must above and deferred to #404/PRS-521, so not re-raising it. Found a few smaller things below — none of these block merge, flagging as fast-follow candidates.

[shahan-khatchadourian-anchorage]

Minor, non-blocking: this fallback still has a gap in the "harness failure vs. endpoint outage" distinction the comment above (lines 96-98) says it maintains.

If docker image inspect/docker pull both fail (e.g. GHCR unreachable from the runner) and no --turnkey-client-path is given, we fall through to CLIENT="$CONTAINER_CLIENT" here and try it anyway at line 116. docker run will attempt its own implicit pull, hit the same network failure, and emit text like dial tcp: lookup ghcr.io: ... or temporary failure in name resolution into $ERRFILE — which matches the SKIP regex at line 139, not the "harness couldn't run" path. So a CI-runner-side GHCR outage gets reported as SKIP: dev endpoint unreachable (exit 0) instead of the exit 2 harness failure this comment says it should be, and a real deploy regression underneath it wouldn't get caught.

Would be worth either treating a failed pull explicitly (bail with exit 2 before ever invoking $CLIENT) or having the SKIP/FAIL classifier distinguish "docker itself couldn't run" from "the target endpoint refused/timed out."

[shahan-khatchadourian-anchorage]

Minor, non-blocking: --image-url isn't validated as digest-pinned (e.g. ...@sha256:...), only --expected-digest's hex format is checked (validate_digest, line 329). The workflow's own doc comment asks callers to pass a pinned reference, but nothing here enforces it.

If a mutable tag is passed, verify_image_digest (line 211) hashes whatever that tag resolves to right now and the gate passes — but the same tag string is also written into pivotContainerImageUrl for the TVC deploy config, so if TVC ever re-resolves that tag later (redeploy, restart) it could pull a different image than the one actually verified. A cheap image.contains("@sha256:") check before the digest gate would close this without much code.

[shahan-khatchadourian-anchorage]

Minor, non-blocking: msg.contains("not found") in the transient-retry set is fairly broad — it'd also match a permanent error like a typo'd --operator-id or --deploy-id (e.g. "operator not found in manifest set"), causing set_live to retry every 15s for the full 300s timeout before bailing with a generic message, instead of failing fast with something actionable.

Separately (very minor): the loop checks start.elapsed() < timeout before sleeping (line 270) rather than after, so on a transient message near the deadline it can overshoot SETLIVE_TIMEOUT by up to one POLL_INTERVAL (~15s) — poll_health's loop checks elapsed right before bailing and doesn't have this gap.

[shahan-khatchadourian-anchorage]

Minor, non-blocking, efficiency nit: verify_image_digest spawns 4 subprocesses + a host temp file (docker create → docker cp → sha256sum → docker rm) to hash one file already inside the image. Since parser_app's final image stage is stagex/core-busybox (which ships sha256sum), docker run --rm --entrypoint sha256sum {image} /parser_app would get the identical guarantee (still hashing the real image content, not a rebuilt binary) in one subprocess, with --rm handling cleanup and no host temp file needed at all.