Add Porter sandbox backend and deploy provider - #876
Open
16francej wants to merge 18 commits into
Open
Conversation
…ored OpenAPI spec
The fake Porter's exec remap replaced guest /tmp/ after substituting host paths, so on Linux (where tmpdir() lives under /tmp) the substituted home/app/volume paths were corrupted and volume persistence, restart, and abort tests failed. Remap guest /tmp before the host-path substitutions instead. Also drop the unknown-DEPLOY_PROVIDER fallback test that contradicted the branch's own (and main's) throw-on-unknown validation, and run Prettier on test/secret-schema-drift.test.ts.
- knip: drop the unused PorterClientLike/PorterSandboxLike re-export from porter-sandbox.ts (consumers import from porter-client.ts) - abort test: the fake Porter's exec used spawnSync, which blocks the event loop for the exec's full duration, so the abort timer could never fire and the kill exec could never run concurrently; switch to async spawn (the macOS pass was accidental — setsid is missing there, so the killable wrapper failed instantly) - redeploy flakes: under concurrent sharded test load a freshly forked node can take several seconds to start, so widen the test's readyWindowSec from 3 to 10, and have the test server send Connection: close so readiness-probe keep-alive sockets to a retired body are never reused
Deployed apps now get private visibility unless PORTER_DEPLOY_VISIBILITY opts into public, matching the fly provider's refusal of public ingress by default. A provision without an egress token no longer rotates a live proxy-mode body to an open one — it reuses the pinned body without injecting proxy env, matching the sprites backend's one-way in-place policy. Rotating killed the running computer whenever a token-less caller (e.g. the monitor poller) provisioned mid-turn, and silently downgraded egress from proxy to open. Upgrading an open body to proxy still rotates, since Porter pins allowed destinations only at creation.
--sandbox porter validates the Porter token/project/cluster env, passes the PORTER_* config through to core, and opens the same self-API tunnel as the other cloud backends. Verified live against a real Porter EKS cluster: a dev-instance agent turn executed on a Porter gVisor sandbox.
porter apply is the default path: Porter builds the six surfaces from their Dockerfiles and assigns each web service a hostname, so a new operator needs no DNS record, TLS certificate, or ingress controller. An apps domain is the upgrade step that gives published apps stable hostnames, and is what the Helm chart expects. Core now warns at config load when DEPLOY_PROVIDER=porter has no PORTER_DEPLOY_APPS_DOMAIN, so the missing address surfaces at startup rather than at the first agent deploy.
A fresh-context agent deployed the full stack onto a real Porter EKS cluster following docs/porter.md; every fix here is a wall it hit. - porter apply reads one app per invocation and silently ignores the rest of a multi-document file, so porter/porter.yaml becomes one file per app under porter/apps/ - auth, web-ui, and admin carry private: true — Porter was minting a public onporter.run ingress for every service, contradicting the portal-only topology in deploy/README.md - the Helm chart's image reference could never name the published ghcr.io/yc-software/qm/<service> images (wrong org, dash joiner); the chart now also wires CORE_API_URL, the portal upstreams, and the public-URL variables that had to be hand-assembled - a publish rejected because the cluster has no sandbox ingress now says so and names the dashboard setting, instead of surfacing a bare HTTP 400 - docs/porter.md gains the per-service wiring table and the traps the deploy hit: arm64 builds, env-group resolution, the two-apply hostname dance, RDS force_ssl, image vintage, sandbox ingress
A second fresh-context deploy proved the earlier fixes hold (~35 min, five old walls now documented away) and surfaced one real bug: with DEPLOY_PROVIDER=porter and no PORTER_DEPLOY_APPS_DOMAIN, apply created a body, timed out opaquely, and leaked a running sandbox per attempt. apply now refuses before creating anything, and the startup warning says publishes will fail rather than "may have no address". Also from that run: core gets private: true like the other non-portal surfaces (a literal doc-follower was putting core on a public onporter.run hostname), and docs/porter.md completes the wiring table (secret-schema.ts named as the source of truth), replaces the gated --remote suggestion with pushing to the project registry, and says how to read the assigned hostname.
A third end-to-end pass installed the chart on a real cluster and drove the deploy and egress paths against it. deploy/helm actually installs now: the chart enabled an egress-proxy image the release workflow never built (guaranteed ImagePullBackOff, so the workflow now publishes it), emitted none of the OIDC_*/AUTH_* broker wiring that cli/src/services.ts derives (portal and auth crash-looped on OIDC_JWKS_URI and AUTH_ISSUER), and scripts/deploy-helm.sh built <repo>-<svc> while deploying <repo>/<svc>, so it always shipped images it had not built. Porter refuses to create a sandbox at all when a cluster has egress restriction turned off — the agent gets no computer rather than an unenforced one — so that rejection now says which cluster setting to turn on. porter-sandbox drops a byte-identical copy of forceThroughProxyEnv in favour of the shared helper, and the deploy provider's remediation points at the API path rather than the dashboard. docs/porter.md gains what the run proved: cluster creation and sandbox ingress are both API-able (contract routes), the printed kubeconfig is read-only so Helm needs an EKS access entry, Porter's sandbox NetworkPolicy blocks RFC1918 so an in-cluster egress proxy is unreachable, and the sandbox wildcard cert needs a Route53-capable cert-manager the standard grant does not provide.
Enabling sandbox ingress on a live cluster stranded ours: a revision submitted while another was reconciling superseded it, the interrupted one stopped with CONCURRENT_UPDATE, and the cluster stayed UPDATING with nothing running. Porter then refuses both a new contract and a delete, each citing that status, so only support can clear it. Declare the sandbox load balancer at cluster creation and revise one at a time.
A stranded cluster is not idle. Deleting its EKS cluster and nodes by hand had them rebuilt within ten minutes by Porter's Cluster API controllers, assuming porter-manager from Porter's own account — CloudTrail names the caller and the aws.cluster.x-k8s.io user agent. The teardown only holds once the CAPIManagement trust statements come off porter-manager and porter-access-manager, so for a wedged cluster the IAM revoke comes first rather than last, inverting the usual order. Also: cluster DELETE needs an admin-role Porter token, which is worth checking before provisioning anything, and the status gate runs before protojson parsing, so a wedged cluster cannot be used to probe whether a contract field exists.
A live cluster settled how addressing actually works: declare the sandbox load balancer with a root domain in the creation contract, and Porter names every published app `<app>.<root domain>` on a wildcard Let's Encrypt certificate it obtains itself — creating the delegated hosted zone, the parent-zone NS record, and a per-cluster porter-cert-manager-route53-<id> role scoped to that zone. A probe with no `domains` key came back with a real hostname, so the up-front refusal this branch added was blocking the turnkey path: PORTER_DEPLOY_APPS_DOMAIN is for choosing a different name, not for having an address at all. apply() now proceeds without it and uses whatever host the cluster names, failing only when the cluster names none — and retiring the body on that path, which is what the refusal was really protecting against. The docs' claim that no porter-* role ever gets Route53 actions was wrong, and the manual wildcard-record instructions were unnecessary; both are corrected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Revives #702 (by @GKestenberg) with CI fixed:
/tmp/after host-path substitution, corrupting paths on Linux runners wheretmpdir()is under/tmp(passed on macOS, where it's/var/folders) — remap guest/tmpfirstDEPLOY_PROVIDER-falls-back-to-docker test, which contradicted the PR's own and main's throw-on-unknown validationtest/secret-schema-drift.test.ts, merged latest mainOriginal PR description: adds Porter as a sandbox backend (
SANDBOX_BACKEND=porter) and deploy provider (DEPLOY_PROVIDER=porter) via a small Porter API client, plus a Helm chart underdeploy/helm/. Covered bytest/porter-sandbox.test.tsandtest/porter-deploy-provider.test.tsagainst a fake Porter API.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.