refactor(consul-postgres-ha): collapse platform plumbing to single mesh-sidecar

[h4x3rotab]

Summary

• Per-CVM container count drops from 7→3 on workers (mesh-sidecar + patroni + webdemo) and from 6→1 on coordinators (mesh-sidecar). The new consul-postgres-ha-mesh-sidecar image bundles bootstrap-secrets, mesh-conn, consul, and (workers only) envoy behind a tini-wrapped shell init script that dispatches on ROLE. The old keepalive placeholder, four-image lockstep, and vestigial on-CVM signaling/coturn (documented as unused — Phala dstack apps don't have UDP ingress yet) all drop.
• CI matrix: 6→4 images (mesh-sidecar, patroni, webdemo, signaling). The mesh-sidecar build uses the parent consul-postgres-ha/ as docker context so its multi-stage Dockerfile can pull bootstrap-secrets/ and mesh-conn/ Go sources from sibling subdirs.
• cluster.tf env shrinks: BOOTSTRAP_SECRETS_IMAGE, MESH_CONN_IMAGE, and the coordinator's SIGNALING_IMAGE (plus matching tfvars) all collapse into a single MESH_SIDECAR_IMAGE / mesh_sidecar_image.
• Healthcheck-gated startup: workers' patroni and webdemo wait on service_healthy defined as [ -s /run/instance/info.json ], replacing the old service_completed_successfully gate on bootstrap-secrets.
• Image size 417 MB (sum of old four was ~1 GB on disk).

The compose-service name stays sidecar (so the per-CVM container is still dstack-sidecar-1 regardless of which image it points at). The image name has the mesh- prefix to make it clear that it's the bundle of mesh plumbing — bootstrap-secrets + mesh-conn + consul + envoy — and not just an Envoy sidecar.

Design brief at consul-postgres-ha/design/single-sidecar.md was the spec for this change and is deleted by the PR per the design-doc convention. Open questions decided:

1. Naming → directory mesh-sidecar/, image suffix consul-postgres-ha-mesh-sidecar, terraform var mesh_sidecar_image, env var MESH_SIDECAR_IMAGE. Old envoy-bootstrap shell folded into entrypoint.sh.
2. tester (netshoot) → dropped from both compose files. The new mesh-sidecar image carries curl/jq, so docker exec dstack-sidecar-1 curl ... replaces it. One FAILOVER.md line touched.
3. WEBDEMO_ENABLED=1 conditional → punted. Compose profiles work but introduce a new concept; users adapting the template will be editing webdemo out anyway. Left as a note for a future small change.

Smoke test (2026-05-04, fresh terraform apply on dstack-pha-prod5)

Cluster: 3 coordinators + 3 workers, cluster_name=gap2. Fresh mesh-sidecar image at ttl.sh/dstack-mesh-sidecar-1777870749:24h (the rename-pass apply propagates as an env-only update, no CVM destroy/recreate).

Scenario	This run	2026-05-03 baseline	Δ
Soft-kill RTO	27s	24s	+3s, within noise
Hard-kill RTO	33s	33s	0s
Cheap rejoin	✓ worker-3 came back as streaming replica on the new timeline, lag=0	✓	unchanged
Disk-loss rejoin	26s end-to-end (20s pg_basebackup)	16s end-to-end	within noise (handshake-dominated on a few-MB pgdata)

The hard-kill rejoin path on this run hit the documented worker↔worker ICE-flake (the same case the MESH_CONN_RELAY_ONLY=1 escape hatch in compose/worker.yaml already exists for) — worker-4 couldn't re-establish ICE handshakes after coming back. None of the consolidation work in this branch changed mesh-conn's behavior; the binary is identical, just packaged differently. The escape hatch is documented in both worker.yaml and coordinator.yaml. RTO measurements are unaffected by this rejoin slowness because the failover itself completed in the expected time on the surviving replicas — the cluster reached a new healthy quorum before the slow peer's rejoin became relevant.

Test plan

• Local docker build -f consul-postgres-ha/mesh-sidecar/Dockerfile consul-postgres-ha succeeds; image runs all four bundled binaries (bootstrap-secrets, mesh-conn, consul v1.19.2, envoy 1.30.11)
• Fresh terraform apply -parallelism=1 brings up 6 CVMs in series; each worker shows exactly 3 containers (sidecar/healthy + patroni + webdemo); each coordinator shows exactly 1 container (sidecar/healthy)
• consul members shows 3 servers + 3 clients all alive after fresh deploy
• Patroni cluster healthy on timeline=1 after fresh deploy, then progresses through soft-kill (timeline 1→2→3) and hard-kill (3→4) without manual intervention
• Per-process logs distinguishable inside docker logs dstack-sidecar-1: lines prefixed [init], [bootstrap-secrets], [mesh-conn], [consul], [envoy]
• Env-only rename pass (SIDECAR_IMAGE → MESH_SIDECAR_IMAGE, sidecar_image → mesh_sidecar_image) propagates as 0 added, 6 changed, 0 destroyed. Disks survive (no destroy/recreate).
• CI workflow green on this PR (will run as part of the PR checks; matrix is 4 images)

🤖 Generated with Claude Code