supersonic-interlink: scale from zero, and stop the Failed pods piling up

[kondratyevd]

Two independent changes to supersonic-interlink.

1. Failed Triton pods no longer accumulate

When the Slurm job exits, interLink's virtual kubelet marks the pod Failed rather than restarting
the container in place (restartCount: 0, terminated: exitCode 255). The ReplicaSet then creates
a replacement and the corpse stays — one more every couple of minutes for as long as the job keeps
failing. There were 22 in the namespace while this was being written.

Nothing in Kubernetes sweeps them: PodGC only fires at the cluster-wide terminated-pod-gc-threshold,
and ttlSecondsAfterFinished is a Job field, not a Deployment one. So a five-minute CronJob keeps
the newest — worth having for diagnosis — and deletes the rest.

kubectl get pods -l app.kubernetes.io/instance=supersonic-interlink,app.kubernetes.io/component=triton \
  --field-selector status.phase=Failed --sort-by=.metadata.creationTimestamp -o name \
| sed '$d' | xargs -r kubectl delete --wait=false

Run read-only against the live namespace: 22 failed, 21 selected, newest kept. sed '$d' and
xargs -r were checked inside alpine/kubectl:1.34.1 itself (busybox — no tac, no head -n -1),
including that an empty list is a no-op and not an error.

2. Scale from zero

SuperSONIC main (#90) dropped the
scaleFromZero.enabled is incompatible with envoy.external_config.load_from_configmap guard: a
custom configuration is now responsible for the clusters, Lua filter and routes the generated one
would have added. keda.minReplicaCount goes to 0, so no Slurm allocation is held while idle.

envoy-config.yaml is rewritten as the configuration the chart generates, plus the three things
a Triton behind a Slurm-side wstunnel needs that the chart cannot express:

• a 32 MB request buffer
• outlier ejection on the Triton cluster
• the aggressive retry policy on the inference route

and, from the generated side: the Lua filter, the triton_admission (127.0.0.1:8080) and
envoy_admin (127.0.0.1:9901) clusters, the RepositoryIndex route, dns_refresh_rate: 1s and
connect_timeout: 2s. The Lua filter sits ahead of the buffer so a wake starts on the headers
rather than after a body has been buffered.

The one number worth arguing about

readyTimeoutSeconds: 900, where the other releases use 300. Waking here queues a Slurm job on
Hammer and waits for it to start, rather than scheduling a pod. The route's idle_timeout follows
at 1020s — the readyTimeoutSeconds + 120 margin the chart applies. Both are guesses at Hammer's
queue for one GPU; if RepositoryIndex starts timing out, those are the two to raise, and they move
together by hand because the config file cannot read the values.

A client calling RepositoryIndex on a cold release will now block for up to 15 minutes rather than
fail fast. That is the deliberate trade.

Checks

envoy --mode validate accepts the config against the chart's rendered Lua filter — 3 clusters,
1 listener, Lua compiled; the only warning is the deprecated dns_refresh_rate the chart itself
emits. validate-manifests.sh ✓ 32 HelmReleases, 188 resources · pre-commit --all-files ✓ ·
pytest tests/manifests ✓ 166 passed.

Note the validator skips this release (its Envoy config is read via lookup, which helm template
cannot resolve), so the values were rendered separately with
external_config.load_from_configmap=false, and the sidecar wiring was confirmed by reading the
chart templates: the Lua ConfigMap and the admission sidecar are gated on scaleFromZero alone,
with no external_config condition.

Whether a wake actually completes inside 900s is for the first cold start on Hammer to answer.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!