fix(self-managed): tolerate API Keys cold start in admin token issuer proxy

[mikeyrcamp]

Problem

During a fresh self-managed control-plane installation,
admin-token-issuer-proxy can start before API Keys is listening. Its first
metadata request is refused and the proxy exits, causing repeated container
restarts during an otherwise healthy cold start. Kubernetes eventually
recovers the pod after API Keys becomes ready, but the avoidable restart loop
adds noise and can consume Helm readiness budgets on slower clusters.

This is distinct from API Keys authentication failures: in the observed
recovery, API Keys became ready and the proxy converged without configuration
changes or pod deletion.

Expected behavior

The proxy tolerates temporary API Keys unavailability during startup and
becomes ready once its dependency is available without a crash loop.

Acceptance criteria

• Retry connection-refused and other explicitly transient API Keys startup
  failures with bounded backoff.
• Keep readiness false until metadata initialization succeeds.
• Fail clearly on permanent configuration or authentication errors.
• Add a startup-order regression where the proxy starts before API Keys.
• Verify Helm installation converges within its documented readiness budget.

---

By submitting this issue, you acknowledge that you are an assigned member of
the NVCF development team and agree to follow the project code of conduct and
contributing guidelines.

[mikeyrcamp]

Current-main reproduction and candidate verification:

• On a fresh install, API Keys completed its normal Cassandra-backed startup
  successfully but took about 2m20s to become ready. The released proxy image
  made one metadata request per process start, exited on connection refusal,
  and restarted six times before Kubernetes eventually retried after API Keys
  was available. This evidence does not indicate a Cassandra regression; the
  API Keys application and its migrations completed without intervention.
• A candidate application fix now retries only explicitly transient transport
  failures and HTTP 408/429/5xx responses with exponential backoff capped at
  10s. Malformed URLs, unsupported schemes, certificate verification failures,
  permanent HTTP responses, and invalid metadata still fail immediately.
• The candidate exposes separate liveness and dependency-aware readiness
  endpoints. The related chart change is in fix(admin-token-issuer-proxy): separate readiness from liveness #1237.

Repository verification passed:

go test ./... -count=1
go vet ./...
go build ./cmd/admin-issuer-proxy
tools/ci/check-helm-charts

Fresh k3d verification used released chart 1.4.3 with a locally built
candidate image and generated Helmfile value overrides (no live resource
patches). API Keys took 143.299s to start. During that interval the proxy logged
connection-refused retries at 1s, 2s, 4s, 8s, then 10s; readiness remained
false and the process had zero restarts. After API Keys became ready, the same
proxy process cached metadata, became ready, and issued an admin token through
an accepted/resolved Gateway API route. Only structural response checks were
recorded; no token was printed or retained.

Delivery blocker: the owning application source repository is currently
archived/read-only, so the tested application commit cannot be pushed for
review or released. The repository owner needs to unarchive it or identify the
active source destination. After a compatible image is released, #1237 can
update the default image tag and leave draft status.

[mikeyrcamp]

Third independent fresh-start result using released chart 1.4.3:

• API Keys started normally.
• The admin-token issuer proxy restarted five times, then self-recovered without manual intervention.
• Admin-token-dependent setup subsequently succeeded.

The Cassandra-causation hypothesis is NOT REPRODUCED: API Keys itself started normally in this run, while the proxy repeated the same dependency cold-start behavior.

The restart-loop defect remains STILL BLOCKED on a publishable application artifact with bounded readiness/retry behavior. This result does not restore the deferred draft PR to the required campaign merge queue.

[mikeyrcamp]

A fifth independent fresh install reproduced the same current cold-start behavior:

• API Keys reached 2/2 Running with zero restarts.
• The admin-token issuer proxy reached 2/2 Running after exactly five restarts.
• Recovery required no pod deletion, resource patch, or other intervention.

This continues to disprove the earlier Cassandra-causation hypothesis for the proxy restart loop, but it does not satisfy this issue's no-restart acceptance criterion.

The same validation later stopped for a separate, directly identified ESS/Cassandra schema mismatch during API account bootstrap. That failure is not attributed to the admin-token proxy: focused API logs reported an undefined ESS schema column, while the proxy had independently completed its known five-restart recovery.

[mikeyrcamp]

A sixth independent fresh start reproduced the same bounded cold-start
behavior while validating an unrelated ESS schema compatibility change:

• API Keys reached 2/2 Ready with zero restarts.
• The admin-token issuer proxy self-recovered to 2/2 after exactly five
  restarts.
• No Pod was deleted, patched, or manually restarted.
• API account bootstrap then completed successfully and the full PKI/LLM BDD
  finished with 4/4 scenarios and 69/69 steps passing.

This is direct evidence that the proxy symptom persists independently of the
ESS/Cassandra schema boundary and still does not satisfy the issue's
no-restart acceptance. It also confirms that the symptom remains
self-recovering in this tested topology.

[balajinvda]

🎉 This issue has been resolved in version admin-token-issuer-proxy-v1.1.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀