fix(oauth2-proxy): poll keycloak over the backchannel in wait-for-keycloak

[aweingarten]

📌 Summary

Fixes #3388 — the oauth2-proxy item (and, for this container, the resolution failure in #3420).

Note on scope: #3388 lists three independent items; this PR implements the oauth2-proxy one. The other two (gitops-global ignoring otomi.git, and Loki's object_store/schemaConfig being unsettable through _rawValues) are untouched. Since Fixes will auto-close #3388 on merge, those two should be split into their own issue — happy to open it, or leave this as a plain reference instead if you'd rather keep #3388 open until all three land.

The wait-for-keycloak init container polls the public issuer URL with a vanilla curlimages/curl:

args: ["while [ $(curl -sw '%{http_code}' https://keycloak.<domainSuffix>/realms/otomi -o /dev/null) -ne 200 ]; ..."]

The oauth2-proxy pod is explicitly sidecar-less (sidecar.istio.io/inject: "false"), so that URL is reached without the mesh, and it hits one of two walls depending on the deployment:

• the host is an Istio ServiceEntry with no in-cluster DNS record backing it, so it may not resolve at all (Internal ServiceEntry hosts (<svc>.<domainSuffix>) don't resolve for sidecar-less pods / init containers #3420);
• when it does resolve, the Otomi-signed wildcard cert isn't trusted by a stock curl image, so curl exits 60 (self-signed certificate in certificate chain).

Either way the container loops forever and oauth2-proxy never starts. No chart value exposes the init image, a CA mount, or the curl args, so there's no supported way out.

Worth noting the asymmetry this creates: oauth2-proxy's own config already tolerates the untrusted CA — ssl_insecure_skip_verify = {{ $v._derived.untrustedCA }} — so the gate is currently stricter than the app it gates.

The fix points the poll at _derived.oidcBaseUrlBackchannel (http://keycloak-keycloakx-http.keycloak:8080/realms/otomi), which already exists for exactly this purpose and is used by kubernetes-gateways for its JWKS endpoint. Plain HTTP to a ClusterIP Service: nothing to resolve through the mesh, nothing to trust.

🔍 Reviewer Notes

• The gate condition is unchanged. Both URLs return 200 only once the otomi realm is being served; only the route changes. It does not verify the public ingress path — but the init container never usefully verified that either, since it fails on DNS/TLS before reaching Keycloak.
• I picked this over the alternative (mount the platform CA and pass --cacert, following the custom-ca Secret pattern in argocd-raw/gitea-raw). Mounting a CA fixes the TLS half but leaves the DNS half, and it adds a Secret + volume + mount where a value that already exists does the whole job. Happy to switch if you'd rather the gate stay on the public URL.
• The other two items in Charts hardcode values that can't be overridden via apl-values #3388 (gitops-global ignoring otomi.git; Loki object_store/schemaConfig not settable through _rawValues) are untouched here — they're independent and larger. Splitting that issue may be worthwhile.
• The init container image is still curlimages/curl:latest with no value to override it. That part of the Charts hardcode values that can't be overridden via apl-values #3388 ask stands; it seemed better as its own change.

Template-only change. validate-templates/lint:hf need helmfile, which I don't have locally — leaving those to CI.

🧹 Checklist

• Code is readable, maintainable, and robust.
• Unit tests added/updated — n/a, this is a one-line template change covered by validate-templates

[CasLubbers]

The public URL check here isn't just "is Keycloak up", its an end-to-end smoke test (DNS, cert, ingress) gating oauth2-proxy, which is the platform's SSO entrypoint. Switching to oidcBaseUrlBackchannel fixes the sidecar-less DNS/TLS failure, but it also means the gate now passes even when the public path is broken. That defeats the point of gating on it in the first place.

If the goal is just don't crashloop forever, maybe worth keeping the public URL but adding CA trust + in-mesh DNS path (mount platform CA, skip the ServiceEntry issue) instead of swapping to backchannel. Otherwise, we lose the one check that verifies real users can actually reach Keycloak.

Alternative: make this configurable instead of picking one. Add a value (default true = current public-url behavior) so people who want the real e2e check keep it, and people hitting the DNS/ServiceEntry issue can opt into backchannel. Still need the CA trust fix for the public path though, otherwise the flag is pointless for anyone who leaves it on.

[Copilot]

Pull request overview

This PR fixes oauth2-proxy startup failures caused by the wait-for-keycloak init container polling Keycloak via the public issuer URL from a sidecar-less pod (where ServiceEntry DNS and custom CA trust can break the curl check). It switches the poll target to the existing backchannel in-cluster URL so the init container can reliably reach Keycloak and unblock oauth2-proxy.

Changes:

• Update wait-for-keycloak init container to poll _derived.oidcBaseUrlBackchannel instead of _derived.oidcBaseUrl.
• Add inline template comments documenting why the backchannel URL is required for this sidecar-less init container.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.