OpenBao SSO breaks after every rebuild: management applies before ZITADEL exists, so oidc_client_id is the previous rebuild's

[Smana]

Symptom

OpenBao SSO fails after a platform rebuild. The browser gets an error; ZITADEL logs:

WARN Errors.App.NotFound (QUERY-wu6Ee)
  query.(*Queries).ActiveOIDCClientByID ... parent: "sql: no rows in result set"
WARN request error
  oidc_error: {"description":"Errors.App.NotFound","type":"invalid_request"} status_code:400

bao login -method=userpass username=admin still works, so the platform is reachable — only
the human SSO path is broken.

Root cause: an ordering inversion inside one deploy

aws/openbao/management sorts at position 6 of the run order. aws/eks/init — which
brings up the cluster, hence ZITADEL — is position 7.

When	What happens
early in the deploy	openbao/management reads the openbao-oidc secret (oidc.tf) and writes oidc_client_id into auth/oidc/config. At that moment the secret still holds the previous rebuild's value.
end of the deploy	eks/init stage-3 runs scripts/zitadel-oidc-clients.sh sync. ZITADEL's app is absent from the restored database, so a new app is registered, and the script writes the new client_id to the secret.
never	Nothing re-applies openbao/management. OpenBao keeps pointing at an app that does not exist.

So the stack that consumes the value runs before the step that produces it, and the deploy
exits 0 either way.

Evidence (aws-0, 2026-09-16)

Source	client_id
OpenBao auth/oidc/config	390138579965263549
Secrets Manager openbao-oidc AWSPREVIOUS (2026-09-10)	390138579965263549
Secrets Manager openbao-oidc AWSCURRENT (09:44:48, written by the sync)	390990815813446980

OpenBao held exactly the previous version. zitadel-oidc-clients.sh deliberately skips apps
that already exist (recreating would rotate a secret ZITADEL returns only once), so it is not
at fault — it correctly created a new app because the old one was gone.

Current workaround

Re-apply the management stack after the platform converges:

cd opentofu/aws/openbao/management
tofu apply -auto-approve -parallelism=1 -var-file=variables.tfvars

Plan is tightly scoped — 0 to add, 2 to change, 0 to destroy:
vault_jwt_auth_backend.oidc[0] (client id + secret) and
vault_jwt_auth_backend_role.oidc_default[0] (bound_audiences). Keep -parallelism=1
(OpenBao 2.6 write-concurrency deadlock).

Verify the flow, not just that the ids match:

AUTHURL=$(bao write -field=auth_url auth/oidc/oidc/auth_url role=default \
  redirect_uri="https://bao.priv.aws.ogenki.io:8200/ui/vault/auth/oidc/oidc/callback")
curl -sL -c /tmp/j -b /tmp/j -A "Mozilla/5.0 ... Chrome/131.0" -o /tmp/z.html "$AUTHURL"
grep -o '<title>[^<]*</title>' /tmp/z.html   # expect: Welcome Back!

Two testing traps: without a cookie jar ZITADEL returns
Errors.AuthRequest.UserAgentNotCorresponding and renders "An internal error occurred",
which looks identical to a real failure; and the endpoint is auth/<mount>/oidc/auth_url —
auth/oidc/oidc_auth_url returns unsupported path and reads like a broken mount.

Candidate solutions

A — re-apply openbao/management from eks/init stage 3, after the sync.
Smallest change using existing machinery; eks/init already orchestrates a cross-stack apply.
⚠️ It creates a second path that applies the management stack — the same shape as the
flux-operator bug just fixed in #2044, where one of two apply paths forgot a step. Would need
the same discipline.

B — have zitadel-oidc-clients.sh write OpenBao directly.
The script already holds client_id/client_secret at the moment they are created, and
openbao-adopt-jwt-mount.sh is precedent for a script writing OpenBao with the root token.
No ordering inversion and no second Terraform path. Cost: OpenBao's OIDC config gets two
writers (Terraform + script), though they converge — the next management apply reads the same
current secret and plans no diff.

C — split the OIDC config out of openbao/management into a step that runs after ZITADEL.
Structurally the honest fix: this is the only part of the management stack whose dependency
order differs from the rest, which is exactly what a separate stack expresses. Terraform stays
the single owner. Cost: a new stack per cloud, with its own backend/provider wiring, for two
resources.

D — stop the client_id from changing at all.
The churn only happens because ZITADEL's app is missing after the database restore. If the
CNPG restore reliably carried the app, the value would be stable. Worth understanding
regardless. ⚠️ On its own this makes the hazard rare rather than absent, which is arguably
worse — it stays silent until it fires.

E — detect it, whatever else is chosen.
A post-deploy check comparing OpenBao's oidc_client_id against the secret's current value,
failing loudly. Does not fix anything, but turns a silent breakage into a deploy-time error.
Cheap, and complements A/B/C.

Recommendation: B or C, plus E regardless. D is a separate question worth
answering on its own terms.

Related

• fix(opentofu): forget the flux-operator release in the job that creates it #2044 — the flux-operator forget, same family: a value produced by one path and consumed by
  another, with the deploy exiting 0 in between.
• ADR-0034 (ZITADEL OIDC for human operators), ADR-0033 (OpenBao lineage).