diff --git a/docs/superpowers/plans/2026-09-28-keycloak-prod-lb-health-check.md b/docs/superpowers/plans/2026-09-28-keycloak-prod-lb-health-check.md new file mode 100644 index 00000000..1a4bb316 --- /dev/null +++ b/docs/superpowers/plans/2026-09-28-keycloak-prod-lb-health-check.md @@ -0,0 +1,369 @@ +# Keycloak prod: faster load balancer health check, graceful pod swap, credentials in a Secret + +Cluster `cdip-prod01` (kube context `cdip-prod`), namespace `cdip-auth`, GCP project `cdip-78ca`. +gcloud account: `chrisdo@earthranger.com`. + +## Why + +Prod runs one Keycloak replica. The swap to `11.0.2-gundi-a4fe4ac` on 2026-09-28 had a short outage because: + +- The readiness probe and the load balancer health check both poll `/auth/realms/master` every 20 s. GKE built the load balancer check from the readiness probe. +- Kubernetes stopped the old pod as soon as the new one passed readiness. GKE marked the new pod's load-balancer + readiness true without waiting for a health check (`LoadBalancerNegWithoutHealthCheck` event), so the rollout did not wait for the load balancer to see the new pod as healthy. +- Connection draining is 0 and there is no preStop hook, so the old pod stopped serving before the load balancer deregistered it. + +Separately, the Deployment may carry credentials as plain env values. Moving them into a Secret keeps them out of the manifest, so the manifest can live in git. + +## Status (2026-09-29) + +Steps 1 to 4 are done, and step 5 is applied except for the readiness probe. A `kubectl diff --server-side -f manifests/` on 2026-09-29 showed the BackendConfig, Service and Deployment matching prod apart from the `last-applied-configuration` +annotation, which is harmless bookkeeping from a client-side apply. Prod runs image `11.0.2-gundi-438715d`, built from the newer main commits that centre the login card and fix the screenshot cleanup trap. + +**Remaining:** the readiness probe still polls every 20 s in both the manifest and the cluster. Step 5 edits were +made without it. Do the probe change in **step 5b**, then step 6. + +| Setting | Original (2026-09-28) | Target | Done | +|---|---|---|---| +| Load balancer check interval | 20 s | 5 s | yes | +| Load balancer check timeout | 4 s | 3 s | yes | +| Connection draining | 0 s | 30 s | yes | +| Readiness probe period | 20 s | 5 s | **no, still 20 s** | +| preStop hook | none | `sleep 20` | yes | +| terminationGracePeriodSeconds | 30 | 60 | yes | +| Credential env vars | plain `value:` | `secretKeyRef` to Secret `keycloak-credentials` | yes | + +Liveness probe stays as it is (120 s initial delay, 20 s period). + +Load balancer objects: backend service and health check are both named +`k8s1-29c9a6d8-cdip-auth-keycloak-8080-218e32e7`. + +## Approach + +Export the live Deployment and Service to YAML with `kubectl neat`, edit those files, check the change with +`kubectl diff`, and apply with server-side apply. The files become the record of what prod runs. + +Server-side apply is used because these objects were never managed with client-side `kubectl apply` (the Service's `last-applied-configuration` annotation is empty). It tracks field ownership on the server, so fields left out of a file are not removed, and changing a field another tool last set shows up as a conflict instead of passing silently. + +The Secret is created with `kubectl create` from the live values and is never written to `manifests/`. Only the references to it go in the Deployment file. + +## Before you start + +- **Timing.** Steps 3 and 4 need no restart. Steps 5 and 5b restart the pod, so do them in the 02:00 UTC window or another agreed off-hours slot. +- **Tools.** `kubectl neat` (`kubectl krew install neat`), `yq` (v4), and gcloud with the account above. +- **Old manifests.** The only Keycloak manifests found locally (`padas/keycloak.yaml`, + `cdip/.worktrees/main/keycloak.yaml`) are from 2021, use `jboss/keycloak:11.0.2`, and do not match the live Deployment. Do not apply them. The files exported below replace them. +- **Credential handling.** Commands below never print a credential. Don't `cat` the export or the backup, and don't paste either into chat or a ticket, until step 5 is done. + +## Step 1: back up and export + +Raw backups keep everything the server holds, for rollback and comparison. The backup holds credentials, so the +directory is private: + +```bash +WORK=~/keycloak-prod-2026-09-28 +mkdir -p -m 700 "$WORK" && mkdir -p "$WORK/backup" "$WORK/manifests" && cd "$WORK" + +kubectl --context cdip-prod -n cdip-auth get deploy keycloak -o yaml > backup/deploy-keycloak.yaml +kubectl --context cdip-prod -n cdip-auth get svc keycloak -o yaml > backup/svc-keycloak.yaml +kubectl --context cdip-prod -n cdip-auth rollout history deploy/keycloak | tail -3 > backup/rollout-history.txt +gcloud compute health-checks describe k8s1-29c9a6d8-cdip-auth-keycloak-8080-218e32e7 \ + --account=chrisdo@earthranger.com --project=cdip-78ca > backup/health-check.yaml +``` + +Clean exports to edit: + +```bash +kubectl --context cdip-prod -n cdip-auth get deploy keycloak -o yaml | kubectl neat > manifests/deployment.yaml +kubectl --context cdip-prod -n cdip-auth get svc keycloak -o yaml | kubectl neat > manifests/service.yaml +``` + +## Step 2: review the exports + +**Credentials.** List the env vars in the `keycloak` container that are set as plain values, without printing the +values themselves: + +```bash +yq '.spec.template.spec.containers[] | select(.name == "keycloak") | .env[] + | select(has("value")) | .name' manifests/deployment.yaml +``` + +Write down the names in that list that are credentials. From the 2021 manifest, expect `KEYCLOAK_USER`, +`KEYCLOAK_PASSWORD`, `DB_USER` and `DB_PASSWORD`. `DB_ADDR`, `DB_DATABASE`, `DB_SCHEMA`, `DB_VENDOR` and +`PROXY_ADDRESS_FORWARDING` are configuration and stay as plain values. If no credential is in the list, skip step 4 and the env edit in step 5. + +**Fields the cluster owns.** Delete these if present, so the apply does not take ownership of them: + +- `service.yaml`: `spec.clusterIP`, `spec.clusterIPs`, and the annotations `cloud.google.com/neg-status`, + `networking.gke.io/target-pool` and `kubectl.kubernetes.io/last-applied-configuration`. +- `deployment.yaml`: the annotations `deployment.kubernetes.io/revision` and + `kubectl.kubernetes.io/last-applied-configuration`. + +Keep `cloud.google.com/neg: '{"ingress":true}'` on the Service. The ingress depends on it. + +**Baseline.** Before editing, the diff must be empty. Anything it shows is a field the export got wrong: + +```bash +kubectl --context cdip-prod diff --server-side -f manifests/deployment.yaml -f manifests/service.yaml +``` + +## Step 3: faster load balancer check and connection draining (no restart) + +Create `manifests/backendconfig.yaml`: + +```yaml +apiVersion: cloud.google.com/v1 +kind: BackendConfig +metadata: + name: keycloak + namespace: cdip-auth +spec: + healthCheck: + type: HTTP + requestPath: /auth/realms/master + port: 8080 + checkIntervalSec: 5 + timeoutSec: 3 + healthyThreshold: 1 + unhealthyThreshold: 2 + connectionDraining: + drainingTimeoutSec: 30 +``` + +In `manifests/service.yaml`, add under `metadata.annotations`: + +```yaml + cloud.google.com/backend-config: '{"default":"keycloak"}' +``` + +Diff, then apply. The diff should show the new BackendConfig and that one annotation, nothing else: + +```bash +kubectl --context cdip-prod diff --server-side -f manifests/backendconfig.yaml -f manifests/service.yaml +kubectl --context cdip-prod apply --server-side -f manifests/backendconfig.yaml -f manifests/service.yaml +``` + +The ingress controller reconciles within a few minutes. Verify: + +```bash +gcloud compute health-checks describe k8s1-29c9a6d8-cdip-auth-keycloak-8080-218e32e7 \ + --account=chrisdo@earthranger.com --project=cdip-78ca \ + --format="value(checkIntervalSec,timeoutSec,httpHealthCheck.requestPath)" +# expect: 5 3 /auth/realms/master + +gcloud compute backend-services describe k8s1-29c9a6d8-cdip-auth-keycloak-8080-218e32e7 --global \ + --account=chrisdo@earthranger.com --project=cdip-78ca \ + --format="value(connectionDraining.drainingTimeoutSec)" +# expect: 30 + +kubectl --context cdip-prod -n cdip-auth get ingress keycloak \ + -o jsonpath='{.metadata.annotations.ingress\.kubernetes\.io/backends}{"\n"}' +# expect every backend HEALTHY +``` + +If the backend is not HEALTHY after 5 minutes, roll back step 3 (below) before going further. + +## Step 4: create the credentials Secret (no restart) + +Set `VARS` to the credential names you wrote down in step 2. The loop writes each current value to a private temp file, the Secret is created from those files with the env var names as keys, and the files are removed. Nothing is printed and nothing lands in shell history: + +```bash +VARS="KEYCLOAK_USER KEYCLOAK_PASSWORD DB_USER DB_PASSWORD" + +tmp=$(mktemp -d) && chmod 700 "$tmp" +for v in $VARS; do + printf '%s' "$(V="$v" yq '.spec.template.spec.containers[] | select(.name == "keycloak") | .env[] + | select(.name == strenv(V)) | .value' manifests/deployment.yaml)" > "$tmp/$v" + [ -s "$tmp/$v" ] || echo "EMPTY: $v" +done +kubectl --context cdip-prod -n cdip-auth create secret generic keycloak-credentials --from-file="$tmp" +rm -rf "$tmp" +``` + +Stop if any line printed `EMPTY:`. Delete the Secret, fix `VARS`, and run it again. + +`printf '%s'` drops the newline yq adds, so the Secret holds exactly what the env var holds today. + +Verify the keys, not the values. `describe` shows each key with its size and never the value: + +```bash +kubectl --context cdip-prod -n cdip-auth describe secret keycloak-credentials +# one key per name in VARS, each with a non-zero byte count +``` + +The running pod is unaffected until step 5. + +## Step 5: readiness probe, graceful shutdown, credential references (restarts the pod) + +Done on 2026-09-29, except the readiness probe, which is now step 5b. Kept for the record. + +In `manifests/deployment.yaml`, under `spec.template.spec`: + +- set `terminationGracePeriodSeconds: 60` +- in the `keycloak` container's `readinessProbe`, set `periodSeconds: 5` and leave the other probe fields as they are +- add to the `keycloak` container: + +```yaml + lifecycle: + preStop: + exec: + command: ["sleep", "20"] +``` + +- for each name in `VARS`, replace its `value:` line with a reference, keeping the entry in its current position. For example: + +```yaml + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: keycloak-credentials + key: DB_PASSWORD +``` + +Replace, don't add: an entry with both `value:` and `valueFrom:` is rejected by the API server with +`valueFrom: Invalid value: "": may not be specified when value is not empty`. Confirm no credential value is left in the file. This should print nothing: + +```bash +for v in $VARS; do + V="$v" yq '.spec.template.spec.containers[] | select(.name == "keycloak") | .env[] + | select(.name == strenv(V)) | select(has("value")) | .name' manifests/deployment.yaml +done +``` + +Diff. It should show the grace period, probe period and lifecycle changes, and each credential's `value` replaced by a `valueFrom`. The diff masks nothing in a Deployment, so the removed lines contain the old values: read it yourself, don't share it. + +```bash +kubectl --context cdip-prod diff --server-side -f manifests/deployment.yaml +``` + +Start a downtime monitor in a second terminal: + +```bash +while true; do + printf '%s %s\n' "$(date -u +%T)" \ + "$(curl -s -o /dev/null -m 3 -w '%{http_code}' https://cdip-auth.pamdas.org/auth/realms/master)" + sleep 1 +done | tee ~/keycloak-prod-2026-09-28/swap-monitor.log +``` + +Apply and wait: + +```bash +kubectl --context cdip-prod apply --server-side -f manifests/deployment.yaml +kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m +``` + +If the apply reports conflicts, read which fields it names. Conflicts on only the grace period, readiness probe +period, lifecycle or the credential env entries are expected, because those were last set by another tool. Rerun with `--force-conflicts`. A conflict on any other field means the file differs from prod: stop and compare it with backup/deploy-keycloak.yaml`. + +If a secret reference is wrong, the new pod fails with `CreateContainerConfigError` or Keycloak cannot reach the database, and never becomes ready. With one replica the rollout keeps the old pod serving in that case, so +`rollout status` times out instead of causing an outage. Roll back step 5 (below). + +This rollout's old pod has no preStop hook yet, so it can still blink. The next swap is the first one that gets +the full benefit. + +Verify: + +```bash +kubectl --context cdip-prod -n cdip-auth get deploy keycloak -o jsonpath=\ +'image={.spec.template.spec.containers[0].image}{"\n"}readiness={.spec.template.spec.containers[0].readinessProbe}{"\n"}preStop={.spec.template.spec.containers[0].lifecycle.preStop}{"\n"}grace={.spec.template.spec.terminationGracePeriodSeconds}{"\n"}' +# image must still be the Gundi image that was running before the apply (11.0.2-gundi-438715d as of 2026-09-29) + +kubectl --context cdip-prod -n cdip-auth logs deploy/keycloak --since=10m | grep -E 'WFLYSRV0025|ERROR' | tail -5 +# expect the "Keycloak 11.0.2 ... started" line and no database errors + +curl -s -o /dev/null -w '%{http_code}\n' https://cdip-auth.pamdas.org/auth/realms/master +# expect 200 + +grep -v ' 200$' ~/keycloak-prod-2026-09-28/swap-monitor.log +# lists every second the endpoint was not answering 200 + +kubectl --context cdip-prod diff --server-side -f manifests/ +# expect no output: the files now match prod +``` + +Stop the monitor. Log in through the prod portal once, and once to the admin console at +`https://cdip-auth.pamdas.org/auth/admin/`. + +## Step 5b: readiness probe period and theme cache lifetime (restarts the pod) + +Two edits, one restart. Both are already made in `manifests/deployment.yaml` as of 2026-09-30. Confirm before applying, with no values printed: + +```bash +yq '.spec.template.spec.containers[] | select(.name == "keycloak") | .readinessProbe.periodSeconds' manifests/deployment.yaml +# expect 5 +yq '.spec.template.spec.containers[] | select(.name == "keycloak") | .env[] | select(.name == "JAVA_OPTS") | .value' manifests/deployment.yaml +# expect the existing flags followed by -Dkeycloak.theme.staticMaxAge=3600 +``` + +- **Readiness probe:** in the `keycloak` container's `readinessProbe`, `periodSeconds: 5`. `failureThreshold`, `httpGet`, `successThreshold` and `timeoutSeconds` stay as they are. +- **Theme cache:** `-Dkeycloak.theme.staticMaxAge=3600` appended to `JAVA_OPTS`. Keycloak 11 serves theme files with `Cache-Control: max-age=2592000` (30 days) and no ETag, and the `/auth/resources//` URL token changes only with the Keycloak version, so browsers keep old CSS for up to a month after a theme change. One hour bounds that. Verified on prod 2026-09-30: `curl -sI` on `login/gundi/css/gundi.css` returned `cache-control: max-age=2592000`. + +Diff. It should show exactly two changed lines: `periodSeconds: 20` to `5` inside `readinessProbe`, and the `JAVA_OPTS` value gaining the new flag. A change to the `last-applied-configuration` annotation may appear as well and is fine. Anything else means the file has drifted from prod: stop and re-export. + +```bash +kubectl --context cdip-prod diff --server-side -f manifests/deployment.yaml +``` + +This changes the pod template, so it restarts Keycloak. Start the downtime monitor from step 5 in a second terminal, then apply and wait: + +```bash +kubectl --context cdip-prod apply --server-side -f manifests/deployment.yaml +kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m +``` + +Conflicts on `readinessProbe.periodSeconds` and the `JAVA_OPTS` env entry are expected; rerun with `--force-conflicts`. Any other conflict means stop. + +This is the first swap where the old pod has the preStop hook and the load balancer check is 5 s, so the monitor log is the first real measurement of the fix. Verify with the same commands as step 5. Expect the probe to show `"periodSeconds":5`, the image unchanged, and an empty diff for `manifests/`. Note how many non-200 seconds the monitor logged; on 2026-09-28 the swap without these changes lost roughly 20 to 30 seconds. + +Check the new cache lifetime once the pod is up: + +```bash +curl -s 'https://cdip-auth.pamdas.org/auth/realms/cdip-dev/protocol/openid-connect/auth?client_id=cdip-admin-portal&response_type=code&scope=openid&redirect_uri=https%3A%2F%2Fexample.invalid%2F' \ + | grep -o '/auth/resources/[^"]*login/gundi/css/gundi.css' | head -1 \ + | xargs -I{} curl -sI 'https://cdip-auth.pamdas.org{}' | grep -i cache-control +# expect: cache-control: max-age=3600 +``` + +Users who loaded the login page before this change still hold the 30-day copy of the old CSS; the shorter lifetime only applies to files fetched from now on. + +## Step 6: keep the manifests, remove the backups + +`manifests/` now describes what prod runs and holds no credentials. It is committed as `keycloak/prod/` in the cdip repo (branch `docs/keycloak-prod-lb-health-check`), with a README describing the diff-then-apply workflow. The runbook's image-swap step now edits `keycloak/prod/deployment.yaml` instead of using `kubectl set image`, so the files stop drifting. Delete `~/keycloak-prod-2026-09-28/manifests/` once the branch is merged, and work from the repo copy. + +Once prod has run cleanly for a day, delete the raw backup, which still holds the old plain values: + +```bash +rm -rf ~/keycloak-prod-2026-09-28/backup +``` + +The Secret is not in git. It lives only in the cluster. To rotate a credential, update the Secret and restart the +pod with `kubectl rollout restart deploy/keycloak`. For `DB_PASSWORD`, change it in Cloud SQL first. +`KEYCLOAK_USER` and `KEYCLOAK_PASSWORD` only create the admin user on an empty database, so changing them does not change the existing admin password. + +## Rollback + +**Step 5b:** `rollout undo` returns to the revision with the 20 s probe, the old `JAVA_OPTS` and everything else from step 5 intact. Then set `periodSeconds` back to 20 and remove the `staticMaxAge` flag in `manifests/deployment.yaml`. + +**Step 5:** return to the previous revision, which is the Gundi image with the old probe, no preStop hook and the plain env values. Then revert the same edits in `manifests/deployment.yaml` so the file matches prod again. Leave the Secret in place; nothing reads it after the rollback. + +```bash +kubectl --context cdip-prod -n cdip-auth rollout undo deploy/keycloak +kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m +``` + +**Step 4:** nothing reads the Secret until step 5, so deleting it is safe as long as step 5 is not applied. + +```bash +kubectl --context cdip-prod -n cdip-auth delete secret keycloak-credentials +``` + +**Step 3:** remove the annotation from `manifests/service.yaml` and re-apply it, which drops the annotation because this apply owns it. Then delete the BackendConfig. GKE re-infers the health check from the readiness probe. + +```bash +kubectl --context cdip-prod apply --server-side -f manifests/service.yaml +kubectl --context cdip-prod delete -f manifests/backendconfig.yaml +``` + +## Not in scope + +A second replica would make swaps seamless, but Keycloak 11 needs cluster-aware session caching for that. It fits better with the Keycloak 26 upgrade. diff --git a/keycloak/RUNBOOK.md b/keycloak/RUNBOOK.md index b6f90cd0..2edc478a 100644 --- a/keycloak/RUNBOOK.md +++ b/keycloak/RUNBOOK.md @@ -26,30 +26,40 @@ generated by `keycloak/dev/generate-realm.py`; rerun it when the base realm expo ## Prod rollout — Keycloak 11 (`cdip-prod01`, namespace `cdip-auth`) -Prod Keycloak is a single replica applied from a raw manifest, not ArgoCD. Order matters: image first, then realms. +Prod Keycloak is a single replica, not managed by ArgoCD. Its manifests live in `keycloak/prod/` (see the README +there) and are applied with server-side apply. Order matters: image first, then realms. -### 1. Confirm the base image and pick the tag +### 1. Confirm the files match prod and pick the tag ```bash -kubectl --context cdip-prod -n cdip-auth get deploy keycloak \ - -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' +kubectl --context cdip-prod diff --server-side -f keycloak/prod/ ``` -Must print `quay.io/keycloak/keycloak:11.0.2`. If it prints anything else, stop: `keycloak/Dockerfile.kc11` must be -rebased on the running image before continuing. +Must print nothing. A diff means prod was changed by hand since the last commit: reconcile `keycloak/prod/` first +(README, "Always diff first"). + +The running image's version prefix must match the `FROM` tag in `keycloak/Dockerfile.kc11` (currently `11.0.2`). +If the base was bumped in prod without a Dockerfile change, stop and rebase the Dockerfile first. Pick the tag from the latest successful `Keycloak Gundi theme images` run on `main`: `europe-west3-docker.pkg.dev/serca-artifact-registry/gundi/keycloak:11.0.2-gundi-`. ### 2. Swap the image (restart; do this in the 02:00 UTC window or an agreed off-hours slot) +Edit the `image:` line in `keycloak/prod/deployment.yaml` to the new tag, then: + ```bash -TAG=11.0.2-gundi- -kubectl --context cdip-prod -n cdip-auth set image deploy/keycloak \ - keycloak=europe-west3-docker.pkg.dev/serca-artifact-registry/gundi/keycloak:$TAG +kubectl --context cdip-prod diff --server-side -f keycloak/prod/deployment.yaml # exactly one changed line: image +kubectl --context cdip-prod apply --server-side -f keycloak/prod/deployment.yaml kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m ``` +Commit the one-line change. If the apply reports a conflict on `image` only, rerun with `--force-conflicts`; any +other conflict means the file has drifted from prod, so stop. + +The swap restarts Keycloak (about 30 s to boot). The readiness probe, preStop hook and load balancer health check in +`keycloak/prod/` keep the gap to a few seconds; expect a blip, not an outage. + If the pod reports `ImagePullBackOff`, the node service account lacks read access to the `gundi` Artifact Registry repository; the portal in the same cluster pulls from `gundi/admin-portal`, so compare that Deployment's `imagePullSecrets`. Roll back (below) rather than leaving the Deployment in that state. @@ -61,9 +71,9 @@ curl -s -o /dev/null -w '%{http_code}\n' \ 'https://cdip-auth.pamdas.org/auth/realms/cdip-dev/protocol/openid-connect/auth?client_id=cdip-admin-portal&response_type=code&scope=openid&redirect_uri=https%3A%2F%2Fexample.invalid%2F' ``` -Expected `400`. Keycloak answers a bad `redirect_uri` with its "We are sorry" error page and HTTP 400 on both -11 and 26; the rendered error page is what proves the pod is up. A `5xx`, a connection error, or an empty -response means the swap broke something: roll back. +Expected `200` or `400`. A client that restricts redirect URIs gets Keycloak's "We are sorry" error page (400); +`cdip-admin-portal` in `cdip-dev` currently accepts any redirect, so it returns the login page (200). Either proves +the pod is up. A `5xx`, a connection error, or an empty response means the swap broke something: roll back. ### 3. Stage on the `cdip-dev` realm @@ -101,6 +111,8 @@ kubectl --context cdip-prod -n cdip-auth rollout undo deploy/keycloak kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m ``` +Then put the previous tag back in `keycloak/prod/deployment.yaml` and commit, so the file matches prod again. + ## Keycloak 26 upgrade hand-off The realm `loginTheme` value lives in the database, so after the upgrade copies the auth DB both realms already point at diff --git a/keycloak/prod/README.md b/keycloak/prod/README.md new file mode 100644 index 00000000..b5877576 --- /dev/null +++ b/keycloak/prod/README.md @@ -0,0 +1,50 @@ +# Prod Keycloak 11 manifests (`cdip-prod01`, namespace `cdip-auth`) + +These files are the record of what prod runs. Prod is not managed by ArgoCD; changes are made by editing a file +here, checking the diff, and applying with server-side apply. Rollout steps are in `../RUNBOOK.md`. + +| File | Object | Restart on change? | +|---|---|---| +| `deployment.yaml` | Deployment `keycloak` | Yes, if anything under `spec.template` changes | +| `service.yaml` | Service `keycloak` (GKE load balancer, NEG) | No | +| `backendconfig.yaml` | BackendConfig `keycloak` (load balancer health check, draining) | No | + +Not in git: Secret `keycloak-credentials` (`KEYCLOAK_USER`, `KEYCLOAK_PASSWORD`, `DB_USER`, `DB_PASSWORD`). The +Deployment references it by key. To rotate a value, update the Secret and run `kubectl rollout restart deploy/keycloak`. +`KEYCLOAK_USER` / `KEYCLOAK_PASSWORD` only seed the admin user on an empty database; changing them does not change +the existing admin password. + +## Workflow + +```bash +kubectl --context cdip-prod diff --server-side -f keycloak/prod/ # empty before you edit; only your edit after +kubectl --context cdip-prod apply --server-side -f keycloak/prod/ +kubectl --context cdip-prod -n cdip-auth rollout status deploy/keycloak --timeout=5m +``` + +Commit the edit in the same change as the apply, so the files never lag prod. + +- **Always diff first.** A non-empty diff before editing means someone changed prod by hand. Re-export with + `kubectl get ... -o yaml | kubectl neat`, reconcile, and commit that before making your change. +- **Conflicts.** Server-side apply refuses to change a field last set by another tool (`kubectl set image`, + `rollout restart`, a client-side apply). Read the field names it reports. If they are exactly the fields you + edited, rerun with `--force-conflicts`. Anything else means the file has drifted: stop and re-export. +- **Do not use `kubectl set image`, `kubectl edit` or `kubectl patch` on these objects.** They work, but the files + here stop matching prod and the next apply becomes a surprise. +- **`kubectl.kubernetes.io/restartedAt`** in the pod template is left over from a `rollout restart`. Leave it alone: + removing or changing it restarts the pod. +- **Fields the cluster owns** are deliberately absent (`status`, `resourceVersion`, `clusterIP`, the NEG status + annotation, the last-applied annotation). Do not add them back when re-exporting. + +## What the values are for + +- `readinessProbe.periodSeconds: 5`, `lifecycle.preStop` (`sleep 20`), `terminationGracePeriodSeconds: 60`, and the + BackendConfig's 5 s health check with 30 s draining together keep the load balancer pointed at a serving pod + during an image swap. Prod has one replica, so a swap still restarts Keycloak; these limit the gap to a few + seconds. Background: `docs/superpowers/plans/2026-09-28-keycloak-prod-lb-health-check.md`. +- `-Dkeycloak.theme.staticMaxAge=3600` in `JAVA_OPTS` caps browser caching of theme files at one hour. Keycloak's + default is 30 days with no ETag, and the resource URLs only change with the Keycloak version, so without this a + theme change can take a month to reach returning users. +- `-Dkeycloak.profile.feature.upload_scripts=enabled` was already set before these files existed. Keep it unless + you have confirmed nothing depends on it. +- `DB_ADDR` is the Cloud SQL instance's private IP. It is internal-only, which is why it is acceptable in git. diff --git a/keycloak/prod/backendconfig.yaml b/keycloak/prod/backendconfig.yaml new file mode 100644 index 00000000..d6659699 --- /dev/null +++ b/keycloak/prod/backendconfig.yaml @@ -0,0 +1,16 @@ +apiVersion: cloud.google.com/v1 +kind: BackendConfig +metadata: + name: keycloak + namespace: cdip-auth +spec: + healthCheck: + type: HTTP + requestPath: /auth/realms/master + port: 8080 + checkIntervalSec: 5 + timeoutSec: 3 + healthyThreshold: 1 + unhealthyThreshold: 2 + connectionDraining: + drainingTimeoutSec: 30 diff --git a/keycloak/prod/deployment.yaml b/keycloak/prod/deployment.yaml new file mode 100644 index 00000000..23793dc9 --- /dev/null +++ b/keycloak/prod/deployment.yaml @@ -0,0 +1,122 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + app: keycloak + name: keycloak + namespace: cdip-auth +spec: + progressDeadlineSeconds: 600 + replicas: 1 + revisionHistoryLimit: 10 + selector: + matchLabels: + app: keycloak + strategy: + rollingUpdate: + maxSurge: 25% + maxUnavailable: 25% + type: RollingUpdate + template: + metadata: + annotations: + kubectl.kubernetes.io/restartedAt: "2026-09-16T02:08:20Z" + labels: + app: keycloak + spec: + affinity: + nodeAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - preference: + matchExpressions: + - key: workload + operator: In + values: + - auth + weight: 100 + containers: + - env: + - name: KEYCLOAK_USER + valueFrom: + secretKeyRef: + name: keycloak-credentials + key: KEYCLOAK_USER + - name: KEYCLOAK_PASSWORD + valueFrom: + secretKeyRef: + name: keycloak-credentials + key: KEYCLOAK_PASSWORD + - name: PROXY_ADDRESS_FORWARDING + value: "true" + - name: DB_VENDOR + value: POSTGRES + - name: DB_ADDR + value: 172.22.160.3 + - name: DB_DATABASE + value: cdip_auth_db + - name: DB_USER + valueFrom: + secretKeyRef: + name: keycloak-credentials + key: DB_USER + - name: DB_SCHEMA + value: public + - name: DB_PASSWORD + valueFrom: + secretKeyRef: + name: keycloak-credentials + key: DB_PASSWORD + - name: JAVA_OPTS + value: -Xms1g -Xmx3g -XX:MaxMetaspaceSize=256m -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError + -Dkeycloak.profile.feature.upload_scripts=enabled -Dkeycloak.theme.staticMaxAge=3600 + image: europe-west3-docker.pkg.dev/serca-artifact-registry/gundi/keycloak:11.0.2-gundi-438715d + imagePullPolicy: IfNotPresent + lifecycle: + preStop: + exec: + command: ["sleep", "20"] + livenessProbe: + failureThreshold: 3 + httpGet: + path: /auth/realms/master + port: 8080 + scheme: HTTP + initialDelaySeconds: 120 + periodSeconds: 20 + successThreshold: 1 + timeoutSeconds: 5 + name: keycloak + ports: + - containerPort: 8080 + name: http + protocol: TCP + - containerPort: 8443 + name: https + protocol: TCP + readinessProbe: + failureThreshold: 3 + httpGet: + path: /auth/realms/master + port: 8080 + scheme: HTTP + periodSeconds: 5 + successThreshold: 1 + timeoutSeconds: 4 + resources: + limits: + cpu: "2" + memory: 4Gi + requests: + cpu: "2" + memory: 4Gi + terminationMessagePath: /dev/termination-log + terminationMessagePolicy: File + dnsPolicy: ClusterFirst + restartPolicy: Always + schedulerName: default-scheduler + terminationGracePeriodSeconds: 60 + tolerations: + - effect: NoSchedule + key: dedicated + operator: Equal + value: auth diff --git a/keycloak/prod/service.yaml b/keycloak/prod/service.yaml new file mode 100644 index 00000000..c11e701a --- /dev/null +++ b/keycloak/prod/service.yaml @@ -0,0 +1,22 @@ +apiVersion: v1 +kind: Service +metadata: + annotations: + cloud.google.com/neg: '{"ingress":true}' + cloud.google.com/backend-config: '{"default": "keycloak"}' + labels: + app: keycloak + app.version: "1" + name: keycloak + namespace: cdip-auth +spec: + ipFamilies: + - IPv4 + ipFamilyPolicy: SingleStack + ports: + - name: http + nodePort: 32431 + port: 8080 + selector: + app: keycloak + type: LoadBalancer