feat(k8s): Kubernetes manifests for the three runtimes

.github/workflows/k8s.yml

@@ -0,0 +1,166 @@
+name: Kubernetes
+
+# Proves the manifests in k8s/ actually work, rather than asserting it in a
+# README. Two jobs, deliberately separated by cost:
+#
+#   validate  — schema-checks every manifest. Seconds. Catches typos and wrong
+#               apiVersions without starting anything.
+#   e2e       — builds the image, creates a real cluster with kind, applies the
+#               CI overlay and drives it until the three runtimes are ready.
+#
+# The e2e job is what closes the gap declared in PR #25: until this exists, the
+# manifests were valid YAML that had never been applied to a cluster.
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "k8s/**"
+      - "Dockerfile"
+      - ".github/workflows/k8s.yml"
+  pull_request:
+    paths:
+      - "k8s/**"
+      - "Dockerfile"
+      - ".github/workflows/k8s.yml"
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  validate:
+    name: Manifest schemas
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install kustomize and kubeconform
+        run: |
+          curl -sSfL https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh | bash
+          sudo mv kustomize /usr/local/bin/
+          curl -sSfL -o kubeconform.tar.gz \
+            https://github.com/yannh/kubeconform/releases/download/v0.6.7/kubeconform-linux-amd64.tar.gz
+          tar xzf kubeconform.tar.gz kubeconform
+          sudo mv kubeconform /usr/local/bin/
+
+      - name: Build overlays
+        run: |
+          kustomize build k8s/base > /tmp/base.yaml
+          kustomize build k8s/overlays/ci > /tmp/ci.yaml
+          echo "--- base ---" && grep -c '^kind:' /tmp/base.yaml
+          echo "--- ci ---" && grep -c '^kind:' /tmp/ci.yaml
+
+      - name: Validate against Kubernetes schemas
+        run: |
+          kubeconform -strict -summary \
+            -kubernetes-version 1.31.0 \
+            /tmp/base.yaml /tmp/ci.yaml
+
+      # KEDA and Prometheus Operator objects are CRDs, so their schemas are not
+      # in the upstream Kubernetes catalogue. They are checked against the
+      # community CRD schema mirror instead; -ignore-missing-schemas would make
+      # this step pass by simply skipping them, which is worse than useless.
+      - name: Validate CRD-backed manifests
+        run: |
+          kubeconform -strict -summary \
+            -schema-location default \
+            -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
+            k8s/keda k8s/monitoring
+
+  e2e:
+    name: kind end-to-end
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1
+        with:
+          cluster_name: feedpulse
+          node_image: kindest/node:v1.31.0
+
+      - name: Build image
+        run: docker build -t feedpulse:ci .
+
+      # kind nodes have their own container runtime, so an image built on the
+      # runner is invisible to them until it is side-loaded. The manifests use
+      # imagePullPolicy: IfNotPresent, which is what makes this work without a
+      # registry.
+      - name: Load image into kind
+        run: kind load docker-image feedpulse:ci --name feedpulse
+
+      - name: Apply manifests
+        run: kubectl apply -k k8s/overlays/ci
+
+      - name: Wait for dependencies
+        run: |
+          kubectl -n feedpulse rollout status deploy/postgres --timeout=180s
+          kubectl -n feedpulse rollout status deploy/redis --timeout=120s
+
+      # Migrations must land before the API can pass readiness: /ready resolves
+      # the base schema, so it fails by design until this Job completes.
+      - name: Run migrations
+        run: |
+          kubectl -n feedpulse wait --for=condition=complete job/feedpulse-migrate --timeout=240s \
+            || { kubectl -n feedpulse logs job/feedpulse-migrate --tail=100; exit 1; }
+
+      - name: Wait for the three runtimes
+        run: |
+          kubectl -n feedpulse rollout status deploy/feedpulse-api --timeout=240s
+          kubectl -n feedpulse rollout status deploy/feedpulse-worker --timeout=240s
+          kubectl -n feedpulse rollout status deploy/feedpulse-scheduler --timeout=240s
+
+      # Asserts the probes mean what k8s/README.md claims they mean. A green
+      # rollout only proves readiness passed; this proves both endpoints answer
+      # and are distinct.
+      - name: Verify liveness and readiness
+        run: |
+          kubectl -n feedpulse port-forward svc/feedpulse-api 8080:80 &
+          sleep 5
+          curl -fsS http://127.0.0.1:8080/health && echo " <- /health OK"
+          curl -fsS http://127.0.0.1:8080/ready  && echo " <- /ready OK"
+
+      # The rollout is where graceful shutdown is exercised: every pod receives
+      # SIGTERM and must drain within terminationGracePeriodSeconds. If
+      # SHUTDOWN_TIMEOUT_MS ever exceeds the grace period, this step is what
+      # catches it.
+      - name: Restart the worker and confirm it drains and returns
+        run: |
+          kubectl -n feedpulse rollout restart deploy/feedpulse-worker
+          kubectl -n feedpulse rollout status deploy/feedpulse-worker --timeout=180s
+
+      - name: Install KEDA and apply the ScaledObject
+        run: |
+          helm repo add kedacore https://kedacore.github.io/charts
+          helm repo update
+          helm install keda kedacore/keda --namespace keda --create-namespace --wait --timeout 5m
+          kubectl -n feedpulse apply -f k8s/keda/
+
+      # A ScaledObject that the admission webhook accepts can still be inert.
+      # READY=True means KEDA resolved the Redis trigger and created the
+      # underlying HPA, which is the actual claim being made in k8s/README.md.
+      - name: Confirm KEDA took ownership of the worker
+        run: |
+          kubectl -n feedpulse wait --for=condition=Ready scaledobject/feedpulse-worker --timeout=120s \
+            || { kubectl -n feedpulse describe scaledobject feedpulse-worker; exit 1; }
+          kubectl -n feedpulse get hpa
+
+      - name: Dump diagnostics on failure
+        if: failure()
+        run: |
+          kubectl -n feedpulse get pods -o wide
+          kubectl -n feedpulse get events --sort-by=.lastTimestamp | tail -40
+          kubectl -n feedpulse describe pods | tail -120
+          for d in feedpulse-api feedpulse-worker feedpulse-scheduler; do
+            echo "===== $d ====="
+            kubectl -n feedpulse logs deploy/$d --tail=80 --all-containers || true
+            kubectl -n feedpulse logs deploy/$d --tail=80 --previous || true
+          done

k8s/README.md

@@ -0,0 +1,110 @@
+# Kubernetes deployment
+
+Manifests for running FeedPulse's three runtimes on Kubernetes. Tracking issue:
+[#15](https://github.com/FullFran/feedpulse/issues/15).
+
+## Why Kubernetes, honestly
+
+At the volume this project currently runs, Kubernetes is **not** the cheapest or
+simplest option. The existing `compose.dokploy.yml` is. Anyone claiming
+otherwise is selling something.
+
+What Kubernetes buys here is specific and measurable:
+
+| Capability                              | Why Compose cannot do it                                     |
+| --------------------------------------- | ------------------------------------------------------------ |
+| Scale `worker` on **queue depth**       | Compose has no autoscaler, and no way to read BullMQ backlog |
+| Independent scaling per runtime         | `api`, `scheduler` and `worker` have unrelated load profiles |
+| Rolling updates with automatic rollback | Compose replaces containers; it does not gate on health      |
+| Declarative recovery                    | A node dying is rescheduling, not an incident                |
+
+The crossover point — the feed count above which this stops being overkill — is
+measured in [#24](https://github.com/FullFran/feedpulse/issues/24) using
+`npm run benchmark:stages:mvp`. The number belongs in the README, including the
+range where Kubernetes loses.
+
+## Layout
+
+```
+k8s/base/          Deployments, Service, ConfigMap, Ingress, migration Job
+k8s/keda/          ScaledObject: worker autoscaling on BullMQ queue depth
+k8s/monitoring/    ServiceMonitor / PodMonitor for kube-prometheus-stack
+```
+
+## Apply
+
+```bash
+kubectl create namespace feedpulse
+
+# Secrets come from a secret manager, never from the repository.
+# k8s/base/secret.example.yaml documents the required keys.
+
+kubectl apply -k k8s/base
+kubectl apply -f k8s/keda          # requires KEDA installed
+kubectl apply -f k8s/monitoring    # requires kube-prometheus-stack installed
+```
+
+## The decisions worth knowing
+
+**`scheduler` is a Deployment, not a CronJob.** It owns its own timing loop
+(`SCHEDULER_TICK_MS`, default 15s) and holds warm database and Redis
+connections. A CronJob would pay full process startup every tick for a 15-second
+interval. It runs with `strategy: Recreate` so two schedulers never tick at
+once; enqueue is deduplicated by job id (`feed-<id>`) so a brief overlap is
+survivable, but a visible gap is easier to reason about.
+
+**Liveness never touches a dependency.** `/health` is process-only; `/ready`
+resolves the base schema. Wiring a dependency check into liveness converts a
+Postgres blip into a cluster-wide CrashLoopBackOff, because every restarted pod
+comes back to the same unreachable database.
+
+**`terminationGracePeriodSeconds` must exceed `SHUTDOWN_TIMEOUT_MS`.** On
+SIGTERM the runtime stops accepting jobs and drains the in-flight one. If the
+kubelet SIGKILLs first, that job dies mid-flight and BullMQ only recovers it
+once the lock expires.
+
+**The worker Deployment declares no `replicas`.** KEDA owns that field. Setting
+it here would fight the ScaledObject on every apply.
+
+**Worker probes are TCP, and that is a known compromise.** The worker has no
+HTTP API, only its metrics server. A TCP check proves the process is up and its
+event loop accepts connections — it cannot prove the BullMQ connection is alive.
+Closing that gap needs a readiness endpoint reporting consumer state.
+
+**Migrations must be backward compatible.** During a RollingUpdate both versions
+serve at once. Expand-and-contract; never a destructive change in the release
+that also ships the code depending on it.
+
+## How this is verified
+
+`.github/workflows/k8s.yml` runs on every change under `k8s/`:
+
+| Job                  | What it proves                                                                                                                                                                                                                                                              |
+| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Manifest schemas** | `kubeconform -strict` over both overlays, plus the KEDA and Prometheus CRDs against the community catalogue. No `-ignore-missing-schemas`: a skipped schema is worse than no check                                                                                          |
+| **kind end-to-end**  | Builds the image, side-loads it into a real cluster, applies the CI overlay, runs migrations, waits for all three runtimes, curls `/health` and `/ready`, restarts the worker to exercise graceful shutdown, installs KEDA and waits for the ScaledObject to report `Ready` |
+
+The ScaledObject check is the one that matters: an object the admission webhook
+accepts can still be inert. `Ready=True` means KEDA resolved the Redis trigger
+and created the underlying HPA.
+
+### Two bugs this caught that reasoning did not
+
+**The KEDA trigger address was wrong twice.** It first used
+`addressFromEnv: REDIS_ADDRESS`, but the worker exposes `REDIS_URL` — a full
+URL, not a `host:port` pair. Then it used the short name `redis:6379`, which
+failed with `server misbehaving`: the connection is opened by the **KEDA
+operator**, which runs in the `keda` namespace, and short service names only
+resolve within the caller's own namespace. It needs the FQDN.
+
+**The CI overlay disabled `ENABLE_AUTH`.** `env.schema.ts` refuses to boot with
+auth disabled under `NODE_ENV=production`, because that resolves every request
+to the shared legacy tenant. The guard was right and the overlay was wrong; CI
+now runs with the same guarantee production has.
+
+## Not done yet
+
+- Terraform for the cluster itself ([#21](https://github.com/FullFran/feedpulse/issues/21))
+- CI/CD deploy with automatic rollback ([#22](https://github.com/FullFran/feedpulse/issues/22))
+- Grafana dashboards ([#20](https://github.com/FullFran/feedpulse/issues/20))
+- Worker readiness reflecting consumer state