GitOps reference

A small, complete GitOps setup you can run on a laptop in one command: Flux reconciles two services from Git, in dependency order, onto restricted workloads; image automation picks the newest main-<ts>-<sha> build of each; and the observability stack is code too — dashboards and alerts checked against the services' telemetry contracts.

The service is the library API from spec-driven-ddd-python, published to ghcr.io/hasanozkan/library-sample on every merge.

flowchart LR
  subgraph app["application repo"]
    M[merge to main] --> CI[CI: build + push<br/>main-&lt;ts&gt;-&lt;sha&gt;]
  end
  CI --> REG[(ghcr.io)]
  subgraph cluster["cluster (Flux)"]
    IR[ImageRepository] --> IP[ImagePolicy<br/>newest by ts]
    IP --> IUA[ImageUpdateAutomation]
    GR[GitRepository] --> K1[Kustomization<br/>infrastructure] --> K2[Kustomization<br/>apps]
  end
  REG --> IR
  IUA -- "commit: new tag" --> G[(this repo)]
  G --> GR
  K2 --> D[Deployment library]

Loading

Run it

Needs Docker, k3d, the flux CLI and kubectl.

make up        # k3d cluster + Flux, reconciled from this repo's main branch
make smoke     # borrow a book through the deployed API, check the catalog follows
make status    # flux get all -A
kubectl -n library scale deploy/library --replicas=0   # drift...
flux reconcile kustomization apps                      # ...corrected from Git
make down

make up deploys and resolves the image policies (flux get images policy shows the newest tag of each) but writes nothing back; tags in apps/base/*/deployment.yaml are bumped by commit. The write-back half — Flux committing chore(image): … -> … itself — is configured in clusters/local/image-update.yaml and enabled with GITOPS_REPO=<your fork> GITHUB_TOKEN=<contents:write> make up, but it is not exercised by CI or by the runs behind this README.

Observability

make up also installs a laptop-sized stack — Prometheus, Alertmanager, Grafana, Loki (logs via Alloy) and Tempo (traces) — and make traffic gives it something to show. make grafana opens two dashboards that live in observability/dashboards/ as JSON:

• Library — request rate and p95 by route, 5xx ratio, loans opened and closed, late fees, refusals by code, and the pods' logs from Loki.
• Assistant — tokens and p95 latency by model, spend, suspicious proposals and injection flags, tool calls by outcome, escalations, fallbacks, budget stops — the metrics from its telemetry contract.

Each app declares its ServiceMonitor and PrometheusRule. CI checks the whole thing without a cluster (ADR-0004): every series and label a panel or alert queries must exist in a service's telemetry contract (scripts/check_queries.py — a dashboard on a renamed metric fails the build, verified), the assistant's rules must equal its repository's (scripts/alerts_mirror.py --check), and every rule is unit-tested with promtool.

What running it found. The first real scrape showed LibrarySlowRoute firing for /healthz at a p95 of 4.75 s. The service was fine; the histogram was not: the SDK's default bucket boundaries are sized for milliseconds, so every request recorded in seconds landed in the first bucket. The advised boundaries now live in the telemetry contracts, and the services' tests compare them — p95 reads ~5 ms. Unit tests could not have caught it; an alert evaluated on real data did. The full story.

Layout

Path	What it holds
clusters/local/sync.yaml	The entry point: Git source → infrastructure → apps
clusters/local/image-policy.yaml	Registry scan + "newest main-<ts>-<sha>" policy
clusters/local/image-update.yaml	Write-back: new tags committed to Git
clusters/local/observability.yaml	The observability stack, as its own Kustomization (local only)
infrastructure/	Namespace with restricted Pod Security, default-deny network policy, Helm sources, Prometheus Operator CRDs
apps/base/	The library and the assistant: hardened Deployments (with $imagepolicy setters), Services, ServiceMonitors, PrometheusRules
observability/	kube-prometheus-stack, Loki, Tempo, Alloy (HelmReleases), dashboards as JSON, alert tests
apps/local/, apps/production/	Overlays: local ingress · production replicas, spread and disruption budget

What CI proves

Job	Proves
validate	Every overlay and the cluster entry point render, and pass strict schema validation (Kubernetes, Flux and Prometheus Operator CRDs)
observability	Dashboards and alerts query only contract series and labels; the assistant's alert mirror is current; every rule passes its promtool tests
e2e	On a fresh kind cluster, Flux reconciles this commit; infrastructure then apps become Ready; the service passes the smoke test through its Service; the image policy resolves a tag from the registry

Decisions

• ADR-0001 — A merge to main is a deploy, and the deploy is a commit
• ADR-0002 — Platform before apps, and Ready means serving
• ADR-0003 — Workloads are restricted by default

Deliberate simplifications

One cluster, one app, no database. In a production setup the same shape grows: secrets encrypted in Git (SOPS + age) or pulled from a secret store, a migration Job gated before each rollout, alerting on failed reconciliations, and more environments as more clusters/<name> entry points over the same apps/ overlays.

---

Part of a set: spec-driven-ddd-python · llm-tool-calling-assistant · ai-native-engineering. By Hasan Özkan · LinkedIn