diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 28bc2ce..238010b 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -13,6 +13,11 @@ permissions: contents: read jobs: + # Krateo Documentation Standard conformance (shared org workflow): the invariant + # docs/ file set, frontmatter, link integrity, banned dead-org strings, examples + # pairing. + lint-docs: + uses: krateo-platformops/.github/.github/workflows/lint-docs.yaml@main lint: runs-on: ubuntu-latest steps: diff --git a/README.md b/README.md index 5bdb6fe..f897fdc 100644 --- a/README.md +++ b/README.md @@ -1,41 +1,70 @@ -# krateo-clickstack-chart +# clickstack-chart -Krateo PlatformOps **observability** blueprint repo. Groups the ClickStack wrapper with the -collectors and event proxy that feed and surface it. All charts publish to the consolidated -registry `oci://ghcr.io/krateo-platformops/charts`. +The deployment unit for Krateo's observability stack: four Helm charts — the ClickStack +wrapper, two OpenTelemetry collectors, the SSE proxy — published to +`oci://ghcr.io/krateo-platformops/charts` and turned into Krateo compositions by the +[installer](https://github.com/krateo-platformops/installer). -Part of the [krateo-installer](https://github.com/krateo-platformops/installer) ecosystem. +## What is this -## Charts +A chart repo, not a code repo: it wraps the upstream ClickStack chart (ClickHouse + OTel +gateway + HyperDX + MongoDB) and the upstream `opentelemetry-collector` chart, folds in the +Krateo glue (the `/events` handler, the composition-id enrichment wiring, the portal-bell +SSE proxy), and ships a `values.schema.json` per chart so `core-provider` can generate typed +composition CRDs. The collector and sse-proxy **code** lives in +`krateo-platformops/otel-collector` and `krateo-platformops/sse-proxy`. +Full picture: [docs/index.md](docs/index.md). -| Path | Chart | OCI artifact | Purpose | -|------|-------|--------------|---------| -| `charts/krateo-observability` | `krateo-observability` | `oci://ghcr.io/krateo-platformops/charts/observability` | ClickStack (ClickHouse + OTel gateway + HyperDX + MongoDB) wrapper: `values.schema.json`, the ClickHouse http-handlers ConfigMap, the otel-clickhouse credentials Secret and a HyperDX LoadBalancer Service. The composition the installer uses (Kind `KrateoObservability`) | -| `charts/otel-collector-deployment` | `otel-collector-deployment` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment` | Cluster-level OTel collector that enriches K8s events with `krateo.io/composition-id` and exports to ClickHouse | -| `charts/otel-collector-daemonset` | `otel-collector-daemonset` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset` | Node-level OTel collector for pod logs, host and kubelet metrics | -| `charts/krateo-sse-proxy` | `krateo-sse-proxy` | `oci://ghcr.io/krateo-platformops/charts/sse-proxy` | Polls ClickHouse and pushes new K8s events to the portal via Server-Sent Events | +## Install -## How the installer consumes it - -The installer umbrella emits a `CompositionDefinition` per chart, pointing `core-provider` at the -OCI artifacts above; `core-provider` generates the typed CRDs and reconciles one Composition per -instance. The collectors depend on the `krateo-observability` composition (for the ClickHouse -credentials Secret), and `krateo-sse-proxy` is exposed so the portal's events bell can reach it. - -## Local validation +Normally installed by the **Krateo installer** (portal feature, `tier: observability`). +Standalone, in dependency order (wrapper first — it creates the credentials Secret the +collectors mount; ClickHouse + MongoDB operators must already be installed): ```sh -helm lint charts/krateo-observability -helm template smoke charts/krateo-observability +helm install krateo-observability oci://ghcr.io/krateo-platformops/charts/krateo-observability \ + --version 0.1.11 --namespace krateo-system --create-namespace +helm install otel-collector-deployment oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment \ + --version 0.3.3 --namespace krateo-system +helm install otel-collector-daemonset oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset \ + --version 0.1.5 --namespace krateo-system +helm install krateo-sse-proxy oci://ghcr.io/krateo-platformops/charts/krateo-sse-proxy \ + --version 0.1.6 --namespace krateo-system ``` -## Release +Details and the composition path: [docs/usage.md](docs/usage.md). + +## Configure + +See [docs/configuration.md](docs/configuration.md). Most used: + +| Setting | Default | Effect | +|---|---|---| +| `global.storageClassName` (wrapper) | `standard-rwo` | StorageClass for ClickHouse/Keeper/Mongo volumes (GKE default; override off-GKE). | +| `hyperdxLoadBalancer.enabled` (wrapper) | `true` | The extra LoadBalancer Service exposing the HyperDX UI on `:3000`. | +| `clickstack.clickhouse.cluster.spec.*` (wrapper) | 8Gi limit / 15Gi volume | The ONLY path upstream honours for ClickHouse resources/storage — top-level `clickhouse.*` keys are inert. | + +## Examples -Push a semver tag (`X.Y.Z`) — CI packages every chart under `charts/*` at its declared version -and publishes to `oci://ghcr.io/krateo-platformops/charts`. Charts that still carry the `CHART_VERSION` -placeholder (e.g. `clickstack`) track the tag; independently-pinned charts keep their own version. +- [examples/observability-composition](examples/observability-composition) — deploy the + ClickStack wrapper as a Krateo composition (CompositionDefinition + a `KrateoObservability` CR). -## Links +## Docs + +- [docs/index.md](docs/index.md) — the map (bundle + the `ops/` reference corpus) +- [docs/overview.md](docs/overview.md) — what the four charts deploy and how +- [docs/usage.md](docs/usage.md) — installer path, standalone install, local validation +- [docs/configuration.md](docs/configuration.md) — the whole per-chart values surface +- [docs/api.md](docs/api.md) — generated composition types, HTTP surfaces, upstream CRs +- [docs/examples.md](docs/examples.md) — examples index +- [docs/release.md](docs/release.md) — how a release ships +- [docs/log.md](docs/log.md) — curated history + +## Develop & release + +```sh +for d in charts/*/; do helm dependency build "$d"; helm lint "$d"; helm template smoke "$d" >/dev/null; done +``` -- Installer umbrella: https://github.com/krateo-platformops/installer -- ClickStack: https://github.com/ClickHouse/ClickStack-helm-charts +Push a plain-semver tag (`X.Y.Z`, no `v`) — CI publishes every chart at its own +literally-pinned `Chart.yaml` version. Runbook: [docs/release.md](docs/release.md). diff --git a/compositiondefinition.yaml b/compositiondefinition.yaml index 91012e8..192d88a 100644 --- a/compositiondefinition.yaml +++ b/compositiondefinition.yaml @@ -8,5 +8,5 @@ metadata: namespace: krateo-system spec: chart: - url: oci://ghcr.io/krateo-platformops/charts/observability - version: "0.1.8" \ No newline at end of file + url: oci://ghcr.io/krateo-platformops/charts/krateo-observability + version: "0.1.11" \ No newline at end of file diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..b1a77f3 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,90 @@ +--- +type: API +title: clickstack-chart — api +description: The contract — the four generated composition types and their CompositionDefinitions, the HTTP surfaces (/events, SSE, HyperDX API, OTLP), and the upstream CRs the charts deploy but do not own. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [api, compositiondefinition, crd, clickhouse] +timestamp: 2026-08-07T00:00:00Z +--- + +# API + +## No hand-authored CRDs + +**This repo ships NO CRD of its own.** There is no `crds-subchart/`, and none of its own +templates render a `CustomResourceDefinition` — only ConfigMaps, Secrets, Services, +Deployments and RBAC. The typed surface it exposes to the platform is **generated**: +`core-provider` reads each chart's `values.schema.json` and derives a composition CRD, so +the contract is edited by changing the chart values surface, never by editing CRD YAML. + +## The generated composition types + +The installer registers each chart as a composition (`tier: observability`, portal-gated): + +| Kind | Chart | OCI artifact | +|------|-------|--------------| +| `KrateoObservability` | `charts/krateo-observability` | `oci://ghcr.io/krateo-platformops/charts/krateo-observability` | +| `OtelCollectorDeployment` | `charts/otel-collector-deployment` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment` | +| `OtelCollectorDaemonset` | `charts/otel-collector-daemonset` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset` | +| `KrateoSseProxy` | `charts/krateo-sse-proxy` | `oci://ghcr.io/krateo-platformops/charts/krateo-sse-proxy` | + +All live in API group `composition.krateo.io`; the served version is derived from the +pinned chart version (`0.1.11` → `v0-1-11`), so **a chart-version bump changes the CR's +apiVersion**. The CR `spec` is the chart's values (typed by `values.schema.json`). The +deployed version is always readable from the cluster: +`CompositionDefinition.spec.chart.version`. + +The repo-root [`compositiondefinition.yaml`](../compositiondefinition.yaml) registers the +wrapper standalone (`core.krateo.io/v1alpha1`, name `krateo-observability`, namespace +`krateo-system`); in a real install the installer umbrella owns this and the per-collector +CompositionDefinitions. + +## HTTP surfaces the deployed stack exposes + +- **`GET /events?composition_id=`** on ClickHouse HTTP (port 8123, via + `krateo-clickstack-clickhouse-clickhouse-headless`): a `predefined_query_handler` + declared in the wrapper's `cluster.spec.settings.extraConfig.http_handlers`. Returns the + latest 50 K8s events for a composition UID as JSON (name/namespace/uid/kind, reason, + message, type, eventTime, source_component) from `otel_logs` rows where + `telemetry.source = 'k8s-events'`. This is the ONLY supported event-query path — the + catch-all `/` dynamic-query handler exists to preserve ClickHouse's native API, not for + portal consumers. `/ping`, `/replicas_status` and `/metrics` are re-declared alongside it. +- **The sse-proxy** (ClusterIP `:8080`, exposed by the installer for the browser): serves + the portal bell — the historical events snapshot and the `/notifications` SSE stream — + plus `/health` probes. Its in-cluster address is published as the fixed-name + **`sse-proxy-internal-endpoint` Secret** (`server-url`, `insecure: "true"`), which + snowplow RESTActions reference by exact name. The endpoint/stream contract is owned by the + code repo `krateo-platformops/sse-proxy`. +- **HyperDX**: the UI at the `krateo-clickstack-app-lb` LoadBalancer (`:3000`) and the + Bearer-authenticated external API (`/api/v2/{alerts,webhooks,dashboards,sources}`, + HyperDX ≥ 2.28) at the **`krateo-clickstack-api`** ClusterIP (`:8000`) — the direct + backend port, because the port-3000 proxy strips `/api` and 404s the v2 routes. +- **OTLP ingest**: the daemonset collector's `otlp` receiver (`:4317` grpc / `:4318` http), + reachable at the headless Service `otel-collector-daemonset-opentelemetry-collector..svc` + under the installer — the endpoint platform components export traces/metrics to. + +## CRs it deploys but does NOT own + +The wrapper's heavy lifting is done through upstream operators' CRs, owned by those +operators' CRDs: + +- **`ClickHouseCluster`** and **`KeeperCluster`** (the clickhouse.com operator) — rendered by + the vendored `clickstack` subchart, configured here only through + `clickstack.clickhouse.cluster.spec`. The operator (and its CRDs) must already be + installed; this repo neither ships nor versions them. Remember: upstream renders ONLY + `cluster.spec` — top-level `clickhouse.*` keys are inert + ([configuration](./configuration.md)). +- **`MongoDBCommunity`** (the MongoDB community operator) — rendered by the subchart; + `spec.statefulSet.spec` passes through, which is how the chart sets the PVC retention + policy. +- The two `otel-collector-*` charts render standard workloads + collector config from the + upstream `opentelemetry-collector` chart — no Krateo-owned CRD. + +## Where the real schemas live + +- **Composition types:** each chart's `values.schema.json` in THIS repo, read at the chart + tag. +- **`ClickHouseCluster` / `MongoDBCommunity`:** the upstream operators. +- **Runtime contracts** (the `compositionresolver` label semantics, the SSE stream shape): + the code repos `krateo-platformops/otel-collector` and `krateo-platformops/sse-proxy`, at + the image tags the charts pin. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..e5de622 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,213 @@ +--- +type: Configuration +title: clickstack-chart — configuration +description: The whole per-chart values surface — the ClickStack passthrough, the /events handler and retention TTLs, the collector pipelines, the single-replica sse-proxy — and the operational gotchas. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [configuration, values, clickhouse, opentelemetry] +timestamp: 2026-08-07T00:00:00Z +--- + +# Configuration + +The per-chart `values.yaml` surface and the real operational gotchas. Everything is traced to +the charts; where a stale comment disagrees with the rendered chart, the chart wins. Every +chart ships a `values.schema.json` typing its full surface (that schema is also what +`core-provider` turns into the composition CRD — [api](./api.md)). + +## `charts/krateo-observability` — the wrapper surface + +The heavy upstream values live under `clickstack:` and are passed through verbatim; Helm's +top-level `global` propagates into clickstack and all its subcharts +(clickhouse / otel / hyperdx / mongodb). + +### Global / exposure + +- `global.storageClassName: "standard-rwo"` — overrides the upstream default (`local-path`) + for GKE. +- **HyperDX UI exposure:** `hyperdxLoadBalancer.enabled: true`, `name: + krateo-clickstack-app-lb`, `port: 3000`. Renders an ADDITIONAL `LoadBalancer` Service + (upstream only emits the ClusterIP `krateo-clickstack`, ports 3000/4320); selector mirrors + the upstream app pods, no hardcoded `loadBalancerIP`. Expose HyperDX through the installer + CR / this flag, not by hand-patching the upstream Service. +- **HyperDX API exposure (in-cluster):** the `krateo-clickstack-api` ClusterIP Service + (always rendered, `templates/hyperdx-api-service.yaml`) reaches the HyperDX Express + backend directly at port 8000 — required for the Bearer-authenticated `/api/v2/*` external + API, which 404s through the port-3000 Next.js proxy (it strips the `/api` prefix). +- `clickstack.hyperdx.ingress.enabled: false`; `clickstack.fullnameOverride: + krateo-clickstack` (keeps ClickHouse/Keeper/Mongo names stable under `core-provider`'s + random release names — every collector/sse-proxy ClickHouse reference hardcodes + `krateo-clickstack-clickhouse-clickhouse-headless`). + +### Krateo additions (this chart's own values) + +- `otelCredentials` → the `otel-clickhouse-credentials` Secret: `secretName`, + `username: otelcollector`, `password: otelcollectorpass`. **These MUST match the ClickStack + ClickHouse `otelcollector` user** (clickstack provisions it with the same password). + Created in the release namespace (the collectors' namespace), so no cross-namespace + replication. +- **The `/events` HTTP handler** lives under + `clickstack.clickhouse.cluster.spec.settings.extraConfig.http_handlers` — NOT a ConfigMap. + The operator merges `settings.extraConfig` into `/etc/clickhouse-server/config.d/`. The + rule set declares `GET /events?composition_id=` (a `predefined_query_handler` over + `otel_logs`, filtered to `telemetry.source = 'k8s-events'` and the + `krateo.io/composition-id` attribute) **plus re-declared defaults** — `/ping`, + `/replicas_status`, `/metrics`, and the catch-all `/` `dynamic_query_handler` — because + once `http_handlers` is defined ClickHouse stops serving its default routes (operator + probes would fail without them). +- `settings.extraConfig` also carries the **3-day system-log TTLs** + (`query_log`/`text_log`/`metric_log`/`part_log`/`asynchronous_metric_log`/`trace_log`) — + retention-incident fix, do not remove (see Gotchas). + +### ClickHouse resources & storage (the load-bearing path) + +- `clickstack.clickhouse.cluster.spec.containerTemplate.resources`: limits `cpu 4 / memory + 8Gi`, requests `cpu 1 / memory 2Gi`. **`cluster.spec` is the ONLY path upstream clickstack + honours** — it renders only `clickhouse.cluster.spec` into the `ClickHouseCluster` CR. +- `clickstack.clickhouse.cluster.spec.dataVolumeClaimSpec.resources.requests.storage: 15Gi` + — the REAL data-volume size (raised from the upstream 10Gi after the retention incident). + Helm deep-merges `cluster.spec` over the upstream default, so only the overridden key is + set. +- `clickstack.clickhouse.image: clickhouse/clickhouse-server:26.3-alpine` and + `clickstack.clickhouse.persistence.size: 50Gi` are **INERT** (top-level keys upstream + ignores) and kept un-moved on purpose — realizing the image = a deliberate ClickHouse + version upgrade; to realize later move `image` → `cluster.spec.containerTemplate.image` + (storage is already real at 15Gi via `dataVolumeClaimSpec`). +- `clickstack.otel.resources` (the OTel gateway): limits `cpu 1 / memory 512Mi`, requests + `cpu 100m / memory 256Mi`. +- `clickstack.otel-collector.enabled: false` — the bundled OpAMP otel-collector subchart is + DISABLED: it crashlooped (OpAMP supervisor mode ignoring the chart `--config`) and is + redundant (Krateo's own collectors do the ClickHouse ingestion with + `krateo.io/composition-id`). +- `clickstack.mongodb`: `enabled: true`, `persistence.size: 10Gi`, plus a StatefulSet PVC + retention policy (`whenDeleted: Delete`) passed through `spec.statefulSet.spec` so MongoDB + PVCs are reaped on teardown. The ClickHouse/Keeper orphaned-PVC reaper is **not** a chart + hook (a composition post-delete hook doesn't run reliably under cdc uninstall) — it lives + in the installer's ordered-teardown path (installer ≥ 0.2.191). + +## `charts/otel-collector-deployment` — cluster-level collector + +All under `opentelemetry-collector:` (the upstream dep): + +- `mode: deployment`, `replicaCount: 1`, image + `ghcr.io/krateo-platformops/otel-collector:1.0.2` (`1.0.2` = clickhouse-exporter + schema-recreate-on-reconnect; `1.0.1` = patched k8sobjects receiver), `command.name: + otelcol-krateo` (the binary kept its name when the image repo was renamed). +- `clusterRole.create: true` with read rules on core/apps/batch/autoscaling resources plus + `get` on the Krateo CR groups (`composition.krateo.io`, `templates.krateo.io`, + `widgets.templates.krateo.io`, `deployment.krateo.io`, `core.krateo.io`) — needed so + `compositionresolver` can resolve an involvedObject to its owning composition. +- `presets` all disabled (the chart hand-rolls `config`). +- `config.receivers`: `k8sobjects` (watch Events) + `k8s_cluster` (60s); + `jaeger`/`otlp`/`zipkin`/`prometheus` are **nulled**. The `prometheus` null is + load-bearing: the minimal `krateo-otel-collector` image compiles in only + `k8sobjects`+`k8s_cluster`, and the collector type-validates every *declared* receiver — + declaring `prometheus` crashloops the default install. +- **Prometheus→OTLP bridge (scaffolded, off):** `collector.prometheusScrape.enabled` is a + MARKER only. Enabling the bridge is a three-part change: (1) a collector image built WITH + the prometheus receiver, (2) restore the scrape config preserved in comments under + `config.receivers`, (3) add `prometheus` to the metrics pipeline receivers. +- `config.processors`: `memory_limiter` (75% / 20% spike), `k8sattributes` (metadata + all + pod labels), `resource` (inserts `telemetry.source: k8s-events`), `compositionresolver` + (`cache_ttl 5m`, `negative_cache_ttl 30s`, `label_key: krateo.io/composition-id`), `batch`. +- `config.exporters.clickhouse`: + `tcp://krateo-clickstack-clickhouse-clickhouse-headless.krateo-system.svc:9000`, database + `default`, tables `otel_logs`/`otel_traces`/`otel_metrics`, `create_schema: true`, + **`ttl: 168h`** (7-day TTL stamped into table creation — retention incident, do not + remove), retry-on-failure enabled. +- Pipelines: `logs` (k8sobjects → … → clickhouse), `metrics` (k8s_cluster → memory_limiter, + batch → clickhouse), `traces: null`. +- `extraEnvs`: `CH_USERNAME`/`CH_PASSWORD` from the `otel-clickhouse-credentials` Secret. +- `resources`: limits `cpu 500m / memory 512Mi`, requests `cpu 100m / memory 256Mi`. + +## `charts/otel-collector-daemonset` — node-level collector + +Wraps the same upstream chart in `daemonset` mode, stock +`otel/opentelemetry-collector-contrib` image (`command.name: otelcol-contrib`): + +- **Headless Service** (`service.enabled: true`, `clusterIP: None`) — a stable in-cluster + DNS name for the node-local OTLP receiver + (`-opentelemetry-collector..svc:4317` grpc / `:4318` http; under the + installer the release is `otel-collector-daemonset`, which is the endpoint core-provider's + `otel.endpoint` points at). Headless → consumes no Service-CIDR address. +- `presets`: `logsCollection` (without collector logs), `hostMetrics`, + `kubernetesAttributes` (with `extractAllPodLabels`), `kubeletMetrics`. +- `config.receivers`: `filelog` with a container parser + multiline recombine; + `hostmetrics`/`kubeletstats` at 60s (kubeletstats adds pod/container cpu+memory + limit/request utilization and uptime metrics). +- `config.processors.transform`: strips ANSI escapes from log bodies and copies the pod + label `krateo.io/composition-id` into the log attributes (label-based attribution — + the daemonset does not run `compositionresolver`). +- `config.exporters.clickhouse`: same endpoint/creds as the deployment chart; + `create_schema: true` here is what creates `otel_traces` (the gateway has no traces + pipeline); **`ttl: 168h`** — do not remove. +- Pipelines: `logs` (filelog), `metrics` (`hostmetrics, kubeletstats, otlp`), `traces` + (`otlp`) — the `otlp` entries are how engine/cdc/app OTLP metrics and traces reach + ClickHouse (KOS-1 #112). +- Same `extraEnvs` Secret wiring and resources as the deployment chart. + +## `charts/krateo-sse-proxy` + +- **`replicaCount: 1` is deliberate.** sse-proxy is a STATEFUL in-memory SSE hub (each pod + runs its own ClickHouse poller + client hub). Behind an L4 LoadBalancer with + `sessionAffinity: None`, a client pins to one backend, so a degraded replica + deterministically 503s a fixed subset of users on `/notifications` — exactly what broke + the portal bell. A single hub eliminates the split; the poller resumes from `lastSeenUnix` + on restart. (To scale >1: gate readiness on poller health + add session affinity, or move + to a shared event store.) +- `image`: `registry: ghcr.io` + `repository: krateo-platformops/sse-proxy` + `tag: "1.1.2"` + (registry split out so a mirror relocation only swaps the host); + `global.imageRegistry` overrides the registry host for mirror/air-gapped installs. +- `service.type: ClusterIP`, `service.port: 8080`; container port 8080, `/health` + liveness+readiness probes. Env: `CLICKHOUSE_URL/USER/PASSWORD`, `LISTEN_ADDR :8080`. +- `clickhouse.url: + http://krateo-clickstack-clickhouse-clickhouse-headless.krateo-system.svc:8123`, + `clickhouse.user: default`, password from the `clickhouse-credentials` Secret + (`passwordSecret.optional: true`). +- Renders the fixed-name **`sse-proxy-internal-endpoint` Secret** (`server-url` → this + chart's own Service) — snowplow RESTActions reference it by that exact name; don't rename. +- `resources`: limits `cpu 200m / memory 128Mi`, requests `cpu 50m / memory 32Mi`. Hardened + securityContext (`runAsNonRoot`, `readOnlyRootFilesystem`, drop ALL caps). + +## Dependencies (what must exist around the stack) + +- **The ClickHouse operator** (the `ClickHouseCluster` CRD) and the **MongoDB community + operator** (`MongoDBCommunity`) — the wrapper renders both CRs but ships neither CRD + ([api](./api.md)). Under the installer they are the `clickhouse-operator` / + `mongodb-operator` compositions the wrapper depends on. +- **Upstream chart deps** (`Chart.lock`): `clickstack 3.0.2`, `opentelemetry-collector + 0.158.1`. +- **The `otel-clickhouse-credentials` Secret** (rendered by the wrapper) — both collectors' + `extraEnvs` reference it. +- **Stable names** via `clickstack.fullnameOverride: krateo-clickstack` — the ClickHouse + endpoint every consumer hardcodes derives from it. + +## Gotchas + +- **A behavioral fix MUST ride a version bump.** `core-provider`/helm cache a chart by + version tag and never re-pull an unchanged version. `0.1.2` was once mutably overwritten + with the 8Gi ClickHouse change; live clusters kept the cached old artifact and ClickHouse + OOMed the bell `/notifications` query — the fix was re-shipped as a new version. Never + overwrite a published version. +- **ClickHouse settings only count under `cluster.spec`.** Upstream clickstack ignores + top-level `clickhouse.resources` / `clickhouse.image` / `clickhouse.persistence` / + `clickhouse.extraVolumes`. A top-level `resources:` is INERT and silently leaves + ClickHouse at the operator default (~1Gi) → OOM; an `extraVolumes` ConfigMap mount is a + dead object (why the `/events` handler moved into `settings.extraConfig` in `0.1.6`). +- **Retention TTLs are load-bearing.** With no TTL the `otel_*` tables + `system.*` logs + filled the 10Gi volume in 8 days → inserts failed with code 243 → the collectors + OOM-crash-looped → telemetry dead (2026-07 incident). The paired fixes: 15Gi + `dataVolumeClaimSpec`, 3-day system-log TTLs (wrapper `0.1.9`), `ttl: 168h` in both + collectors' clickhouse exporter (`0.3.3`/`0.1.5`). `ttl` applies only at table CREATION + (`CREATE … IF NOT EXISTS`); pre-existing tables need a live `ALTER TABLE … MODIFY TTL`. +- **sse-proxy is single-replica on purpose** (above). +- **otel credentials must match the ClickHouse user.** `otelCredentials.username/password` + must equal the `otelcollector` user clickstack provisions; a mismatch silently fails + ClickHouse writes. +- **Keep the bundled otel-collector disabled.** Re-enabling `clickstack.otel-collector` + brings back the crashlooping OpAMP collector and duplicates ingestion without + `krateo.io/composition-id`. +- **Don't declare the prometheus receiver on the minimal image.** The collector + type-validates declared receivers even when unused — `unknown type: "prometheus"` → + CrashLoopBackOff (why it is nulled, not gated). +- **The composition types are generated, not authored.** Change the chart values / + `values.schema.json` surface and let `core-provider` re-derive the CRD ([api](./api.md)). diff --git a/docs/crds.md b/docs/crds.md deleted file mode 100644 index 7c6b24e..0000000 --- a/docs/crds.md +++ /dev/null @@ -1,53 +0,0 @@ -# clickstack CRDs (chart repo) - -**This component is a wrapper — it ships NO hand-authored CRD of its own.** There is no `crds-subchart/` -in this repo, and none of its own templates render a `CustomResourceDefinition` -(`charts/krateo-observability/templates/`, `charts/krateo-sse-proxy/templates/` — only ConfigMaps, -Secrets, Services, Deployments, RBAC). If you came looking for an observability CRD to author against, -there isn't one here. This page says so plainly and points at what the component *does* deploy. - -## The only generated "CRD": the composition type - -Like every Krateo blueprint, the wrapper chart's `values.schema.json` is consumed by `core-provider`, -which **generates** a typed CompositionDefinition CRD so the platform can reconcile instances: - -| Kind | Source | What it is | -|------|--------|------------| -| `KrateoObservability` | generated by `core-provider` from `charts/krateo-observability/values.schema.json` | the composition type the installer creates to deploy the ClickStack wrapper. NOT authored in this repo — it is derived from the chart's values schema at install time. | - -You do not edit a CRD YAML for this; you edit the chart's `values.yaml`/`values.schema.json` surface -(see [wiring.md](wiring.md)) and `core-provider` re-derives the type. The deployed chart version behind -a `KrateoObservability` is `CompositionDefinition.spec.chart.version`. - -## CRs it deploys but does NOT own - -The wrapper's heavy lifting is done through **upstream operators' CRs**, owned by those operators' -CRDs, not by this repo: - -- **`ClickHouseCluster`** (the ClickHouse / Altinity operator's CRD) — rendered by the vendored - upstream `clickstack` `3.0.0` subchart (`templates/clickhouse/cluster.yaml`), configured here only - through `clickstack.clickhouse.cluster.spec` in - `charts/krateo-observability/values.yaml`. The CRD itself comes from the ClickHouse operator, which - must already be installed in the cluster; this repo neither ships nor versions it. - - > **Resource-path gotcha (documented in `values.yaml`):** upstream `clickstack` `3.0.0` renders ONLY - > `clickhouse.cluster.spec` into the `ClickHouseCluster` CR and ignores the top-level - > `clickhouse.*` keys. Container resources MUST live under - > `clickstack.clickhouse.cluster.spec.containerTemplate.resources` to take effect — a top-level - > `resources:` is INERT (the cause of the ClickHouse OOM that broke the bell `/notifications` - > query). See [wiring.md](wiring.md). - -- **OpenTelemetry collector CRs / config** — the two `otel-collector-*` charts wrap the upstream - `opentelemetry-collector` `0.158.1` chart; they render standard workloads + collector config, not a - Krateo-owned CRD. - -## Where the real schemas live - -- **The composition (`KrateoObservability`) schema:** `charts/krateo-observability/values.schema.json` in - THIS repo (read at the chart tag) — it is the typed surface `core-provider` turns into the CRD. -- **`ClickHouseCluster` / collector CRDs:** the upstream operators (ClickHouse operator, - OpenTelemetry operator/helm chart) — not in this repo. -- **Runtime contracts** (the `compositionresolver` label, the `/events` query shape): the code repo - `krateo-platformops/otel-collector` `docs/` (at the image tag) and `files/http-handlers.xml` here. - -See [overview.md](overview.md) for the chart layout and [wiring.md](wiring.md) for the values surface. diff --git a/docs/examples.md b/docs/examples.md new file mode 100644 index 0000000..18c3735 --- /dev/null +++ b/docs/examples.md @@ -0,0 +1,20 @@ +--- +type: ExampleIndex +title: clickstack-chart — examples +description: Runnable examples under examples/, each paired with a README stating preconditions and the one apply command. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [examples, composition] +timestamp: 2026-08-07T00:00:00Z +--- + +# Examples + +Each example is a runnable manifest + a README with preconditions and the one apply command. + +- [observability-composition](../examples/observability-composition/README.md) — deploy the + ClickStack wrapper as a Krateo composition: register the + [`CompositionDefinition`](../compositiondefinition.yaml), then create a + `KrateoObservability` CR with a minimal block-style values override. + +The reference manifests under [`ops/`](../ops/README.md) (ClickHouse config, HA policies, +alert bootstrap) are operational material, not examples — the live artifacts are the charts. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..460e31f --- /dev/null +++ b/docs/index.md @@ -0,0 +1,50 @@ +--- +type: ChartRepo +title: clickstack-chart — index +description: The map of the clickstack-chart doc bundle — the four Helm charts that deploy Krateo's observability stack (ClickStack wrapper, two OTel collectors, the SSE proxy). +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [observability, clickstack, clickhouse, opentelemetry, chart-repo] +timestamp: 2026-08-07T00:00:00Z +--- + +# clickstack-chart + +This repo is the **deployment unit** for Krateo's observability stack: four +independently-versioned Helm charts, published to `oci://ghcr.io/krateo-platformops/charts`, +that the [installer](https://github.com/krateo-platformops/installer) turns into Krateo +compositions — the ClickStack wrapper (`krateo-observability`: ClickHouse + OTel gateway + +HyperDX + MongoDB), the cluster-level and node-level OTel collectors, and the SSE proxy that +feeds the portal events bell. This is the **deployment / wiring** half of the component docs; +the **internals** half (the custom collector code, the `compositionresolver` processor, the +sse-proxy binary) lives in the code repos `krateo-platformops/otel-collector` and +`krateo-platformops/sse-proxy`. + +## The bundle (start here) + +- [overview](./overview.md) — what the four charts deploy, how they version, and how they + become Krateo compositions. +- [usage](./usage.md) — how the installer consumes the charts, direct `helm install oci://…`, + and local `helm template` validation. +- [configuration](./configuration.md) — the whole per-chart values surface: the ClickStack + passthrough, the `/events` handler, retention TTLs, collector pipelines, the single-replica + sse-proxy, and the operational gotchas. +- [api](./api.md) — the contract: the four generated composition types, the + CompositionDefinition, the HTTP surfaces (`/events`, SSE, HyperDX API), and the upstream + CRs it deploys but does not own. +- [examples](./examples.md) — the runnable examples under `examples/`. +- [release](./release.md) — how a release ships (semver tag → all first-class charts to GHCR). +- [log](./log.md) — curated history (incidents, renames, version-bump lessons). +- [llms.txt](./llms.txt) — the version-pinned agent index of this bundle. + +## Deep corpus (code-adjacent, kept in place) + +- [`ops/`](../ops/README.md) — deployment-adjacent reference config folded in from the former + code repo: ClickHouse config references, HA policies, alert bootstrap scripts, reference + collector manifests. The **live** artifacts are the charts under `charts/`; `ops/` is + reference material (e.g. `ops/clickhouse-config/http-handlers.xml` is the historical copy of + the `/events` handler that now lives in + `charts/krateo-observability/values.yaml` under `cluster.spec.settings.extraConfig`). +- Code-repo internals: `krateo-platformops/otel-collector` (the custom collector image and the + `compositionresolver` processor, versioned at the **image** tag) and + `krateo-platformops/sse-proxy` (the SSE hub/poller). Read those at the image tag that the + charts here pin. diff --git a/docs/llms.txt b/docs/llms.txt index c40e2f5..e11a03d 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -1,71 +1,26 @@ -# krateo-clickstack-chart (chart repo) — LLM doc index - -> krateo-clickstack-chart is the **deployment unit** for Krateo's observability stack: the -> ClickStack wrapper (ClickHouse + OTel gateway + HyperDX + MongoDB), the two custom OTel -> collectors that feed ClickHouse, the SSE proxy that surfaces events to the portal bell, and the -> federated observability agent chart. This index is the agent entry point to the **deployment, -> CRD and wiring** view. The **internals & runtime** view (the OTel collector code, the -> `compositionresolver` processor, the sse-proxy binary) lives in the code repo -> `krateo-platformops/otel-collector` `docs/`. - -## How to use this index (agents) - -You are reading the docs for a SPECIFIC deployed version. This is the **chart-repo half** of the -component docs — deployment, CRDs, and wiring. Fetch every file below **at the CHART tag that matches -the running build**, not `main`: - -- chart version = `CompositionDefinition.spec.chart.version` for clickstack (Kind `KrateoClickstack`) - → THIS repo's tag. Read this repo's `docs/` at that tag. -- The **internals view** (the custom OTel collector image + the `compositionresolver` processor — - the telemetry path into ClickHouse) is the OTHER half: read `krateo-platformops/otel-collector` - `docs/llms.txt` at the **image-version** tag (the collector deployment's container image tag). The - portal events-bell path is a sibling component, the SSE proxy (code in `krateo-platformops/sse-proxy`). - -If something isn't in the docs, read the chart/source at that same tag — don't guess, and don't trust -`main` for a deployed older version. The agent chart (`kagent/chart`, `0.1.x`) versions independently -of the clickstack component it speaks for — do not read its version as the component's. - -## Start here - -- [overview.md](overview.md): what the ClickStack blueprint is and **how it deploys** as a Krateo - composition — the four-chart layout (`krateo-clickstack` wrapper, the two `otel-collector-*` - charts, `krateo-sse-proxy`), the `kagent/chart` agent, the `KrateoClickstack` CompositionDefinition - (owned by the installer umbrella), and what each chart renders. Read first. - -## Deployment / CRD / wiring - -- [crds.md](crds.md): the CRDs this component owns. **It ships NO hand-authored CRD** — it is a - wrapper. The only CR is the generated `KrateoClickstack` composition type (`core-provider` derives - it from the wrapper chart's `values.schema.json`); the heavy CRs it deploys (e.g. the upstream - `ClickHouseCluster` from the vendored `clickstack` subchart) belong to upstream operators, not this - repo. See the page for what that means and where the real schemas live. -- [wiring.md](wiring.md): the per-chart `values.yaml` surface — the ClickStack passthrough - (`fullnameOverride`, ClickHouse resources under `cluster.spec`, disabled bundled otel-collector), - the Krateo additions (http-handlers ConfigMap, otel-clickhouse Secret, HyperDX LoadBalancer), the - collectors' pipelines/RBAC, the single-replica sse-proxy, installer pinning/wiring, dependencies, - and the real gotchas (the inert-top-level-key trap, GOMEMLIMIT-style ClickHouse OOM via wrong - resource path, single-replica SSE hub, version-bump-forces-re-pull). - -## Cross-repo (the internals & runtime half) - -- [code-repo docs/llms.txt](https://github.com/krateo-platformops/otel-collector/blob/main/docs/llms.txt): - the code repo's index → the custom OTel collector build, the `compositionresolver` processor - (how K8s events get the `krateo.io/composition-id` label), and the sse-proxy poller/hub runtime. - Versioned at the **image** tag. - -## Reference - -- [README.md](../README.md): blueprint overview, what each chart ships, how the installer consumes it. -- `charts/krateo-clickstack/` — the ClickStack wrapper chart (`values.yaml` + `values.schema.json` + - the http-handlers ConfigMap, otel-credentials Secret, HyperDX LoadBalancer templates). -- `charts/otel-collector-deployment/` — the cluster-level OTel collector (events + cluster metrics). -- `charts/otel-collector-daemonset/` — the node-level OTel collector (pod logs, host/kubelet metrics). -- `charts/krateo-sse-proxy/` — the SSE proxy that pushes ClickHouse events to the portal bell. -- `kagent/chart/` — the `krateo-clickstack-agent` chart (the federated observability agent; versions - independently on its own `0.1.x` line). - -## Notes - -Each doc set is versioned in its own repo: THIS repo's `docs/` at the **chart** tag, the code repo's -`docs/` at the **image** tag. Where a comment in `values.yaml` or the `README.md` table disagrees with -what the chart actually renders, the rendered chart wins. +# clickstack-chart — LLM doc index (chart repo; charts version independently) +# One line per file. This repo is the DEPLOYMENT half of Krateo observability; the +# INTERNALS half (collector code, compositionresolver, sse-proxy binary) lives in +# krateo-platformops/otel-collector and krateo-platformops/sse-proxy, versioned at the +# IMAGE tags the charts pin. Read this repo's docs at the tag matching the deployed +# chart version (cluster-observable from CompositionDefinition.spec.chart.version). +# Current chart pins: krateo-observability 0.1.11, otel-collector-deployment 0.3.3, +# otel-collector-daemonset 0.1.5, krateo-sse-proxy 0.1.6. +docs/index.md: the map — what this chart repo is + links to everything below +docs/overview.md: what the four charts deploy and how they become Krateo compositions +docs/usage.md: installer path, standalone CompositionDefinition, direct helm install, local validation +docs/configuration.md: the whole per-chart values surface — ClickStack passthrough, /events handler, retention TTLs, pipelines, gotchas +docs/api.md: generated composition types, HTTP surfaces (/events, SSE, HyperDX API, OTLP), upstream CRs not owned here +docs/examples.md: index of the runnable examples +docs/release.md: how a release ships — one plain-semver tag publishes all four charts at their pinned versions +docs/log.md: curated history — incidents, renames, load-bearing version bumps +examples/observability-composition/README.md: deploy the ClickStack wrapper as a Krateo composition +examples/observability-composition/composition.yaml: the KrateoObservability CR — kubectl apply -f +# Reference corpus (code-adjacent; the LIVE artifacts are the charts under charts/): +ops/README.md: index of the deployment-adjacent reference config folded in from the former code repo +compositiondefinition.yaml: registers the wrapper standalone (outside the installer) +# Charts (ground truth for every doc claim; read values.yaml comment blocks): +charts/krateo-observability/values.yaml: the ClickStack wrapper surface + the /events handler + retention TTLs +charts/otel-collector-deployment/values.yaml: cluster-level collector (events + cluster metrics, compositionresolver) +charts/otel-collector-daemonset/values.yaml: node-level collector (logs, host/kubelet metrics, OTLP ingest) +charts/krateo-sse-proxy/values.yaml: the portal-bell SSE proxy (single replica by design) diff --git a/docs/log.md b/docs/log.md new file mode 100644 index 0000000..9061fb5 --- /dev/null +++ b/docs/log.md @@ -0,0 +1,69 @@ +--- +type: Log +title: clickstack-chart — log +description: Curated history of the observability chart repo — incidents, load-bearing fixes, renames and structural changes, newest first. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [log, history, incidents] +timestamp: 2026-08-07T00:00:00Z +--- + +# Log + +Curated, newest first. Release notes live in GitHub Releases; the `Chart.yaml` comment +blocks carry the per-bump rationale. + +- **2026-08-07 — Documentation Standard adoption.** The bundle converted to the invariant + OKF file set; `docs/wiring.md` and `docs/crds.md` folded into + [configuration](./configuration.md) and [api](./api.md); the stale repo-root + `compositiondefinition.yaml` re-pointed from the nonexistent OCI artifact + `charts/observability@0.1.8` to `charts/krateo-observability@0.1.11`. +- **2026-08-04 — `global.imageRegistry`** (sse-proxy chart): mirror/air-gap registry-host + override; `image.registry` split out of `image.repository`. +- **2026-08-03 — org migration.** Everything re-pointed to `krateo-platformops` (CI caller, + OCI registry, images). The federated `clickstack-agent` that lived under `kagent/` was + **extracted** to its own repo and is no longer part of this one. +- **2026-07-26 — wrapper `0.1.10`/`0.1.11`.** Upstream clickstack `3.0.0 → 3.0.2` + (HyperDX 2.27 → 2.30.1, unlocking the Bearer-authenticated `/api/v2/*` external API), then + the `krateo-clickstack-api` ClusterIP Service (`:8000`) so in-cluster clients bypass the + `/api`-stripping Next.js proxy at `:3000`. +- **2026-07-22 — the retention incident fix** (wrapper `0.1.9`, collectors `0.3.3`/`0.1.5`). + The 10Gi ClickHouse data volume filled in 8 days (no TTL on `otel_*` tables, ~3.6GiB of + unbounded `system.*` logs) → inserts failed with code 243 → the collectors + OOM-crash-looped → telemetry dead. Durable fix: 15Gi `dataVolumeClaimSpec`, 3-day + system-log TTLs in `settings.extraConfig`, `ttl: 168h` in both collectors' clickhouse + exporter. See [configuration → gotchas](./configuration.md). +- **2026-07-10 — collector image `1.0.2`** (chart `0.3.2`): the forked clickhouse exporter + recreates the `otel_*` schema on reconnect (a fresh ClickHouse PVC no longer leaves the + stack table-less until a manual collector restart). +- **2026-07-08 — PVC reaper moved out of the chart.** A composition post-delete hook does + not run reliably under cdc uninstall; the wrapper's `0.1.7` hook was reverted in `0.1.8`. + MongoDB PVCs are reaped chart-side via StatefulSet `persistentVolumeClaimRetentionPolicy`; + the ClickHouse/Keeper reaper lives in the installer's ordered teardown (installer ≥ 0.2.191). +- **2026-07-04 — daemonset `0.1.4`:** headless Service for the node-local OTLP receiver — a + stable DNS name for engine/cdc/app OTLP export without consuming a Service-CIDR address. +- **2026-07-02 — prometheus receiver nulled** (deployment collector). The minimal + `krateo-otel-collector` image compiles in only `k8sobjects` + `k8s_cluster`; merely + *declaring* the prometheus receiver crashlooped the default install. The Prometheus→OTLP + bridge (added 2026-06-29) remains scaffolded-but-off. +- **2026-06-22 — OTLP ingest wired** (daemonset `0.1.2`/`0.1.3`): `otlp` added to the + daemonset's traces and metrics pipelines (KOS-1 #112) — the path platform components' + OTLP telemetry takes into ClickHouse. sse-proxy `0.1.4` added the fixed-name + `sse-proxy-internal-endpoint` Secret for snowplow RESTActions. +- **2026-06-20 — chart renamed** `krateo-clickstack` → `krateo-observability` (composition + kind `KrateoObservability`). Chart renames change the generated CRD kind — they are + API-breaking for compositions. +- **2026-06-18 — `/events` moved to `settings.extraConfig`** (wrapper `0.1.6`). The + http-handlers ConfigMap + `clickhouse.extraVolumes` mount was inert (upstream renders only + `cluster.spec`; the CRD has no `extraVolumes`) and `/events` 404'd. The operator-native + config merge is the only working path; ClickHouse default routes are re-declared alongside. +- **2026-06-16 — bundled OpAMP otel-collector disabled** (`clickstack.otel-collector.enabled: + false`): it crashlooped (OpAMP supervisor mode ignores the chart `--config`) and duplicated + Krateo's own ingestion without `krateo.io/composition-id`. +- **2026-06-15 — the inert-resources OOM + the mutable-version lesson.** ClickHouse + resources moved to `cluster.spec.containerTemplate.resources` (the only path upstream + honours; the old top-level `resources:` left ClickHouse at ~1Gi and the bell + `/notifications` query OOMed). The fix had earlier been overwritten onto the + already-published `0.1.2` — consumers cache by version tag and never re-pulled — so it was + re-shipped as `0.1.3`: **a behavioral fix MUST ride a version bump.** Same day: sse-proxy + pinned to a single replica (stateful in-memory SSE hub), and the collector/sse-proxy code + split out to their own repos (`ops/` folded in here as reference material). diff --git a/docs/overview.md b/docs/overview.md index 45be0f5..0c6ef6d 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -1,133 +1,143 @@ -# clickstack — deployment overview (chart repo) - -What the Krateo **ClickStack observability blueprint** is, and **how it deploys** as a set of Krateo -compositions. This is the deployment view; the internals/runtime view (the custom OTel collector, the -`compositionresolver` processor, the sse-proxy binary) lives in the code repo -`krateo-platformops/otel-collector` (`docs/`). Every claim below is traced to a file in this repo — if a -comment disagrees with what the chart actually renders, the rendered chart wins. - -## What clickstack is - -The Krateo portal's **observability stack**. It ingests Kubernetes telemetry (pod logs, K8s Events, -host/kubelet/cluster metrics) into ClickHouse, enriches each event with a -`krateo.io/composition-id` so telemetry can be sliced per Krateo composition, lets operators explore -it through HyperDX, and feeds two consumers: - -- the **portal events bell**, via a ClickHouse `GET /events?composition_id=` http-handler and - the `krateo-sse-proxy` that streams new events to the browser over Server-Sent Events; -- the **`krateo-clickstack-agent`** (this repo's `kagent/chart`), which troubleshoots Krateo/K8s - issues by querying the same ClickHouse tables. - -This repo is the **krateo-platformops packaging as a Krateo blueprint**: it wraps the upstream ClickStack -Helm chart and the upstream OpenTelemetry collector chart, folds in the Krateo-specific glue, and -ships a `values.schema.json` so `core-provider` can generate a typed CompositionDefinition CRD. It -replaces the old imperative `obs-stack` `install.sh` phases (no hand-run `kubectl patch`, -`README.md`). - -## Repo layout — four deployable charts + the agent - -| Path | Chart name | OCI artifact | Versioning | -|------|------------|--------------|------------| -| `charts/krateo-observability` | `krateo-observability` | `oci://ghcr.io/krateo-platformops/charts/observability` | tracks the git tag (`Chart.yaml` `version: CHART_VERSION`); current literal `0.1.5`, `appVersion 3.0.0` | -| `charts/otel-collector-deployment` | `otel-collector-deployment` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment` | pinned `0.2.0`, independent | -| `charts/otel-collector-daemonset` | `otel-collector-daemonset` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset` | pinned `0.1.1`, independent | -| `charts/krateo-sse-proxy` | `krateo-sse-proxy` | `oci://ghcr.io/krateo-platformops/charts/sse-proxy` | pinned `0.1.3`, independent | -| `kagent/chart` | `krateo-clickstack-agent` | `oci://ghcr.io/krateo-platformops/charts/clickstack-agent` | `0.1.x`, independent (`kagent/chart/Chart.yaml`) | - -They version **independently**: - -- **The ClickStack wrapper** (`charts/krateo-observability`) is the heaviest piece. `version` is the - `CHART_VERSION` placeholder substituted to the git tag at release; `appVersion` is `3.0.0` (the - upstream `clickstack` dependency version, `Chart.yaml`). It vendors the upstream `clickstack` - `3.0.0` chart as a dependency and passes most values through under the `clickstack:` key. The - `KrateoObservability` composition the installer creates is this chart. - > **Version bumps are load-bearing for re-pull.** `core-provider`/helm cache a chart by its version - > tag and never re-pull an *unchanged* version, so a behavioral fix MUST ride a version bump or live - > clusters keep the old artifact (`Chart.yaml` comment, the `0.1.5` history) — see - > [wiring.md](wiring.md). -- **The two OTel collector charts** (`otel-collector-deployment`, `otel-collector-daemonset`) each - wrap the upstream `opentelemetry-collector` `0.158.1` chart and are pinned independently of the - wrapper. The deployment-mode one runs the custom `krateo-otel-collector` image (the - `compositionresolver` processor); the daemonset-mode one runs node-level log/metric collection. -- **The SSE proxy** (`krateo-sse-proxy`) is a small Krateo-built Go service (image - `ghcr.io/krateo-platformops/sse-proxy`); pinned `0.1.3`, single replica by design (it is a stateful - in-memory SSE hub — see [wiring.md](wiring.md)). -- **The agent chart** (`kagent/chart`) is the federated specialist agent (`krateo-clickstack-agent`) - registered on `krateo-autopilot`; it versions on its own `0.1.x` line and is **not** any - observability workload. `kagent/compositiondefinition.yaml` ships its CompositionDefinition (pinned - `0.1.0`). - -## The CompositionDefinition - -The repo-root `compositiondefinition.yaml` registers the wrapper with Krateo: `core.krateo.io/v1alpha1`, -name `krateo-observability`, namespace `krateo-system`, `spec.chart.url` = -`oci://ghcr.io/krateo-platformops/charts/observability`, `spec.chart.version` pinned (currently `"0.1.2"` -in this file). In a real install the [krateo-installer](https://github.com/krateo-platformops/installer) -umbrella owns this and the per-collector CompositionDefinitions (`README.md`). `core-provider` reads -the wrapper chart's `values.schema.json`, generates the typed `KrateoObservability` CRD, and reconciles -one Composition per instance. The deployed chart version is cluster-observable from -`CompositionDefinition.spec.chart.version` (the tag at which an agent should fetch THIS repo's docs — -see [llms.txt](llms.txt)). - -> The pinned version in `compositiondefinition.yaml` (`0.1.2`) is the *registered* version, not -> necessarily the latest chart tag in the repo (`Chart.yaml` is `0.1.5`). Always read the cluster's -> live `CompositionDefinition.spec.chart.version` to know what is actually deployed. - -The CompositionDefinition that also lives here is the agent's -(`kagent/compositiondefinition.yaml`): `core.krateo.io/v1alpha1`, name `krateo-clickstack-agent`, -namespace `krateo-system`, `spec.chart.version: "0.1.0"`. +--- +type: Architecture +title: clickstack-chart — overview +description: What the four charts deploy — the ClickStack wrapper, the two OTel collectors, the SSE proxy — how they version independently, and how they become Krateo compositions. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [observability, clickstack, architecture] +timestamp: 2026-08-07T00:00:00Z +--- + +# Overview + +What the Krateo **observability stack** is and **how it deploys** as a set of Krateo +compositions. This is the deployment view; the internals/runtime view (the custom OTel +collector, the `compositionresolver` processor, the sse-proxy binary) lives in the code repos +`krateo-platformops/otel-collector` and `krateo-platformops/sse-proxy`. Every claim below is +traced to a file in this repo — where a comment disagrees with what the chart actually +renders, the rendered chart wins. + +## What the stack is + +The Krateo portal's observability backend. It ingests Kubernetes telemetry (pod logs, K8s +Events, host/kubelet/cluster metrics, OTLP traces) into ClickHouse, enriches each K8s event +with a `krateo.io/composition-id` attribute so telemetry can be sliced per Krateo +composition, lets operators explore it through HyperDX, and feeds the **portal events bell**: +a ClickHouse `GET /events?composition_id=` predefined-query handler plus the +`krateo-sse-proxy` that streams new events to the browser over Server-Sent Events. + +This repo is the packaging as Krateo blueprints: it wraps the upstream ClickStack Helm chart +and the upstream OpenTelemetry collector chart, folds in the Krateo-specific glue, and ships +a `values.schema.json` per chart so `core-provider` can generate typed composition CRDs. + +## Repo layout — four deployable charts + +| Path | Chart name | OCI artifact | Version (`Chart.yaml`) | +|------|------------|--------------|------------------------| +| `charts/krateo-observability` | `krateo-observability` | `oci://ghcr.io/krateo-platformops/charts/krateo-observability` | `0.1.11`, `appVersion 3.0.2` | +| `charts/otel-collector-deployment` | `otel-collector-deployment` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment` | `0.3.3`, image `otel-collector:1.0.2` | +| `charts/otel-collector-daemonset` | `otel-collector-daemonset` | `oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset` | `0.1.5` | +| `charts/krateo-sse-proxy` | `krateo-sse-proxy` | `oci://ghcr.io/krateo-platformops/charts/krateo-sse-proxy` | `0.1.6`, image `sse-proxy:1.1.2` | + +All four versions are **literally pinned** in each `Chart.yaml` (no `CHART_VERSION` +placeholder), so a release tag publishes each chart at its own declared version — the OCI +artifact name is the chart name (see [release](./release.md)). + +- **The ClickStack wrapper** (`charts/krateo-observability`) is the heaviest piece. It + vendors the upstream `clickstack` `3.0.2` chart as a dependency and passes values through + under the `clickstack:` key. The `KrateoObservability` composition the installer creates is + this chart. + > **Version bumps are load-bearing for re-pull.** `core-provider`/helm cache a chart by its + > version tag and never re-pull an *unchanged* version, so a behavioral fix MUST ride a + > version bump or live clusters keep the old artifact (`Chart.yaml` history: `0.1.3`). +- **The two OTel collector charts** each wrap the upstream `opentelemetry-collector` + `0.158.1` chart. The deployment-mode one runs the custom + `ghcr.io/krateo-platformops/otel-collector` image (binary `otelcol-krateo`, the + `compositionresolver` processor); the daemonset-mode one runs the stock + `otel/opentelemetry-collector-contrib` image for node-level collection. +- **The SSE proxy** (`krateo-sse-proxy`) is a small Krateo-built Go service + (`ghcr.io/krateo-platformops/sse-proxy:1.1.2`), deliberately single-replica (a stateful + in-memory SSE hub — see [configuration](./configuration.md)). + +> The federated `clickstack-agent` chart used to live here under `kagent/`; it was +> **extracted** during the org migration (2026-08-03) and is no longer part of this repo. ## What each chart deploys ### `charts/krateo-observability` (the wrapper) -The vendored upstream `clickstack` subchart renders ClickHouse (via the ClickHouse operator's -`ClickHouseCluster` CR), the OTel gateway, HyperDX, and MongoDB. Krateo additions rendered by THIS -chart's own templates (`charts/krateo-observability/templates/`): +The vendored `clickstack` subchart renders ClickHouse (a `ClickHouseCluster` CR for the +ClickHouse operator), a Keeper cluster, HyperDX (Deployment + the `krateo-clickstack` +ClusterIP Service on ports 3000/app and 4320/opamp) and MongoDB (a `MongoDBCommunity` CR). +Krateo additions rendered by THIS chart's own templates +(`charts/krateo-observability/templates/`): -- **`clickhouse-http-handlers` ConfigMap** (`http-handlers-configmap.yaml`, gated on - `httpHandlers.enabled`) — the `GET /events?composition_id=` predefined-query handler XML from - `files/http-handlers.xml`, mounted into the ClickHouse pods at - `/etc/clickhouse-server/config.d/http-handlers.xml` via `clickstack.clickhouse.extraVolumes`. - **`otel-clickhouse-credentials` Secret** (`otel-credentials-secret.yaml`, gated on - `otelCredentials.enabled`) — `username`/`password` for the collectors' ClickHouse `otelcollector` - user, created in the release namespace (`krateo-system`). + `otelCredentials.enabled`) — `username`/`password` for the collectors' ClickHouse + `otelcollector` user, created in the release namespace. - **`krateo-clickstack-app-lb` LoadBalancer Service** (`hyperdx-loadbalancer.yaml`, gated on - `hyperdxLoadBalancer.enabled`) — an ADDITIONAL Service exposing the HyperDX UI on port 3000 - externally (upstream only emits a ClusterIP), with no hardcoded `loadBalancerIP`. - -`clickstack.fullnameOverride: krateo-observability` keeps the ClickHouse/Keeper/Mongo Service names -stable regardless of the random release name `core-provider` assigns, so the collectors and sse-proxy -can resolve `krateo-clickstack-clickhouse-clickhouse-headless`. + `hyperdxLoadBalancer.enabled`) — an ADDITIONAL Service exposing the HyperDX UI on port + 3000 externally (upstream only emits the ClusterIP `krateo-clickstack`), no hardcoded + `loadBalancerIP`. +- **`krateo-clickstack-api` ClusterIP Service** (`hyperdx-api-service.yaml`, always + rendered, since `0.1.11`) — exposes the HyperDX Express backend directly on port 8000, so + in-cluster clients (Composition RESTDefinitions) can call the Bearer-authenticated + `/api/v2/*` external API without the Next.js proxy's `/api`-stripping at port 3000. + +The `GET /events?composition_id=` ClickHouse HTTP handler is **not a template**: since +`0.1.6` it lives in `values.yaml` under +`clickstack.clickhouse.cluster.spec.settings.extraConfig.http_handlers` (the operator-native +config merge). The former `clickhouse-http-handlers` ConfigMap + `extraVolumes` mount was +inert — upstream clickstack renders only `clickhouse.cluster.spec` — and was removed. + +`clickstack.fullnameOverride: krateo-clickstack` keeps the ClickHouse/Keeper/Mongo resource +names stable regardless of the release name `core-provider` assigns, so the collectors and +sse-proxy can resolve `krateo-clickstack-clickhouse-clickhouse-headless`. ### `charts/otel-collector-deployment` (cluster-level) A single-replica `opentelemetry-collector` in `deployment` mode running the custom -`krateo-otel-collector` image (binary `otelcol-krateo`). Pipelines: a `logs` pipeline -(`k8sobjects` events → `memory_limiter, k8sattributes, resource, compositionresolver, batch` → -ClickHouse) and a `metrics` pipeline (`k8s_cluster` → ClickHouse). The `compositionresolver` processor -stamps `krateo.io/composition-id`. It ships its own `clusterRole` (read on K8s + Krateo CR groups) and -pulls ClickHouse creds from the `otel-clickhouse-credentials` Secret via `extraEnvs`. +`krateo-otel-collector` image. Pipelines: `logs` (`k8sobjects` Events → +`memory_limiter, k8sattributes, resource, compositionresolver, batch` → ClickHouse) and +`metrics` (`k8s_cluster` → ClickHouse); `traces: null`. The `compositionresolver` processor +stamps `krateo.io/composition-id`. It ships its own ClusterRole (read on K8s workloads + `get` +on the Krateo CR groups) and pulls ClickHouse creds from the `otel-clickhouse-credentials` +Secret via `extraEnvs`. A Prometheus→OTLP bridge is scaffolded but **nulled by default** (the +minimal collector image has no `prometheus` receiver — see +[configuration](./configuration.md)). ### `charts/otel-collector-daemonset` (node-level) -A daemonset-mode `opentelemetry-collector` collecting pod logs, host metrics and kubelet metrics and -exporting them to ClickHouse. +A daemonset-mode `opentelemetry-collector` (stock contrib image) collecting pod logs +(`filelog`), host metrics and kubelet metrics, plus **OTLP ingest**: its `metrics` and +`traces` pipelines consume the node-local `otlp` receiver so the engine/cdc/apps can export +OTLP → ClickHouse. A **headless Service** (`0.1.4`) gives that receiver a stable in-cluster +DNS name (`-opentelemetry-collector..svc:4317` grpc / `:4318` http). ### `charts/krateo-sse-proxy` -A Deployment + ClusterIP Service (`templates/`). One container, port 8080, `/health` probes, polls -ClickHouse (`clickhouse.url` → `krateo-clickstack-clickhouse-clickhouse-headless:8123`) and pushes new -K8s events to connected browsers over SSE. `replicaCount: 1` is deliberate (stateful in-memory hub). +A Deployment + ClusterIP Service (port 8080, `/health` probes) polling ClickHouse +(`clickhouse.url` → `krateo-clickstack-clickhouse-clickhouse-headless:8123`) and pushing new +K8s events to browsers over SSE. It also renders the **`sse-proxy-internal-endpoint` +Secret** — the fixed-name Endpoint that snowplow RESTActions use to fetch the historical +`/events` snapshot in-cluster. `replicaCount: 1` is deliberate. + +## How it becomes compositions -For the full per-chart `values.yaml` surface, the installer wiring, and the operational gotchas, see -[wiring.md](wiring.md). For what CRDs this component does (and does not) own, see [crds.md](crds.md). +The [installer](https://github.com/krateo-platformops/installer) umbrella pins **all four +charts** as `tier: observability` components gated on the portal feature, ordered by +dependencies: the ClickHouse + MongoDB operators install first, then `krateo-observability`, +then `otel-collector-deployment` → `otel-collector-daemonset`, and `krateo-sse-proxy` +(exposed on port 8080 for the portal bell). For each, `core-provider` reads the chart's +`values.schema.json` and generates a typed composition CRD — `KrateoObservability`, +`OtelCollectorDeployment`, `OtelCollectorDaemonset`, `KrateoSseProxy`. The repo-root +[`compositiondefinition.yaml`](../compositiondefinition.yaml) registers the wrapper +standalone (outside the installer). Details: [api](./api.md), [usage](./usage.md). ## Cross-references -- **Code repo (internals & runtime):** `krateo-platformops/otel-collector` — - [`docs/llms.txt`](https://github.com/krateo-platformops/otel-collector/blob/main/docs/llms.txt). That set - is versioned at the **image** tag (the collector/sse-proxy image tags); this set is versioned at the - **chart** tag. +- **Code repos (internals & runtime):** `krateo-platformops/otel-collector` (collector image, + `compositionresolver`) and `krateo-platformops/sse-proxy` — versioned at the **image** + tags the charts pin; this repo's docs are versioned at the **chart** tags. - **Installer umbrella:** `krateo-platformops/installer` (owns the CompositionDefinitions). -- **Upstream:** `ClickHouse/ClickStack-helm-charts`, `open-telemetry/opentelemetry-helm-charts`. +- **Upstream:** [`ClickHouse/ClickStack-helm-charts`](https://github.com/ClickHouse/ClickStack-helm-charts), + [`open-telemetry/opentelemetry-helm-charts`](https://github.com/open-telemetry/opentelemetry-helm-charts). diff --git a/docs/release.md b/docs/release.md new file mode 100644 index 0000000..4bbca66 --- /dev/null +++ b/docs/release.md @@ -0,0 +1,53 @@ +--- +type: Runbook +title: clickstack-chart — release +description: How a release ships — a plain-semver tag publishes every first-class chart at its own literally-pinned version to oci://ghcr.io/krateo-platformops/charts. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [release, ci, oci] +timestamp: 2026-08-07T00:00:00Z +--- + +# Release + +This repo uses the canonical shape-agnostic +[`release-oci.yaml`](../.github/workflows/release-oci.yaml) workflow (byte-identical across +krateo-* package repos). One tag publishes **all four charts**, each at its own version. + +## The runbook + +1. **Bump the changed chart's `version` in its `Chart.yaml`** — all four charts are + *literally pinned* (no `CHART_VERSION` placeholder), so the chart version is authored in + the PR, not derived from the tag. **This is mandatory for any behavioral change**: + `core-provider`/helm cache charts by version tag and never re-pull an unchanged version + ([configuration → gotchas](./configuration.md)). Keep the `Chart.yaml` comment history + (each bump documents why). +2. **Merge to `main`** with PR CI green: [`lint.yaml`](../.github/workflows/lint.yaml) + (helm lint + schema well-formedness + render smoke per chart, and the docs-standard + lint), plus the org security workflow. +3. **Tag with plain semver — `X.Y.Z`, no `v` prefix.** The workflow triggers on + `[0-9]+.[0-9]+.[0-9]+` only. Historically this repo's tags track the upstream + clickstack line (`3.0.x`, `3.1.0`) — the tag value does not touch chart versions here + (no placeholders), it only triggers the publish. +4. **CI publishes** every first-class chart (any `Chart.yaml` not vendored inside another + chart's `charts/`) to `oci://ghcr.io/krateo-platformops/charts` — the artifact name is + the **chart name**: `krateo-observability`, `otel-collector-deployment`, + `otel-collector-daemonset`, `krateo-sse-proxy`. Upstream dependencies + (`clickstack`, `opentelemetry-collector`) are `helm dependency build`-vendored at package + time. +5. **Wire the consumers**: bump the version pin in the installer's component pins (and, for + standalone installs, [`compositiondefinition.yaml`](../compositiondefinition.yaml)). + A wrapper-version bump changes the composition CR apiVersion + (`v0-1-11` style — [api](./api.md)); note that a composition chart-version bump is + handled by cdc as an in-place handover, but stateful re-rolls of ClickHouse/Keeper are + the risk to watch on live clusters. + +## Verify + +```sh +helm show chart oci://ghcr.io/krateo-platformops/charts/krateo-observability --version +``` + +Image versions are pinned inside values (`otel-collector:1.0.2`, `sse-proxy:1.1.2`) and ship +from their own code repos (`krateo-platformops/otel-collector`, +`krateo-platformops/sse-proxy`) — releasing a new image requires a chart values bump here +plus a chart version bump. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..fffd01d --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,74 @@ +--- +type: Usage +title: clickstack-chart — usage +description: How the installer consumes the four charts, how to install the wrapper standalone via CompositionDefinition or direct helm install, and local helm template validation. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [usage, installer, helm] +timestamp: 2026-08-07T00:00:00Z +--- + +# Usage + +## The canonical way: via the Krateo installer + +The observability stack ships as part of a Krateo platform deploy. The +[installer](https://github.com/krateo-platformops/installer) umbrella pins all four charts as +`tier: observability` components gated on the **portal** feature — enabling the portal brings +up (in dependency order) the ClickHouse and MongoDB operators, `krateo-observability`, +`otel-collector-deployment`, `otel-collector-daemonset` and `krateo-sse-proxy` (exposed on +port 8080 for the portal events bell). There is nothing to install by hand from this repo: +the installer emits a CompositionDefinition per chart and `core-provider` reconciles the +compositions. + +The deployed chart version is cluster-observable from +`CompositionDefinition.spec.chart.version` — that tag is the version of THIS repo's docs an +agent should read (see [llms.txt](./llms.txt)). + +## Standalone: register the wrapper as a composition + +Outside the installer you need `core-provider` plus the ClickHouse and MongoDB operators +already installed (the wrapper renders a `ClickHouseCluster` and a `MongoDBCommunity` CR but +ships neither CRD — see [api](./api.md)). Then: + +```sh +kubectl apply -f compositiondefinition.yaml # registers oci://ghcr.io/krateo-platformops/charts/krateo-observability +``` + +`core-provider` generates the `KrateoObservability` CRD from the chart's +`values.schema.json`; create an instance to deploy the stack +([examples/observability-composition](../examples/observability-composition/README.md)). + +## Direct helm install (no Krateo engine) + +Each chart is a plain Helm chart on GHCR: + +```sh +helm install krateo-observability oci://ghcr.io/krateo-platformops/charts/krateo-observability \ + --version 0.1.11 --namespace krateo-system --create-namespace +helm install otel-collector-deployment oci://ghcr.io/krateo-platformops/charts/otel-collector-deployment \ + --version 0.3.3 --namespace krateo-system +helm install otel-collector-daemonset oci://ghcr.io/krateo-platformops/charts/otel-collector-daemonset \ + --version 0.1.5 --namespace krateo-system +helm install krateo-sse-proxy oci://ghcr.io/krateo-platformops/charts/krateo-sse-proxy \ + --version 0.1.6 --namespace krateo-system +``` + +Order matters: the wrapper first (it creates the `otel-clickhouse-credentials` Secret both +collectors mount), collectors after. The hardcoded ClickHouse Service references +(`krateo-clickstack-clickhouse-clickhouse-headless.krateo-system.svc`) assume the +`krateo-system` namespace and the wrapper's `clickstack.fullnameOverride: krateo-clickstack` +— see [configuration](./configuration.md) before deviating. + +## Local validation + +```sh +for d in charts/*/; do + helm dependency build "$d" 2>/dev/null || true # clickstack / opentelemetry-collector deps + helm lint "$d" + helm template smoke "$d" --namespace krateo-system >/dev/null +done +``` + +This is what PR CI runs ([lint.yaml](../.github/workflows/lint.yaml)): helm lint, +`values.schema.json` well-formedness, and a render smoke test per chart — plus the +docs-standard lint (shared org workflow). diff --git a/docs/wiring.md b/docs/wiring.md deleted file mode 100644 index 595e44d..0000000 --- a/docs/wiring.md +++ /dev/null @@ -1,143 +0,0 @@ -# clickstack — composition wiring & operations (chart repo) - -The per-chart `values.yaml` surface, how the installer pins and wires the observability stack, the -dependencies, and the real operational gotchas. Everything is traced to the charts; where a stale -comment disagrees with the rendered chart, the chart wins. - -## `charts/krateo-observability` — the wrapper surface - -The heavy upstream values live under `clickstack:` and are passed through verbatim; Helm's top-level -`global` propagates into clickstack and all its subcharts (clickhouse / otel / hyperdx / mongodb). - -### Global / exposure - -- `global.storageClassName: "standard-rwo"` (`values.yaml`) — overrides the upstream default - (`local-path`) for GKE. -- **HyperDX exposure:** `hyperdxLoadBalancer.enabled: true`, `name: krateo-clickstack-app-lb`, - `port: 3000`. Renders an ADDITIONAL `LoadBalancer` Service (upstream only emits the ClusterIP - `krateo-clickstack-app`); selector mirrors the upstream app pods, no hardcoded `loadBalancerIP`. - Expose HyperDX through the installer CR / this flag, not by hand-patching the upstream Service. -- `clickstack.hyperdx.ingress.enabled: false`, `clickstack.fullnameOverride: krateo-observability` - (keeps ClickHouse/Keeper/Mongo Service names stable under `core-provider`'s random release name). - -### Krateo additions (this chart's own templates) - -- `httpHandlers.enabled: true` → the `clickhouse-http-handlers` ConfigMap (the `/events` handler), - mounted into ClickHouse via `clickstack.clickhouse.extraVolumes` /`extraVolumeMounts` at - `/etc/clickhouse-server/config.d/http-handlers.xml`. -- `otelCredentials` → the `otel-clickhouse-credentials` Secret: `secretName`, - `username: otelcollector`, `password: otelcollectorpass`. **These MUST match the ClickStack - ClickHouse `otelcollector` user** (clickstack provisions it via `extraUsersConfig` with the same - password). Created in `krateo-system` (the collectors' namespace), so no cross-namespace replication. - -### ClickHouse resources (the load-bearing path) - -- `clickstack.clickhouse.cluster.spec.containerTemplate.resources`: limits `cpu 4 / memory 8Gi`, - requests `cpu 1 / memory 2Gi`. **This is the ONLY path upstream `clickstack` 3.0.0 honours** — it - renders only `clickhouse.cluster.spec` into the `ClickHouseCluster` CR. -- `clickstack.clickhouse.image: clickhouse/clickhouse-server:26.3-alpine` and - `clickstack.clickhouse.persistence.size: 50Gi` are **INERT** (top-level keys upstream ignores). They - are kept un-moved on purpose: realizing them = a ClickHouse version upgrade (25.7→26.3) + PVC resize - (10Gi→50Gi), a deliberate change that must not ride along with the memory fix. To realize later, - move `image` → `cluster.spec.containerTemplate.image` and `size` → - `cluster.spec.dataVolumeClaimSpec.resources.requests.storage`. -- `clickstack.otel.resources`: limits `cpu 1 / memory 512Mi`, requests `cpu 100m / memory 256Mi`. -- `clickstack.otel-collector.enabled: false` — the bundled OpAMP otel-collector subchart is DISABLED: - it crashlooped (exit 2, OpAMP supervisor mode ignoring the chart `--config`) and is redundant here - (Krateo's own collectors do the real ClickHouse ingestion with `krateo.io/composition-id`). -- `clickstack.mongodb.enabled: true`, `mongodb.persistence.size: 10Gi`. - -## `charts/otel-collector-deployment` — cluster-level collector - -All under `opentelemetry-collector:` (the upstream dep): - -- `mode: deployment`, `replicaCount: 1`, image `ghcr.io/krateo-platformops/otel-collector:1.0.0`, - `command.name: otelcol-krateo` (image repo renamed; binary name unchanged). -- `clusterRole.create: true` with read rules on core/apps/batch/autoscaling K8s resources plus `get` - on the Krateo CR groups (`composition.krateo.io`, `templates.krateo.io`, - `widgets.templates.krateo.io`, `deployment.krateo.io`, `core.krateo.io`) — needed so the - `compositionresolver` can resolve an involvedObject to its owning composition. -- `presets` all disabled (the chart hand-rolls `config` instead). -- `config`: `receivers` `k8sobjects` (watch Events) + `k8s_cluster` (60s metrics); `processors` - `memory_limiter` (75% / 20% spike), `k8sattributes`, `resource` (`telemetry.source: k8s-events`), - `compositionresolver` (`cache_ttl 5m`, `negative_cache_ttl 30s`, - `label_key: krateo.io/composition-id`), `batch`; `exporters` `clickhouse` → - `tcp://krateo-clickstack-clickhouse-clickhouse-headless.krateo-system.svc:9000`, db `default`, - tables `otel_logs`/`otel_traces`/`otel_metrics`, `create_schema: true`. Two pipelines: `logs` - (k8sobjects → … → clickhouse) and `metrics` (k8s_cluster → memory_limiter, batch → clickhouse); - `traces: null`. -- `extraEnvs` `CH_USERNAME`/`CH_PASSWORD` from the `otel-clickhouse-credentials` Secret. -- `resources`: limits `cpu 500m / memory 512Mi`, requests `cpu 100m / memory 256Mi`. - -## `charts/otel-collector-daemonset` — node-level collector - -Wraps the same upstream `opentelemetry-collector` `0.158.1` in daemonset mode for pod logs, host and -kubelet metrics → ClickHouse. - -## `charts/krateo-sse-proxy` - -- **`replicaCount: 1` is deliberate.** sse-proxy is a STATEFUL in-memory SSE hub (each pod runs its - own ClickHouse poller + client hub). Behind an L4 LoadBalancer with `sessionAffinity: None`, GCP - hashes each client to one backend, so a degraded replica deterministically 503s a fixed subset of - users on `/notifications` — exactly what broke the portal bell. A single hub eliminates the split; - the poller resumes from `lastSeenUnix` on restart. (To scale >1: gate readiness on poller health + - add `sessionAffinity`, or move to a shared event store.) -- `image: ghcr.io/krateo-platformops/sse-proxy:1.0.0`; `service.type: ClusterIP`, `service.port: 8080`; - container port 8080, `/health` liveness+readiness. -- `clickhouse.url: http://krateo-clickstack-clickhouse-clickhouse-headless.krateo-system.svc:8123`, - `clickhouse.user: default`, password from the `clickhouse-credentials` Secret - (`passwordSecret.optional: true`). Env: `CLICKHOUSE_URL/USER/PASSWORD`, `LISTEN_ADDR :8080`. -- `resources`: limits `cpu 200m / memory 128Mi`, requests `cpu 50m / memory 32Mi`. Hardened - securityContext (`runAsNonRoot`, `readOnlyRootFilesystem`, drop ALL caps). - -## Dependencies (what must exist around the stack) - -- **The ClickHouse operator** (provides the `ClickHouseCluster` CRD) — must be installed; the wrapper - renders a `ClickHouseCluster` CR but does not ship its CRD (see [crds.md](crds.md)). -- **Upstream chart deps** vendored under each chart's `charts/`: `clickstack 3.0.0`, - `opentelemetry-collector 0.158.1`. -- **The `otel-clickhouse-credentials` Secret** (rendered by the wrapper) — both collectors `extraEnvs` - reference it; its `username`/`password` must match the ClickHouse `otelcollector` user. -- **The `clickhouse-http-handlers` ConfigMap** (rendered by the wrapper) — the `/events` handler the - sse-proxy / bell depend on. -- **Stable Service names** via `clickstack.fullnameOverride` — every collector/sse-proxy ClickHouse - reference hardcodes `krateo-clickstack-clickhouse-clickhouse-headless`. - -## How the installer wires it - -The [krateo-installer](https://github.com/krateo-platformops/installer) umbrella owns the -CompositionDefinitions for the wrapper and the collectors (`README.md`). It pins each -`spec.chart.version` to a released chart tag and points `core-provider` at the OCI artifacts; -`core-provider` reads `charts/krateo-observability/values.schema.json`, generates the `KrateoObservability` -CRD, and reconciles one Composition per instance. The collectors depend on the `krateo-observability` -composition (for the credentials Secret), and `krateo-sse-proxy` is exposed so the portal events bell -can reach it. The deployed chart version is readable from `CompositionDefinition.spec.chart.version` -(the tag at which to fetch THIS repo's docs). - -## Gotchas - -- **A behavioral fix MUST ride a version bump.** `core-provider`/helm cache a chart by version tag and - never re-pull an *unchanged* version. `0.1.2` was once mutably overwritten with the 8Gi ClickHouse - change, but live clusters kept the cached 0.1.2 (old 1Gi) and ClickHouse OOMed the bell - `/notifications` query — so the fix was re-shipped as a new version (`Chart.yaml` comment, the - `0.1.5` history). Never overwrite a published version; bump it. -- **ClickHouse resources only count under `cluster.spec`.** Upstream `clickstack` 3.0.0 ignores - top-level `clickhouse.resources` / `clickhouse.image` / `clickhouse.persistence`. Put resources at - `clickstack.clickhouse.cluster.spec.containerTemplate.resources`; a top-level `resources:` is INERT - and silently leaves ClickHouse at the operator default (~1Gi) → OOM. -- **sse-proxy is single-replica on purpose.** Don't bump `replicaCount` without gating readiness on - poller health and adding session affinity, or a subset of users gets stuck 503s on the bell. -- **otel credentials must match the ClickHouse user.** `otelCredentials.username/password` must equal - the `otelcollector` user clickstack provisions; a mismatch silently fails ClickHouse writes. -- **Keep the bundled otel-collector disabled.** Re-enabling `clickstack.otel-collector` brings back the - crashlooping OpAMP collector and duplicates ingestion without `krateo.io/composition-id`. -- **`KrateoObservability` is generated, not authored.** Don't look for a CRD YAML to edit — change the - chart values surface and let `core-provider` re-derive the type (see [crds.md](crds.md)). - -## See also - -- [overview.md](overview.md) — chart layout, the CompositionDefinitions, what gets deployed. -- [crds.md](crds.md) — why this component owns no hand-authored CRD; the generated composition type. -- Code repo runtime view: `krateo-platformops/otel-collector` - [`docs/llms.txt`](https://github.com/krateo-platformops/otel-collector/blob/main/docs/llms.txt) - (the custom OTel collector, the `compositionresolver` processor, the sse-proxy poller/hub). diff --git a/examples/observability-composition/README.md b/examples/observability-composition/README.md new file mode 100644 index 0000000..fb85d43 --- /dev/null +++ b/examples/observability-composition/README.md @@ -0,0 +1,46 @@ +--- +type: Example +title: Deploy the ClickStack wrapper as a Krateo composition +description: Register the krateo-observability CompositionDefinition, then create a KrateoObservability CR with a minimal block-style values override. +resource: oci://ghcr.io/krateo-platformops/charts/krateo-observability +tags: [composition, clickstack, observability] +timestamp: 2026-08-07T00:00:00Z +--- + +# Deploy the ClickStack wrapper as a composition + +Registers the wrapper chart with `core-provider` and creates one instance. This is the +standalone path — under a full installer deploy the installer owns the +CompositionDefinition and you create nothing by hand ([usage](../../docs/usage.md)). + +## Preconditions + +- `core-provider` running (a stock Krateo installer deploy, `bootstrap` profile is enough). +- The **ClickHouse operator** and the **MongoDB community operator** installed — the chart + renders a `ClickHouseCluster` and a `MongoDBCommunity` CR but ships neither CRD + ([api](../../docs/api.md)). Under the installer these are the `clickhouse-operator` and + `mongodb-operator` components. +- A `krateo-system` namespace, and a StorageClass matching + `spec.global.storageClassName` in [composition.yaml](./composition.yaml) (`standard` + there, for kind; the chart default is GKE's `standard-rwo`). + +## Apply + +```sh +kubectl apply -f ../../compositiondefinition.yaml +# wait for core-provider to generate the CRD, then: +kubectl apply -f ./composition.yaml +``` + +## Verify + +```sh +kubectl get compositiondefinition krateo-observability -n krateo-system # Ready: True +kubectl get krateoobservabilities.composition.krateo.io -n krateo-system +kubectl get pods -n krateo-system -l app.kubernetes.io/instance --show-labels | grep clickstack +``` + +The composition installs the wrapper chart: ClickHouse (+Keeper), HyperDX, MongoDB, the +`otel-clickhouse-credentials` Secret and the `krateo-clickstack-api` Service. The +apiVersion in `composition.yaml` (`v0-1-11`) tracks the pinned chart version — if you bump +the CompositionDefinition's `spec.chart.version`, regenerate the CR accordingly. diff --git a/examples/observability-composition/composition.yaml b/examples/observability-composition/composition.yaml new file mode 100644 index 0000000..3d6b0ea --- /dev/null +++ b/examples/observability-composition/composition.yaml @@ -0,0 +1,18 @@ +# A KrateoObservability composition instance. The apiVersion is DERIVED from the +# chart version the CompositionDefinition pins (0.1.11 -> v0-1-11): bumping the pin +# changes this apiVersion. The spec is the chart's values, typed by +# charts/krateo-observability/values.schema.json — block YAML, only overrides needed. +apiVersion: composition.krateo.io/v0-1-11 +kind: KrateoObservability +metadata: + name: observability + namespace: krateo-system +spec: + global: + # Match your cluster's StorageClass (chart default standard-rwo is the GKE default; + # on kind use standard). + storageClassName: standard + hyperdxLoadBalancer: + # No LoadBalancer on local clusters — reach HyperDX via the krateo-clickstack + # ClusterIP Service (:3000) instead. + enabled: false