fix: durable ClickHouse retention — 15Gi volume, 3d system-log TTL, 7d otel exporter TTL - #30
Merged
Merged
Conversation
…d otel_* exporter ttl
Incident: on a live bench the ClickHouse 10Gi data volume filled in 8 days —
no TTL anywhere (otel_* tables unbounded; system.* log tables unbounded at
~3.6GiB) — so every insert failed with code 243 (NOT_ENOUGH_SPACE), the OTel
collectors buffered + OOM-crash-looped, and telemetry went fully dead. Live
relief was applied by hand (PVC 10->15Gi, system tables truncated, partitions
dropped, ALTER TABLE ... MODIFY TTL 7d on otel_*). This makes fresh installs
never hit it:
- krateo-observability 0.1.9:
* cluster.spec.dataVolumeClaimSpec.resources.requests.storage: 15Gi
(minimal deep-merge override of the upstream clickstack 10Gi default;
accessModes still inherited from upstream)
* settings.extraConfig: 3-day engine ttl on query_log, text_log, metric_log,
part_log, asynchronous_metric_log, trace_log — same operator-native
extraConfig->config.d merge the live http_handlers already proves out
- otel-collector-daemonset 0.1.5 + otel-collector-deployment 0.3.3:
* clickhouse exporter ttl: 168h — 7-day TTL stamped into the otel_* tables
at creation (create_schema is CREATE IF NOT EXISTS, so pre-existing
benches keep their live-ALTERed TTL; this covers fresh installs)
* values.schema.json: ttl enumerated alongside the other exporter keys
Validated: helm lint + helm template clean on all three charts; rendered
ClickHouseCluster CR carries 15Gi + all six system-log ttl entries merged
with the upstream settings; both rendered collector configs carry ttl: 168h.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The installer-release bench's 10Gi ClickHouse volume hit 100% in 8 days (insert-fail 243 → OTel collector OOM loops → telemetry dead). Half the disk was ClickHouse's own unbounded system log tables; the other half untTL'd otel_logs.
Durable fix:
dataVolumeClaimSpec15Gi (matches the live expanded PVC) + six system-logttl: event_date + INTERVAL 3 DAY DELETEentries viasettings.extraConfig(the same passthrough http_handlers already rides).ttl: 168hso fresh installs create otel tables with a 7-day TTL natively (create-time only; existing benches were ALTERed live).Validation: helm lint + template clean on all three charts; rendered ClickHouseCluster carries storage+ttls merged next to upstream config; both relay configs carry the exporter ttl. Note for first fresh install: confirm
SHOW CREATE TABLE system.query_logshows the TTL (operator-binary rendering of extraConfig is the one unverifiable-offline step).🤖 Generated with Claude Code