From 5582ba7ee9088b7c3509c9f93bd5cf8cf46df339 Mon Sep 17 00:00:00 2001 From: Jake Jurek Date: Sun, 7 Jun 2026 15:35:19 -0700 Subject: [PATCH] fix(clickhouse): restore network listening hidden by config.d bind-mount The clickhouse-server image ships config.d/docker_related_config.xml, which sets listen_host to 0.0.0.0/:: so the server is reachable over the network. Bind-mounting the host's config.d (to deliver quiet-logs.xml) replaces the image's config.d directory and hides that file, so on the next restart the server falls back to its localhost-only default: 8123/9000 stop answering externally while SSH and local clients still work. Re-supply the listen config as config.d/listen.xml so a bind-mounted config.d keeps the instance publicly reachable. Access stays gated by the security group (8123/9000 admin CIDRs) and admin.xml networks. Note: aws_instance has lifecycle.ignore_changes = [user_data], so this takes effect when the instance is rebuilt, not on apply. The live box was hotfixed with the same file; this makes it durable across instance replacement. Co-Authored-By: Claude Opus 4.8 --- infra/modules/clickhouse-ec2/user-data.sh.tftpl | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/infra/modules/clickhouse-ec2/user-data.sh.tftpl b/infra/modules/clickhouse-ec2/user-data.sh.tftpl index 651f0d0..08152fa 100644 --- a/infra/modules/clickhouse-ec2/user-data.sh.tftpl +++ b/infra/modules/clickhouse-ec2/user-data.sh.tftpl @@ -85,6 +85,23 @@ EOF # admin.xml this stays world-readable rather than chmod 600). chown 101:101 /etc/clickhouse-server/config.d/quiet-logs.xml +# The image ships /etc/clickhouse-server/config.d/docker_related_config.xml, +# which sets listen_host to 0.0.0.0/:: so the server is reachable over the +# network. Bind-mounting this config.d directory (below) REPLACES the image's +# directory and hides that file, so the server falls back to its localhost-only +# default and 8123/9000 stop answering externally (local clients still work). +# Re-supply the listen config here so a bind-mounted config.d keeps the box +# publicly reachable. listen_try=1 lets it boot if one address family is absent. +cat >/etc/clickhouse-server/config.d/listen.xml <<'EOF' + + 0.0.0.0 + :: + 1 + +EOF + +chown 101:101 /etc/clickhouse-server/config.d/listen.xml + # --network=host: nothing else on this box uses 8123 or 9000, so skip the # docker-proxy NAT layer. --restart=always: container survives reboots # (docker.service is enabled above). --ulimit nofile=262144: ClickHouse