Skip to content

feat: add support for multiple labeled clickhouse read clusters#3693

Open
amokan wants to merge 12 commits into
mainfrom
adammokan/o11y-2186-add-logflare-support-for-multiple-clickhouse-read-clusters
Open

feat: add support for multiple labeled clickhouse read clusters#3693
amokan wants to merge 12 commits into
mainfrom
adammokan/o11y-2186-add-logflare-support-for-multiple-clickhouse-read-clusters

Conversation

@amokan

@amokan amokan commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Why?

Every ClickHouse query currently lands on a single cluster, so a heavy workload such as dashboard log scans or a burst of MCP/agent queries can potentially starve resources away from competing queries.

This change allows multiple read-only clusters to be configured for a backend and queries to be routed using an optional LF-ENDPOINT-CLICKHOUSE-READ-CLUSTER-LABEL header provided to an endpoint, similar to the existing LF-ENDPOINT-BIGQUERY-RESERVATION header. Ultimately allowing us to isolate certain classes of queries to their own backing read clusters.

Key Changes

  • ClickHouse backends can now define multiple labeled read clusters via a read_only_urls config option as well as a default_read_cluster label to fall back to.
  • The adaptor reads the header into opts[:read_cluster] and selects a per-label read pool, running one supervised ConnectionManager and pool per {backend, label}. This follows the existing BigQuery reservation header → opts → adaptor path.
  • Resolution degrades gracefully so an unknown or absent label falls back to the default cluster (or a legacy read URL when nothing is configured). An unhealthy labeled pool retries once on the default. With no header and no read_only_urls config, behavior is identical to today, so this is a no-op until a backend has opted in to this feature.
  • Updates the backend-form LV to handle the multiple read only cluster/label situation.

Additional Notes

  • The existing read_only_url config option will be removed in a later PR

@amokan
amokan requested a review from djwhitt July 17, 2026 00:37
@amokan
amokan requested review from Baishan and chasers July 17, 2026 00:37
Comment thread lib/logflare_web/live/backends/read_cluster_urls_component.ex Outdated
Comment thread lib/logflare/backends/adaptor/clickhouse_adaptor.ex
@amokan
amokan requested a review from djwhitt July 17, 2026 23:17

@Ziinc Ziinc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the config handling of read_cluster_url feels like a code smell, doesn't integrate very well with the label mapping system (which i'm fine with)

Comment thread lib/logflare_web/live/backends/read_cluster_urls_component.ex Outdated
Comment thread test/logflare_web/live/backends/backends_live_test.exs

@Ziinc Ziinc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still needs the header tweak

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants