[kafka-connect]: Add troubleshooting entry on handling large connection counts#6158
[kafka-connect]: Add troubleshooting entry on handling large connection counts#6158kurnoolsaketh wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
3 Skipped Deployments
|
| High insertion frequency may result in many open connections to your database. This is common in large task count or distributed connector deployments where the insert rate to a single ClickHouse instance is high. In ClickHouse Cloud, a common symptom of this issue is requests being rate limited by the cloud proxy/load balancer. | ||
|
|
||
| Some strategies to reduce the number of open connections are: | ||
| 1. Adjust the connection pool settings on the Java client (note that these may reduce overall throughput): |
There was a problem hiding this comment.
@chernser are java client options configurable on the connector?
There was a problem hiding this comment.
yes, they are configured via jdbcConnectionProperties this is similar to JDBC url. However if configuration has some unknown property it may be treated as ClickHouse settings for V1
There was a problem hiding this comment.
this is a little confusing to me - are you saying user can set max_open_connections by passing it in jdbcConnectionProperties? This is not a JDBC property though.
| transforms.keyToValue.field=_key | ||
| ``` | ||
|
|
||
| #### "There are too many open connections to my ClickHouse instance" {#too-many-open-connections} |
There was a problem hiding this comment.
I would name it shorter "Too many DB connections".
| ``` | ||
|
|
||
| #### "There are too many open connections to my ClickHouse instance" {#too-many-open-connections} | ||
| High insertion frequency may result in many open connections to your database. This is common in large task count or distributed connector deployments where the insert rate to a single ClickHouse instance is high. In ClickHouse Cloud, a common symptom of this issue is requests being rate limited by the cloud proxy/load balancer. |
There was a problem hiding this comment.
This needs careful explanation because there may be many records and we are fine with that and there can be many small batches.
Current text declares problem way too broadly and do not explain why it happens.
It may be worth having structuring the record in way
<Description>
Symptoms on prem:
Symptoms on cloud:
Metrics to check:
<What configuration to change, how to troubleshoot>
There was a problem hiding this comment.
Symptoms on prem:
AFAIK, we haven't seen a case of this happening in on-prem yet since this issue was originally raised by a cloud customer. Depending on if and how their LB is configured, this may not even be an issue for on-prem customers.
But, I agree with your suggestion to include Metrics to check - i have added it 👍 thanks!
| - `max_open_connections`: defaults to 10. Reducing this will bound the number of open connections per task. | ||
| - `connection_ttl`: defaults to -1 (no ttl). Setting this to >0 will eagerly reclaim connections after the ttl expires. | ||
|
|
||
| See the [Java client Connection & Endpoints configuration tab](https://clickhouse.com/docs/integrations/language-clients/java/client#configuration) for more details. |
There was a problem hiding this comment.
Java client is not configured via endpoint.
But troubleshooting guide expects that user know how to configure client in the connector - if it is not explained before then it should be fixed.
There was a problem hiding this comment.
But troubleshooting guide expects that user know how to configure client in the connector
Agreed, I am just linking it here for ease of reference.
|
|
||
| See the [Java client Connection & Endpoints configuration tab](https://clickhouse.com/docs/integrations/language-clients/java/client#configuration) for more details. | ||
|
|
||
| 2. Increase `bufferCount`: in high data volume/throughput deployments, this will increase the number of records buffered between inserts and reduce the frequency of insert queries sent to your database. This will reduce the number of new connections needed to write to your database. |
There was a problem hiding this comment.
Is ti really for data volume and throughput?
This is not - before that sink was handling big batches just fine.
There was a problem hiding this comment.
yes this is incorrect. I've removed in high data volume/throughput deployments, , you are right that not high data volume/tput cases will encounter this issue.
|
closing this PR until we are more certain of the root cause of the connection count explosion. Will reopen it once we can flesh out this section more. |
2 participants
Summary
Adds an entry addressing an edge case in which high data volume connector deployments may result in a large number of open connections to ClickHouse.