Skip to content

feat: add single retry for heartbeat on network failures - #28

Open
crimson-gao-agents wants to merge 2 commits into
aliyun:masterfrom
crimson-gao-agents:fix/heartbeat-retry-network-errors
Open

feat: add single retry for heartbeat on network failures#28
crimson-gao-agents wants to merge 2 commits into
aliyun:masterfrom
crimson-gao-agents:fix/heartbeat-retry-network-errors

Conversation

@crimson-gao-agents

Copy link
Copy Markdown
Contributor

Background

The heartbeat thread in LogHubHeartBeat currently has no retry logic. When transient network failures occur (connection drops, timeouts, DNS issues), the heartbeat fails immediately without any retry attempt. This can cause unnecessary consumer group shard rebalancing during brief network hiccups.

Solution

Added a single retry mechanism in LogHubHeartBeat.heartBeat() that:

  • Retries only for client-side/network errors (LogException.httpCode <= 0)
  • Does NOT retry for server errors (4xx, 5xx) since those indicate real issues where retrying is unlikely to help
  • Waits 1 second before the retry attempt
  • Adds structured logging to distinguish retryable vs non-retryable failures for production observability

Changes

  • Added isRetryable(LogException) method to determine if an error warrants retry
  • Added handleHeartbeatFailure(long, Exception) helper to consolidate failure handling
  • Wrapped the heartbeat call with retry logic (max 2 attempts total)
  • Added import for com.aliyun.openservices.log.exception.LogException

This change is backward compatible and does not modify any public APIs.

@CLAassistant

CLAassistant commented Mar 27, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Crimson Gao Agents seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@crimson-gao-agents
crimson-gao-agents force-pushed the fix/heartbeat-retry-network-errors branch from 3bb288c to c08168f Compare March 28, 2026 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants