Skip to content

Add retry logic and logging for no-deps telemetry collection - #34

Open
frank-novak wants to merge 3 commits into
mainfrom
frank-telemetry-retry
Open

frank-novak wants to merge 3 commits into
mainfrom
frank-telemetry-retry

Conversation

@frank-novak

Copy link
Copy Markdown
Contributor

kubectl port-forward tunnels have been observed to serve one connection (the /healthz gate) then refuse the next, especially across kubectl/API-server version skew, silently producing empty telemetry files. Retry on connection-refused and surface a warning when a file still comes out empty.

Mirrors the retry/logging fix already applied to support-bundle.sh:
kubectl port-forward tunnels have been observed to serve one
connection (the /healthz gate) then refuse the next, especially
across kubectl/API-server version skew, silently producing empty
telemetry files. Retry on connection-refused and surface a warning
when a file still comes out empty.

@ihorponom ihorponom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems curl --retry doesn't truncate the output file between attempts when writing via shell redirection (>). If the tunnel drops mid-transfer (not just before connecting) and a retry succeeds, the new response gets appended after the leftover partial bytes from the failed attempt, producing a corrupted-but-non-empty JSON file that the new [! -s "$f"] empty check won't catch.

@frank-novak

Copy link
Copy Markdown
Contributor Author

Updated so the output paths are used instead, good catch!

@ihorponom

ihorponom commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Updated so the output paths are used instead; good catch!

All seems good, but the original commit was in support-bundle/support-bundle-no-deps.sh; the new commit (with the -o fix) landed in support-bundle/support-bundle.sh. support-bundle-no-deps.sh still uses shell redirection (> "$file") and is still exposed to the same issue. Maybe we should align them both.

Wrap curl calls in set +e/-e so exhausted retries don't abort the
whole script under set -e, and use the same empty_file_check
function + array pattern as support-bundle.sh.

@ihorponom ihorponom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants