Skip to content

fix(sensor): acknowledge OTLP delivery independently of local capture - #132

Open
barisozbas wants to merge 1 commit into
wt/sensor-diagnosticsfrom
wt/otel-delivery-reliability
Open

barisozbas wants to merge 1 commit into
wt/sensor-diagnosticsfrom
wt/otel-delivery-reliability

Conversation

@barisozbas

@barisozbas barisozbas commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Drain bounded log batches and reconcile submitted/exported counts so the SDK queue
cannot silently drop large captures. Validate protobuf acknowledgements, including
HTTP-success partial rejection, before marking delivery successful.

Track successful session snapshots in atomic hash-only destination checkpoints,
independently of local JSON files. Retry unacknowledged sessions on a later run;
continue sending sensor health even when sessions are already acknowledged.
Document at-least-once semantics, credential configuration, and retry limitations.

This builds on #131 (which builds on #130). Review and merge in that order; the
main-only CI matrix will run after retargeting this PR to main.

What type of PR is this?

  • Bug Fix
  • Documentation Update

Related issue: None; OTLP delivery reliability.

Why? A full background queue can drop records, while existing local session
files can prevent retries after remote delivery fails. HTTP success alone also
does not mean an OTLP collector accepted every record.

Test Plan

Synthetic tests exercise queue overflow protection, dropped-record detection,
partial collector acknowledgements, retries, checkpoint failures, authentication
scope, and health-only runs. No live sessions or collectors are used.

  • All 454 combined sensor tests pass locally on Python 3.9 and 3.12.
  • A 5,122-record mixed session/health export retains every record in bounded batches.
  • A simulated partial rejection leaves delivery unacknowledged; the next run retries;
    the following run sends health without resending the acknowledged session.
  • Base-only installation imports the CLI and displays help without OpenTelemetry
    or requests installed. Ruff, builds, whitespace, and pre-publish validation pass.

Potential risks: Delivery is at least once, not exactly once. Partial collector
rejection can duplicate accepted records on a later run; fix persistent rejection
before rerunning. This is not a persistent payload queue: retries depend on source
sessions still being available and within the configured lookback. Checkpoints
contain only hashes and are bypassed with --no-save. In-place credential-file
rotation requires clearing the destination checkpoint; opaque dynamic HTTP
credential-provider plugins now fail explicitly instead of being silently bypassed.
Native collector deployments and mTLS handshakes were not exercised.

Revert Plan

Revert this commit to restore the previous exporter. Hash-only checkpoint files
remain harmless and are ignored by the previous version.

Summary:
Drain bounded log batches and reconcile submitted/exported counts so the SDK queue
cannot silently drop large captures. Validate protobuf acknowledgements, including
HTTP-success partial rejection, before marking delivery successful.

Track successful session snapshots in atomic hash-only destination checkpoints,
independently of local JSON files. Retry unacknowledged sessions on a later run;
continue sending sensor health even when sessions are already acknowledged.
Document at-least-once semantics, credential configuration, and retry limitations.

This builds on #131 (which builds on #130). Review and merge in that order; the
main-only CI matrix will run after retargeting this PR to main.

Test Plan:
Synthetic tests exercise queue overflow protection, dropped-record detection,
partial collector acknowledgements, retries, checkpoint failures, authentication
scope, and health-only runs. No live sessions or collectors are used.

Revert Plan:
Revert this commit to restore the previous exporter. Hash-only checkpoint files
remain harmless and are ignored by the previous version.
@barisozbas
barisozbas marked this pull request as ready for review September 19, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant