Repository navigation
feat: replace permanent event-delivery disable with a bounded cooldown (SDK-3374) - #68
Conversation
…n (SDK-3374) On an unexpected delivery failure (a non-recoverable HTTP status other than 413, such as 401 or 403), the event processor now enters a bounded, self-recovering cooldown window instead of permanently disabling delivery for the lifetime of the SDK instance, per the EVTPROC spec's Section 1.8. Windows double from 5 minutes to a 1 hour ceiling, one step per failed probe. In-window deliveries are no-ops that dispose of the batch exactly as a sent-and-refused payload would be, without the network or serialization cost. Recording is never gated, retry state resets after two consecutive successful deliveries, and no cooldown state persists across restarts. EventSenderResult gains the Unexpected classification field; MustShutDown is deprecated (and still dual-written by the senders in this package) in favor of the IsUnexpectedFailure accessor.
The cooldown gate in triggerFlush now precedes the empty-outbox check, so an in-window flush with nothing buffered still closes its completion channel promptly instead of leaving FlushBlocking to time out (or hang forever at a zero timeout); the discard path internally skips disposal, metrics, and logging when there is nothing to dispose of. A result claiming success and an unexpected failure at once is now treated as an unexpected failure, matching the precedence the processor had when MustShutDown was the only signal. Also removes a dead warn-flag reset in the resumed branch, restores the interval-normalization order, corrects the constants comment to cover manual flushes, drops a vestigial test flush-interval override, and adds regression tests for these plus previously untested guards: stale-token recoverable failures, the zero token, server-time calibration on unexpected failures, the default clock path, the window-boundary instant, discard count parity with delivered batches, windows shorter than the flush interval, and result-drain priority over queued flush messages.
Round-5 review tidies, all in tests and comments: - The discard-count parity test now drives both production paths (a delivered-and-refused batch and an in-window discard of the same inputs) and compares the failed-send metric, instead of restating the discard formula inline where it could not detect a regression in triggerFlush. - The queued-flush priority test runs 40 iterations (an unprioritized select slips past 12 iterations a few percent of the time) and its sender gate is buffered generously so no regression can wedge it. - New tests pin previously unguarded intentional behavior: the Warn-level discard returning after a successfully-ended window, parallel successes in one non-boundary round each counting toward the reset threshold, the diagnostic batch-size stat zeroing on discards, ladder position surviving a recoverably-ended window, and recording staying ungated throughout a window. - The Unexpected field doc now states the classification criterion in terms of the service refusing requests regardless of payload, so it no longer describes a 413, which the senders deliberately leave ordinary. - Comment fixes: the reset threshold counts successful deliveries rather than cadence rounds outside window boundaries, and in-window no-ops cover shutdown flushes as well as scheduled and manual ones.
Per review feedback, the new files' comments narrated adjacent design instead of stating the constraints the code cannot show. Each comment is now one or two lines; the rationale behind the window ladder, the token staleness rules, and the boundary-round handling lives in the PR discussion and in the tests that pin those behaviors.
…3374) EventSendFailureMetadata gains DiscardedInCooldown, set only for batches disposed of by the cooldown's in-window no-op and never for a batch that was actually sent. Without it, a consumer recording failure metrics sees every in-window discard as a status-zero failure, which is otherwise the signature of a network error, so a bad SDK key reads as a network outage for the length of every window. The field is additive; consumers that ignore it see exactly the previous behavior.
…(SDK-3374) Two review findings, both on the cooldown's interaction with concurrent deliveries. When a window opens, the dispatcher now clears the context key cache: the refused batch's index and identify events were lost with it, so without the wipe, post-window payloads would reference contexts the service never received, until the cache's own expiry healed it. The wipe runs once per window-open (sibling and straggler failures arrive with stale tokens) and can only err toward redundant index events, the same cost the periodic cache expiry already incurs by design. A delivery failure now breaks the consecutive-success streak even when its token is stale: a failure between two successes makes them non-consecutive regardless of which round dispatched it, so the ladder no longer resets across a failure (success, failure, success previously counted as two consecutive successes when the failure's token had gone stale). Stale outcomes still cannot open, step, or end windows, and unstamped tokens remain inert, so an outcome with no delivery behind it cannot touch the streak. Residual: a failure that completes after the streak has already reached the threshold does not retroactively undo the reset; the measurement is in completion order.
…fixes (SDK-3375) Diagnostics are now fully out of scope for the event-delivery cooldown, matching the Go implementation. A diagnostic delivery no longer mints a window token or feeds the tracker; it only calibrates the clock from the Date header as before. This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but rejects /diagnostic would otherwise open a new window on every periodic diagnostic and discard analytics the service would accept. Backpressure on the diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here. Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68): - When a window opens, forget every context key so post-window payloads re-introduce contexts whose index/identify events were lost with the refused batch. The window opens on a flush worker thread while the deduplicator is only safe to touch from the dispatcher thread, so the clear is routed through the inbox as a FLUSH_USERS message. - A delivery failure now breaks the consecutive-success streak even when its token is stale, so success/failure/success no longer resets the ladder across a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a window; unstamped (null) tokens remain fully inert. - Comment trims, and the EventSender.failure() javadoc no longer claims it does not affect the delivery schedule. Tests updated for the window-open wipe: the three diagnostic-coupling tests become decoupling tests (the straddle case is gone with the token), and the dedup tests assert via index-event presence rather than exact flush counts.
beekld
left a comment
There was a problem hiding this comment.
a couple of nits, but otherwise, lgtm
Fix the window-open wipe comment wording (and a typo), and explain why the refused-window-opening-batch test syncs twice in a row.
There was a problem hiding this comment.
Risk: medium.
Automated risk review (dry run): Medium
This PR would not be automatically approved. Human approval required.
Why: It replaces permanent event-delivery shutdown with a bounded cooldown in delivery_cooldown.go and event_processor.go, and it changes the public sender result API in interfaces.go (Unexpected, IsUnexpectedFailure(), DiscardedInCooldown). That is a non-trivial SDK behavior and API change, so it is Medium even with the new tests and a clean Bugbot run.
If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.
Sent by Cursor Approval Agent: Dry-Run PR Approver
🤖 I have created a release *beep* *boop* --- ## [3.9.0](v3.8.1...v3.9.0) (2026-11-13) ### Features * replace permanent event-delivery disable with a bounded cooldown (SDK-3374) ([#68](#68)) ([af95923](af95923)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Cuts the release version from **3.8.1** to **3.9.0** in `.release-please-manifest.json` and adds a **3.9.0** changelog entry. > > That release note documents the shipped behavior change from [#68](#68): failed event delivery now enters a **bounded cooldown/backoff** (via `deliveryCooldownTracker`) instead of turning off delivery permanently. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 9a4d725. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>


What
Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278): on an
unexpecteddelivery failure the processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.flush(), or shutdown) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be, firing the failed-send metric (withDiscardedInCooldownset and no status code) at most once per flush of either kind. The cooldown check only skips the wasted network and serialization work, per 1.8.5 and its closing note. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context-deduplication cache is cleared both when a window opens and when a discarded batch contained an index or identify event, so post-window payloads can introduce afresh any contexts whose index or identify events were lost with the refused or discarded batches (a batch dropped after a recoverable failure still leaves the cache poisoned; that pre-existing gap is being filed separately).API
EventSenderResult.Unexpected(new): the spec's failure classification, set by this package's senders for non-recoverable, non-413 4xx responses.MustShutDownis deprecated as equivalent toUnexpected. The senders here keep setting both until known consumers migrate; consumers should call the newIsUnexpectedFailure(), which accepts either field (so custom senders written against the old API still get cooldown behavior).EventSendFailureMetadatagains the additiveDiscardedInCooldownfield, set only for batches disposed of by the cooldown's in-window no-op and never for a batch that was actually sent. Without it, failure-metric consumers see every discard as a status-zero failure, the signature of a network error, so a bad SDK key would read as a network outage for the length of every window. Consumers that ignore the field see exactly the previous behavior. Nothing else on the public surface changes;EventMetricsand theFlushBlockingcontract are untouched. One behavioral note forFlushBlocking: atruereturn has never guaranteed delivery (failed sends also returnedtrue), and an in-window flush completes as a no-op the same way.Design notes
windowToken; an outcome from a delivery in flight across a window boundary cannot open, lengthen, or end a window or step the ladder. One exception, per RETRY 1.8.3: a stale failure still breaks the consecutive-success streak, since a failure between two successes makes them non-consecutive regardless of which round dispatched them (unstamped outcomes remain fully inert). This confines window and ladder movement to actual probes. The tracker is pure state (clock injection only), inspired by the shared RetryState controller pattern but reduced to this component's needs.Testing
events/delivery cooldowntests pass against the Go server SDK test service advertisingevent-delivery-cooldown, including the long-running real-window resume test (requires feat(events): add event-delivery cooldown contract tests sdk-test-harness#526; the go-server-sdk capability and dependency bump ship separately after this releases).Release notes
EventSenderreturningMustShutDown: truenow causes a temporary cooldown rather than a permanent stop.Follow-ups (deliberately not in this PR)
MustShutDowndirectly and keeps its current behavior until it migrates toIsUnexpectedFailure(), and its metrics recorder needs to mapDiscardedInCooldownwhen it bumps this dependency. Both are captured in a relay adoption story under the same epic as this work.FlushBlockingcompletion gap (the empty-outbox and worker-contention paths never close the reply channel) is unchanged by this PR and will be filed separately.Note
Overview
Replaces the old permanent stop-on-
MustShutDownbehavior with a temporary delivery cooldown after unexpected send failures (e.g. 401/403). The event processor now enters escalating backoff windows (5→10→20→40→60 minutes), discards in-window flush batches without network calls, and automatically resumes after a successful probe or when the window ends.Adds
deliveryCooldownTrackerwith window tokens so concurrent or stale in-flight delivery results cannot extend, shorten, or reset cooldown incorrectly; the dispatcher prioritizes draining send results before handling flushes. In-window discards record failed-send metrics with newDiscardedInCooldownmetadata, clear context-dedup state when index/identify introductions are lost, and log Warn once per window.API: New
EventSenderResult.UnexpectedandIsUnexpectedFailure()(still honors legacyMustShutDown); senders set both for compatibility. HTTP 413 is explicitly treated as an ordinary failure, not an unexpected one.Reviewed by Cursor Bugbot for commit f5997ea. Bugbot is set up for automated code reviews on this repo. Configure here.