Skip to content

feat: replace permanent event-delivery disable with a bounded cooldown (SDK-3374) - #68

Merged
tanderson-ld merged 7 commits into
v3from
ta/sdk-3374/evtproc-cooldown
Nov 13, 2026
Merged

tanderson-ld merged 7 commits into
v3from
ta/sdk-3374/evtproc-cooldown

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

What

Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278): on an unexpected delivery failure the processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.

  • Windows run 5, 10, 20, 40, then 60 minutes (capped at the 1 hour ceiling), stepping once per failed probe. They are deterministic: RETRY 1.4.3 permits zero jitter for a component whose attempts are internally scheduled, and the flush schedule itself paces attempts, which is also why the windows are not floored at the flush interval (the processor can never try faster while failing than while healthy, because the ticker never speeds up).
  • While a window is open, every delivery (scheduled, flush(), or shutdown) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be, firing the failed-send metric (with DiscardedInCooldown set and no status code) at most once per flush of either kind. The cooldown check only skips the wasted network and serialization work, per 1.8.5 and its closing note. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context-deduplication cache is cleared both when a window opens and when a discarded batch contained an index or identify event, so post-window payloads can introduce afresh any contexts whose index or identify events were lost with the refused or discarded batches (a batch dropped after a recoverable failure still leaves the cache poisoned; that pre-existing gap is being filed separately).
  • Recording is never gated (1.8.8): evaluations keep updating summary counters and intake continues during a window.
  • A successful probe returns delivery to the flush cadence immediately (1.8.3, RETRY 1.4.8); retry state resets only after 2 consecutive successful deliveries (1.8.4).
  • No cooldown state persists across restarts (1.8.10). The recoverable-failure retry (1.7.2) and the 413 carve-out (1.7.3.1) are unchanged.

API

  • EventSenderResult.Unexpected (new): the spec's failure classification, set by this package's senders for non-recoverable, non-413 4xx responses.
  • MustShutDown is deprecated as equivalent to Unexpected. The senders here keep setting both until known consumers migrate; consumers should call the new IsUnexpectedFailure(), which accepts either field (so custom senders written against the old API still get cooldown behavior).
  • EventSendFailureMetadata gains the additive DiscardedInCooldown field, set only for batches disposed of by the cooldown's in-window no-op and never for a batch that was actually sent. Without it, failure-metric consumers see every discard as a status-zero failure, the signature of a network error, so a bad SDK key would read as a network outage for the length of every window. Consumers that ignore the field see exactly the previous behavior. Nothing else on the public surface changes; EventMetrics and the FlushBlocking contract are untouched. One behavioral note for FlushBlocking: a true return has never guaranteed delivery (failed sends also returned true), and an in-window flush completes as a no-op the same way.

Design notes

  • With five concurrent flush workers, delivery outcomes are correlated to the window era they were dispatched in via an internal opaque windowToken; an outcome from a delivery in flight across a window boundary cannot open, lengthen, or end a window or step the ladder. One exception, per RETRY 1.8.3: a stale failure still breaks the consecutive-success streak, since a failure between two successes makes them non-consecutive regardless of which round dispatched them (unstamped outcomes remain fully inert). This confines window and ladder movement to actual probes. The tracker is pure state (clock injection only), inspired by the shared RetryState controller pattern but reduced to this component's needs.
  • Diagnostic event delivery is not governed by the cooldown (flush workers do not publish results for diagnostic payloads); its RETRY alignment is separate spec work.
  • A payload already in flight or queued when a window opens may still produce one POST; its result cannot affect cooldown state. A delivery round that mixes concurrent successes and failures at a window boundary can defer the next window by at most one flush interval; the ladder position is preserved.
  • Logging: one Error when a window opens (with its length), a Warn for the first discarded batch of each window (Debug thereafter), and an Info when delivery resumes.

Testing

  • Unit: tracker and dispatcher suites driven by a fake clock, race-clean, including window doubling, reset semantics, in-window disposal and context-introduction recovery, stale-outcome handling, and log behavior.
  • Contract: all four events/delivery cooldown tests pass against the Go server SDK test service advertising event-delivery-cooldown, including the long-running real-window resume test (requires feat(events): add event-delivery cooldown contract tests sdk-test-harness#526; the go-server-sdk capability and dependency bump ship separately after this releases).
  • Four multi-agent review rounds with adversarial proof tests were run against this change (the fourth against the committed PR, with a mutation sweep); all findings were fixed or explicitly dispositioned.

Release notes

  • Minor (feat). Behavior change worth calling out: a custom EventSender returning MustShutDown: true now causes a temporary cooldown rather than a permanent stop.

Follow-ups (deliberately not in this PR)

  • Relay Proxy adoption: it reads MustShutDown directly and keeps its current behavior until it migrates to IsUnexpectedFailure(), and its metrics recorder needs to map DiscardedInCooldown when it bumps this dependency. Both are captured in a relay adoption story under the same epic as this work.
  • A pre-existing FlushBlocking completion gap (the empty-outbox and worker-contention paths never close the reply channel) is unchanged by this PR and will be filed separately.

Note

Overview
Replaces the old permanent stop-on-MustShutDown behavior with a temporary delivery cooldown after unexpected send failures (e.g. 401/403). The event processor now enters escalating backoff windows (5→10→20→40→60 minutes), discards in-window flush batches without network calls, and automatically resumes after a successful probe or when the window ends.

Adds deliveryCooldownTracker with window tokens so concurrent or stale in-flight delivery results cannot extend, shorten, or reset cooldown incorrectly; the dispatcher prioritizes draining send results before handling flushes. In-window discards record failed-send metrics with new DiscardedInCooldown metadata, clear context-dedup state when index/identify introductions are lost, and log Warn once per window.

API: New EventSenderResult.Unexpected and IsUnexpectedFailure() (still honors legacy MustShutDown); senders set both for compatibility. HTTP 413 is explicitly treated as an ordinary failure, not an unexpected one.

Reviewed by Cursor Bugbot for commit f5997ea. Bugbot is set up for automated code reviews on this repo. Configure here.

…n (SDK-3374)

On an unexpected delivery failure (a non-recoverable HTTP status other than
413, such as 401 or 403), the event processor now enters a bounded,
self-recovering cooldown window instead of permanently disabling delivery
for the lifetime of the SDK instance, per the EVTPROC spec's Section 1.8.

Windows double from 5 minutes to a 1 hour ceiling, one step per failed
probe. In-window deliveries are no-ops that dispose of the batch exactly as
a sent-and-refused payload would be, without the network or serialization
cost. Recording is never gated, retry state resets after two consecutive
successful deliveries, and no cooldown state persists across restarts.

EventSenderResult gains the Unexpected classification field; MustShutDown
is deprecated (and still dual-written by the senders in this package) in
favor of the IsUnexpectedFailure accessor.
@tanderson-ld
tanderson-ld marked this pull request as ready for review November 6, 2026 21:38
@tanderson-ld
tanderson-ld requested a review from a team as a code owner November 6, 2026 21:38
The cooldown gate in triggerFlush now precedes the empty-outbox check, so
an in-window flush with nothing buffered still closes its completion
channel promptly instead of leaving FlushBlocking to time out (or hang
forever at a zero timeout); the discard path internally skips disposal,
metrics, and logging when there is nothing to dispose of.

A result claiming success and an unexpected failure at once is now treated
as an unexpected failure, matching the precedence the processor had when
MustShutDown was the only signal.

Also removes a dead warn-flag reset in the resumed branch, restores the
interval-normalization order, corrects the constants comment to cover
manual flushes, drops a vestigial test flush-interval override, and adds
regression tests for these plus previously untested guards: stale-token
recoverable failures, the zero token, server-time calibration on
unexpected failures, the default clock path, the window-boundary instant,
discard count parity with delivered batches, windows shorter than the
flush interval, and result-drain priority over queued flush messages.
Round-5 review tidies, all in tests and comments:

- The discard-count parity test now drives both production paths (a
  delivered-and-refused batch and an in-window discard of the same inputs)
  and compares the failed-send metric, instead of restating the discard
  formula inline where it could not detect a regression in triggerFlush.
- The queued-flush priority test runs 40 iterations (an unprioritized
  select slips past 12 iterations a few percent of the time) and its
  sender gate is buffered generously so no regression can wedge it.
- New tests pin previously unguarded intentional behavior: the Warn-level
  discard returning after a successfully-ended window, parallel successes
  in one non-boundary round each counting toward the reset threshold, the
  diagnostic batch-size stat zeroing on discards, ladder position
  surviving a recoverably-ended window, and recording staying ungated
  throughout a window.
- The Unexpected field doc now states the classification criterion in
  terms of the service refusing requests regardless of payload, so it no
  longer describes a 413, which the senders deliberately leave ordinary.
- Comment fixes: the reset threshold counts successful deliveries rather
  than cadence rounds outside window boundaries, and in-window no-ops
  cover shutdown flushes as well as scheduled and manual ones.
Comment thread delivery_cooldown.go Resolved
Comment thread event_processor.go Resolved
Comment thread event_processor.go Resolved
Comment thread delivery_cooldown.go Resolved
Comment thread delivery_cooldown.go Resolved
Comment thread event_processor.go Resolved
Per review feedback, the new files' comments narrated adjacent design
instead of stating the constraints the code cannot show. Each comment is
now one or two lines; the rationale behind the window ladder, the token
staleness rules, and the boundary-round handling lives in the PR
discussion and in the tests that pin those behaviors.
…3374)

EventSendFailureMetadata gains DiscardedInCooldown, set only for batches
disposed of by the cooldown's in-window no-op and never for a batch that
was actually sent. Without it, a consumer recording failure metrics sees
every in-window discard as a status-zero failure, which is otherwise the
signature of a network error, so a bad SDK key reads as a network outage
for the length of every window. The field is additive; consumers that
ignore it see exactly the previous behavior.
…(SDK-3374)

Two review findings, both on the cooldown's interaction with concurrent
deliveries.

When a window opens, the dispatcher now clears the context key cache:
the refused batch's index and identify events were lost with it, so
without the wipe, post-window payloads would reference contexts the
service never received, until the cache's own expiry healed it. The wipe
runs once per window-open (sibling and straggler failures arrive with
stale tokens) and can only err toward redundant index events, the same
cost the periodic cache expiry already incurs by design.

A delivery failure now breaks the consecutive-success streak even when
its token is stale: a failure between two successes makes them
non-consecutive regardless of which round dispatched it, so the ladder
no longer resets across a failure (success, failure, success previously
counted as two consecutive successes when the failure's token had gone
stale). Stale outcomes still cannot open, step, or end windows, and
unstamped tokens remain inert, so an outcome with no delivery behind it
cannot touch the streak. Residual: a failure that completes after the
streak has already reached the threshold does not retroactively undo the
reset; the measurement is in completion order.
@tanderson-ld
tanderson-ld requested a review from beekld November 12, 2026 20:35
tanderson-ld added a commit to launchdarkly/java-core that referenced this pull request Nov 12, 2026
…fixes (SDK-3375)

Diagnostics are now fully out of scope for the event-delivery cooldown, matching
the Go implementation. A diagnostic delivery no longer mints a window token or
feeds the tracker; it only calibrates the clock from the Date header as before.
This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but
rejects /diagnostic would otherwise open a new window on every periodic
diagnostic and discard analytics the service would accept. Backpressure on the
diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here.

Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68):

- When a window opens, forget every context key so post-window payloads
  re-introduce contexts whose index/identify events were lost with the refused
  batch. The window opens on a flush worker thread while the deduplicator is
  only safe to touch from the dispatcher thread, so the clear is routed through
  the inbox as a FLUSH_USERS message.
- A delivery failure now breaks the consecutive-success streak even when its
  token is stale, so success/failure/success no longer resets the ladder across
  a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a
  window; unstamped (null) tokens remain fully inert.
- Comment trims, and the EventSender.failure() javadoc no longer claims it does
  not affect the delivery schedule.

Tests updated for the window-open wipe: the three diagnostic-coupling tests
become decoupling tests (the straddle case is gone with the token), and the
dedup tests assert via index-event presence rather than exact flush counts.

@beekld beekld left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of nits, but otherwise, lgtm

Comment thread event_processor.go Resolved
Comment thread delivery_cooldown_test.go Resolved
Fix the window-open wipe comment wording (and a typo), and explain why the
refused-window-opening-batch test syncs twice in a row.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium.

Automated risk review (dry run): Medium

This PR would not be automatically approved. Human approval required.

Why: It replaces permanent event-delivery shutdown with a bounded cooldown in delivery_cooldown.go and event_processor.go, and it changes the public sender result API in interfaces.go (Unexpected, IsUnexpectedFailure(), DiscardedInCooldown). That is a non-trivial SDK behavior and API change, so it is Medium even with the new tests and a clean Bugbot run.

If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.

Open in Web View Automation 

Sent by Cursor Approval Agent: Dry-Run PR Approver

@cursor cursor Bot added the risk:medium label Nov 13, 2026
@tanderson-ld
tanderson-ld merged commit af95923 into v3 Nov 13, 2026
15 checks passed
@tanderson-ld
tanderson-ld deleted the ta/sdk-3374/evtproc-cooldown branch November 13, 2026 13:49
@github-actions github-actions Bot mentioned this pull request Nov 13, 2026
tanderson-ld pushed a commit that referenced this pull request Nov 13, 2026
🤖 I have created a release *beep* *boop*
---


##
[3.9.0](v3.8.1...v3.9.0)
(2026-11-13)


### Features

* replace permanent event-delivery disable with a bounded cooldown
(SDK-3374)
([#68](#68))
([af95923](af95923))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Cuts the release version from **3.8.1** to **3.9.0** in
`.release-please-manifest.json` and adds a **3.9.0** changelog entry.
> 
> That release note documents the shipped behavior change from
[#68](#68): failed
event delivery now enters a **bounded cooldown/backoff** (via
`deliveryCooldownTracker`) instead of turning off delivery permanently.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
9a4d725. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants