Repository navigation
feat: replace permanent event-delivery disable with a bounded cooldown - #256
tanderson-ld wants to merge 10 commits into
Conversation
…sifier DefaultEventSender now classifies HTTP failures with HttpErrors.classifyHttpFailure instead of the deprecated isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers. Behavior and log output are unchanged: the set of statuses treated as unexpected is identical, unexpected failures still skip the retry and still log at error level with the same wording, and transport-level exceptions keep their warn-and-retry-once handling.
…n (SDK-3375) On an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor now enters a bounded, self-recovering cooldown window instead of permanently disabling delivery for the lifetime of the SDK instance, per the EVTPROC spec's Section 1.8. Windows double from 5 minutes to a 1 hour ceiling, one step per failed probe. In-window flushes of any kind are no-ops that dispose of the batch exactly as a sent-and-refused payload would be, without the network or serialization cost. Recording is never gated, retry state resets after two consecutive successful analytics deliveries, and no cooldown state persists across restarts. Diagnostic sends are not gated by the window; an unexpected diagnostic failure opens one, as it previously disabled delivery, but diagnostic successes do not drive the delivery schedule. EventSender.Result gains the unexpected classification with factory methods and the isUnexpectedFailure accessor; the legacy constructor and isMustShutDown are deprecated and still dual-written for consumers that have not migrated.
Diagnostics obtain their delivery token through the same atomic cooldown-check-plus-issue call that analytics flushes use, so a delivery dispatched inside a window carries no token at all and every pre-window token goes stale when the window opens. All three in-cooldown guards are therefore uniformly defensive; one of them previously claimed to be reachable from in-window diagnostic failures.
MockEventSender enqueues the captured payload before it reads the result field, so a test that awaits the startup diagnostic and then immediately changes the sender's result can hand the new result to the still-running diagnostic task. The startup diagnostic then fails unexpectedly and opens the cooldown window itself, the test's own window-opening flush is discarded in-window, and the test times out waiting for it - the fake clock never advances, so the accidental window never elapses. Reproduced deterministically by widening the descheduling gap; seen once in CI. The three tests that change the result right after awaiting the startup diagnostic now wait for the diagnostic task to finish first, using the single-threaded executor barrier the file already uses elsewhere.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8358410. Configure here.
beekld
left a comment
There was a problem hiding this comment.
AI review: The description says the Android SDK's own DirectEventProcessor reads the legacy flag and keeps its permanent disable unchanged. Android's main branch has no DirectEventProcessor; it exists only on unmerged branches. Android main builds this module's DefaultEventSender and DefaultEventProcessor directly (ComponentsImpl.java), pinned at internal 1.9.0. So Android's next bump of this dependency picks up the cooldown and the ungated diagnostics without a coordinated change.
Relatedly, nothing in either repo reads the dual-written legacy flag. EventSenderWrapper only writes it, so it isn't a consumer that has not migrated.
|
AI review comment: Mirror work pending from the Go PR's human review (launchdarkly/go-sdk-events#68, threads by beekld). The Go side landed these in ff661e0 and fcf7d6f; this PR needs the equivalents before merge. Parking the checklist here rather than pushing now.
|
…fixes (SDK-3375) Diagnostics are now fully out of scope for the event-delivery cooldown, matching the Go implementation. A diagnostic delivery no longer mints a window token or feeds the tracker; it only calibrates the clock from the Date header as before. This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but rejects /diagnostic would otherwise open a new window on every periodic diagnostic and discard analytics the service would accept. Backpressure on the diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here. Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68): - When a window opens, forget every context key so post-window payloads re-introduce contexts whose index/identify events were lost with the refused batch. The window opens on a flush worker thread while the deduplicator is only safe to touch from the dispatcher thread, so the clear is routed through the inbox as a FLUSH_USERS message. - A delivery failure now breaks the consecutive-success streak even when its token is stale, so success/failure/success no longer resets the ladder across a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a window; unstamped (null) tokens remain fully inert. - Comment trims, and the EventSender.failure() javadoc no longer claims it does not affect the delivery schedule. Tests updated for the window-open wipe: the three diagnostic-coupling tests become decoupling tests (the straddle case is gone with the token), and the dedup tests assert via index-event presence rather than exact flush counts.
…wedge close() (SDK-3375) A flush worker caught only Exception around its send and response handling, so an Error (StackOverflowError, NoClassDefFoundError, a test AssertionError, etc.) escaping the loop body skipped the busy-worker-count decrement. The count stayed elevated and close() blocked forever in waitUntilAllFlushWorkersInactive. Moving the decrement into a finally guarantees it runs even when an Error ends the worker thread. Pre-existing on main; surfaced in review of this change. Adds errorFromEventSenderDoesNotWedgeClose, which times out without the fix.
… failure (SDK-3375) Two review findings from Cursor Bugbot: - The transport-failure warning in DefaultEventSender logged e.toString(), which can include hostnames or URLs. It now logs LogValues.exceptionSummary(e), matching HttpErrors' own transport-failure helper and this module's other exception logging. - A custom EventSender that returns null reported no delivery, but the null was silently ignored, leaving the consecutive-success streak intact. It is now treated as an ordinary failure, the same as an exception from the sender, so it breaks the streak without opening a window. This matches the Go implementation, whose value-typed result has an ordinary-failure zero value and so never had this gap. Adds nullResultBreaksTheSuccessStreak.
… (SDK-3513) A 413 Payload Too Large is locally unrecoverable: DefaultEventSender now drops the payload at the sender call site without retrying it and returns an ordinary failure, so delivery is neither disabled nor put into a cooldown window and a subsequent smaller payload can still succeed (EVTPROC 1.7.3.1). This matches the Go server SDK, which drops immediately (the mobile SDKs retry once before dropping; the server SDKs follow the spec's no-retry reading). The shared HttpErrors classifier is deliberately untouched, since data sources share it. Adds a DefaultEventSenderTest case asserting a 413 is not retried, not unexpected, and not a shutdown.
There was a problem hiding this comment.
Risk: medium.
Automated risk review (dry run): Medium
This PR would not be automatically approved. Human approval required.
Why: This replaces permanent event-delivery shutdown with a concurrent cooldown state machine (and 413 payload drop) in DefaultEventProcessor, DeliveryCooldownTracker, DefaultEventSender, and EventSender, which is a non-trivial shared-SDK behavior change even with extensive tests.
If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.
Sent by Cursor Approval Agent: Dry-Run PR Approver
| } finally { | ||
| // The busy-worker count must always come back down, even if an Error (not an Exception) | ||
| // escaped the work above, or close() would wait forever in | ||
| // waitUntilAllFlushWorkersInactive. The Error still ends this worker thread. | ||
| synchronized (activeFlushWorkersCount) { | ||
| activeFlushWorkersCount.decrementAndGet(); | ||
| activeFlushWorkersCount.notifyAll(); | ||
| } |
There was a problem hiding this comment.
AI review: The finally keeps the count right, but the Error still ends this worker, and nothing replaces it. Once every worker is gone, close() can still hang: its flush increments the busy count in triggerFlush and hands the payload to payloadQueue, where no worker will take it, so waitUntilAllFlushWorkersInactive waits forever. Android runs a single worker, so one Error is enough if any events are pending at close(). errorFromEventSenderDoesNotWedgeClose passes only because its outbox is already empty when it calls close().
This test times out on 6a9a1b0, stuck in close():
@Test(timeout = 15000)
public void errorOnTheOnlyFlushWorkerDoesNotWedgeClose() throws Exception {
EventSender throwingSender = new EventSender() {
@Override
public Result sendAnalyticsEvents(byte[] data, int eventCount, URI eventsBaseUri) {
throw new AssertionError("boom");
}
@Override
public Result sendDiagnosticEvent(byte[] data, URI eventsBaseUri) {
return Result.success(null);
}
@Override
public void close() throws IOException {}
};
// An Error from the sender ends the only flush worker.
DefaultEventProcessor ep = makeEventProcessor(baseConfig(throwingSender).eventSendingThreadPoolSize(1));
ep.sendEvent(identifyEvent(user));
ep.flushAsync();
ep.waitUntilInactive();
// close() must still return when it flushes a pending event that no worker is left to deliver.
ep.sendEvent(identifyEvent(user));
ep.close();
}There was a problem hiding this comment.
AI review comment: Fixed in 5075d56. Both flush-worker work sites now catch Throwable rather than Exception, so an Error (for example an OutOfMemoryError while serializing) is handled as an ordinary failure and the worker survives instead of dying; the finally comment is corrected to match. I also strengthened errorFromEventSenderDoesNotWedgeClose to reproduce exactly this case: a single flush worker (as on Android), a real Error on the first delivery, then an assertion that a second delivery still lands (proving the worker lived) in addition to close() returning. I confirmed it fails when the fix is reverted and passes with it. Leaving this open for you to confirm the approach (catch Throwable to keep the worker alive, rather than replacing a dead worker).
Addresses review feedback on the cooldown PR: - The flush worker now catches Throwable (not only Exception) at both work sites, so an Error such as an OutOfMemoryError during serialization is handled as an ordinary failure instead of ending the worker thread. On Android, which runs a single worker, a lost worker would leave a later close() flush with nobody to drain it and wedge waitUntilAllFlushWorkersInactive forever. Adds a single-worker regression test (proven to fail without the fix). - Replaces spec section numbers in comments with behavioral prose, in the processor's diagnostic comment and two cooldown test comments. - Corrects the DeliveryCooldownTracker class doc: a stale failure still breaks the consecutive-success streak, while tokens confine only window and ladder movement to actual probes.



What
Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278) in the shared internal events code: on an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.
flushAsync()/flushBlocking(), or the shutdown flush) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be disposed of. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context deduplicator is flushed when the discarded batch contained an index or identify event, so a later payload can introduce those contexts again.DefaultEventSender(1.7.2) is unchanged.This is a port of the reviewed Go implementation (launchdarkly/go-sdk-events#68), adapted to this module's threading model: delivery outcomes arrive on flush worker threads and the shared executor rather than on the dispatcher thread, so the cooldown tracker is synchronized pure state, and outcomes are correlated to the window era they were dispatched in via an opaque
WindowToken(an outcome from a delivery in flight across a window boundary updates nothing).API (internal module surface)
EventSender.Resultgains theunexpectedclassification with static factoriessuccess(Date),failure(), andunexpectedFailure(), and the accessorisUnexpectedFailure(), which also accepts the legacy flag so results from senders written against the old API still get cooldown behavior.Resultconstructor andisMustShutDown()are deprecated.unexpectedFailure()keeps writing the legacy flag for consumers that have not migrated (the server SDK'sEventSenderWrapperand the Android SDK).DefaultEventSendermigrates off the deprecatedisHttpErrorRecoverable/checkIfErrorIsRecoverableAndLoghelpers toclassifyHttpFailure(first commit, behavior-identical) and then returns the new classification (second commit). Transport-levelIOExceptions keep their always-retry behavior.EventsConfigurationgains a package-privatecooldownNanoTimeFntest seam (deliberately not part of any constructor); the tracker documents that the source must haveSystem.nanoTime()semantics, and all window arithmetic uses overflow-safe difference comparisons.Design notes
Dateheader remains unconditional, exactly as on main.disabledflag remains as a shutdown-only latch; nothing sets it from delivery results anymore.finally, so even anError(not anException) escaping the loop body cannot leaveclose()waiting forever; theErrorstill ends that worker thread. (This last part fixes a pre-existing hazard on main, surfaced in review.)NanoTimeSourceinterface rather thanjava.util.function.LongSupplier, which does not exist on Android below API 24 (this module ships in the Android SDK at minSdk 21); the fix is verified with ad8 --min-api 21dex sweep showing zerojava.util.functionreferences.Testing
DeliveryCooldownTrackerTest(24 tests): window ladder with every rung asserted past the shift-guard bound, reset semantics, boundary-round staleness on both probe success and recoverable probe failure, stale-outcome and in-window-outcome guards, null token, default clock path, boundary instant, the atomic gate, and the once-per-window discard notice.DefaultEventProcessorCooldownTest(17 tests): cooldown entry and resume across a fake clock, legacy shutdown-flag results, contradictory success+shutdown results, in-window disposal with context-introduction recovery through both the index and identify arms, a window open forgetting even already-delivered introductions, empty in-window flushes doing nothing, shutdown during a window, warn-once-then-debug and resume logging, diagnostics being fully decoupled from the cooldown (a diagnostic failure opens no window; diagnostic outcomes do not affect an open window), a null sender result neither stopping delivery nor wedgingclose(), a sender exception breaking the success streak, simultaneous failures across the full worker pool stepping the ladder exactly once, and the batch-size stat zeroing on discards.DeliveryCooldownTrackerTest(24 tests): the window ladder and ceiling clamp, reset semantics, boundary-round staleness on probe success and recoverable probe failure, stale failures breaking the streak without moving a window, null-token inertness, the atomic gate, the once-per-window discard notice, overflow and wrap safety.diagnosticEventsStopAfter401Erroris rewritten asdiagnosticEventsContinueAfter401Error; the permanent-disable test is renamed to describe the cooldown contract; a transport-failure sender test pins the deliberate never-unexpected classification of connection errors.Out of scope (follow-ups)
classifyHttpFailurehas no locally-unrecoverable classification, so a 413 opens a window rather than dropping only the oversized payload. That is a pre-existing classifier gap with its own Jira bug (it also requires unsuppressing the corresponding contract test), deliberately not fixed here.EventSender.Result.STOPdoc reinterpretation, wrapper migration,event-delivery-cooldowncontract-test capability, and the 413 test suppression) ship in a follow-up PR once this module releases.DirectEventProcessorreads the dual-written legacy flag and keeps its current permanent-disable behavior byte-identical before and after this release; migrating it to the cooldown is its own coordinated change.Release notes
Note
Overview
Unexpected delivery failures (e.g. 401/403) now trigger a temporary cooldown instead of permanently stopping analytics sends. The event processor uses a new
DeliveryCooldownTrackerwith doubling windows (5→60 minutes), probe-based recovery after two consecutive successes, andWindowToken-correlated outcomes so concurrent workers only advance the ladder once.While a cooldown is active, analytics flushes skip the network but dispose batches like a refused send (context deduplicator flush when index/identify were involved, diagnostics
eventsInLastBatch= 0, warn-once-then-debug logging). Diagnostic deliveries stay on their own path and no longer gate cooldown state from analytics responses.EventSender.ResultgainsunexpectedFailure()/isUnexpectedFailure()and static factories; legacyisMustShutDown()still maps into cooldown behavior.DefaultEventSenderclassifies HTTP viaclassifyHttpFailure, treats 413 as a non-retryable ordinary failure, and hardens flush workers soThrowable/null results cannot wedgeclose().Reviewed by Cursor Bugbot for commit 5075d56. Bugbot is set up for automated code reviews on this repo. Configure here.