feat(hybridcloud): Size webhook mailboxes from a rolling rate, not a volume gate

[vaind]

Sentry forwards an integration's webhooks through a "mailbox" -- a queue drained in order, one payload at a time -- so one noisy repository makes every other payload for that integration wait behind it. Splitting an integration across sub-mailboxes lets them drain in parallel; which one a payload lands in is key % count, where the key is the repository or issue it is about.

Two things decide that split today and both are wrong. Whether to split is an hourly Redis counter: an integration splits only after 3,000 payloads inside one clock hour. That window resets on the hour, so a second key with a one-day expiry has to carry the decision across the reset; a burst straddling the boundary trips neither hour; and a Redis error means no split at all. GitHub opted out of the whole mechanism years ago with an always_bucket flag. How far to split is hardcoded per provider -- 100 for GitHub and GitLab, 10 for Jira and Azure DevOps -- so an integration sending a handful of payloads an hour is split as widely as one sending constantly, and the number moves only when someone edits it.

Both are replaced by one rule: count what the mailbox actually received recently, and size the split to that.

How it works

Arrivals are counted over a rolling 15 minutes as five 3-minute Redis counters, summed on read. A counter takes writes only while it is the current one, so its expiry is a real bound rather than one pushed forward on every write -- and nothing needs remembering across a reset, which is why the second key disappears rather than moving.

The split is payloads // (worker_threads × payloads_per_thread), rounded down to a power of two and capped. worker_threads is how many payloads one drain delivers at once, so the unit is a pass of the delivery pool, and four passes is what one mailbox should be worth. Powers of two because key % 2n leaves a key where key % n put it or exactly n along, so doubling moves half the keys instead of nearly all; rounding down means the rate must double before a bucket is added, so an integration sitting on the boundary is not re-mapped every window. A window Redis cannot answer sizes to the cap, because an outage must not put a busy integration back onto one serial queue.

Which providers

Every change of divisor re-maps keys, so a payload can land in a different sub-mailbox from the one its predecessor is still queued in. Only a provider whose delivery tolerates reordering can absorb that, and hybridcloud.webhookpayload.skip_on_failure_providers already records exactly that set -- the same option the drain reads. Reading it here means a provider earns a rate-sized split only once it tolerates what the split costs, rather than sitting on a hardcoded list nobody dares delete. That option now covers every forwarding provider, so all of them are rate-sized; the fixed fallback is what applies where the option is narrower, which today means self-hosted and dev until #123422 realigns the checked-in default.

Worth checking

payloads_per_thread (4) and max_mailbox_buckets (64) are reasoned, not measured. They are options rather than constants so acting on a reading is an edit instead of a deploy, and the new buckets tag on the routing metric is what would let us tune them -- it needs a Datadog tag-config change first, which is not retroactive.

Mailbox names move both ways on deploy: quiet integrations lose their split, and Jira and Azure DevOps integrations busy enough to earn one get it for the first time. Payloads already queued under an old name drain normally.

The window counts arrivals, so it sizes for steady state and not for a backlog -- if delivery stalls, inflow is unchanged and the split stays put.

Refs CW-1987

[linear-code]

CW-1987

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0d208e8. Configure here.}