feat(hybridcloud): Bucket webhook mailboxes on the key, not on volume

[vaind]

An integration was only split into sub-mailboxes once it exceeded 3,000 payloads in an hour. That gate is a fixed-window Redis counter aligned to the clock hour, which makes it the wrong shape three ways: it resets on the hour, so the decision needs a second piece of state — a use_buckets key with a one-day TTL — to survive the reset; a burst straddling the boundary evades it, since 2,900 payloads at 11:59 and 2,900 at 12:01 never exceed the limit in either window; and it fails open on a Redis error.

GitHub already bypassed it with always_bucket, so unconditional bucketing is what the highest-volume forwarding provider has done all along. The key now decides on its own: a payload carrying one is bucketed, one without falls back to the integration-level mailbox. The gate and always_bucket both go.

Ordering

The divisor is the key-to-mailbox map, so a fixed divisor means a key never moves: same repository, same bucket, always. Per-key ordering — what the cell-side handlers require — is preserved. What is given up is ordering across keys within an integration, which no consumer needs, and which github and jira_server had already given up.

Parser	Key	Count
jira, jira_server, vsts	issue.id, resource.workItemId	10
github, gitlab	repository.id, project.id	100

CW-1987 replaces these counts with one sized from a rolling per-integration rate — but only for providers that deliver with skip_on_failure and so tolerate reordering. jira, jira_server and vsts keep a fixed count: they deliver strictly ordered, and a divisor that moves re-maps half the keys, leaving one issue's backlog in one mailbox while its new payloads go to another.

Uniform failure handling

The five hand-rolled key readers each disagreed on bad input: github rejected a numeric string, gitlab returned the value uncoerced so a string id would raise TypeError at the modulo, and jira_server caught ValueError but not TypeError. All five now read through BaseRequestParser.bucket_key_at, which falls back to the integration-level mailbox instead.

This is a consistency fix, not a fix for anything observed. GitHub and GitLab send repository.id and project.id as JSON numbers, so those divergent branches were latent. Jira Server is the one provider whose issue.id really does arrive as a string, and its reader already coerced it.

Reviewer notes

gitlab and jira_server now bucket every payload, not only those from integrations past the gate. Their fixtures carry project.id 15 and issue.id 101, so tests assert gitlab:<id>:15:push and jira_server:<id>:1. GitHub's routing is unchanged, since it already bucketed everything.

A payload carrying no key still lands on the unsplit mailbox, so it stays unordered against the keyed ones. That gap is not new, but it now applies to every integration rather than only those past the gate.

One GitHub fixture claimed "repository": {"id": "1"}. Real payloads send a number — the cell handler does str(event["repository"]["id"]) — so that fixture was wrong and is now 1, which is what every other GitHub test already used.

The low-volume, high-volume, and cache-primed test variants differed only in how they tripped the gate, so they collapse into one; the freed slots cover a GitLab mailbox_bucket_id unit test and a VSTS payload with no work item id.

314 tests pass across the base parser, every integration parser, and the delivery tasks.

Refs CW-1887

[linear-code]

CW-1887

[vaind]

Folded into #123449, which now removes the volume gate and sizes the split from a rolling rate in one change. Splitting them here was a commit boundary, not a behavioural one: landing this alone would have fanned gitlab, jira, jira_server and vsts out to a fixed 10 or 100 buckets each, and #123449 would then have collapsed the quiet ones straight back — two re-mappings of the same keys in opposite directions, for a state nobody wanted to run.

The uniform bucket-key reader is #123678, on its own, because it changes malformed-input handling rather than routing.