Skip to content

fix: patch each Teams card once per event, not once per hook - #48

Merged
babs merged 1 commit into
masterfrom
fix/serialise-mr-deliveries
Sep 15, 2026
Merged

babs merged 1 commit into
masterfrom
fix/serialise-mr-deliveries

Conversation

@babs

@babs babs commented Sep 15, 2026

Copy link
Copy Markdown
Member

GitLab fans every merge request event out through the three skeepers group hooks and, on some projects, a project hook too. Each delivery patched every card of the merge request, and the per-message fingerprint check could not stop it: the deliveries arrive within a second and all read the previous fingerprint before any of them writes. The same Teams activity was updated 3–4 times concurrently, and Teams answers exactly one of those with a fixed ~10.6 s delay — measured on production this morning: in each burst of 4 concurrent PUTs on one activity, three took 0.5–2 s and one took 11 s. That single slow PUT is what pushed activity-api past gitlab-mr-api's 10 s client timeout on 17.5 % of PATCHes over the last week, and each of those became a failed webhook delivery from GitLab's point of view.

Change

  • mr_lock in webhook/merge_request.py: handlers of one merge request run under a per-MR advisory lock, so the 2nd–4th deliveries wait for the first and then skip on the fingerprint it stored. Transaction-scoped because the pool never resets sessions (NoResetConnection); polled with pg_try_advisory_xact_lock so a waiter never pins a pool connection.
  • handler_slots on DatabaseLifecycleHandler: holders capped at pool // 2 - 1. A handler holds its lock connection for its whole body and needs one more for the body itself; without the cap, handlers of distinct merge requests deadlock the pool as soon as there are as many as it has connections, and /healthz shares that pool.
  • DATABASE_ACQUIRE_TIMEOUT_SECONDS (default 30): pool acquisition times out instead of hanging.
  • A lock not acquired within 30 s logs a warning and runs unlocked — the previous behaviour — rather than dropping the event.
  • The mocked activity-api client in tests now yields while "in flight"; without that no handler interleaving happens and the race cannot show up in a test.

Guards

  • e2e: a card exists, the same update arrives 4 times concurrently → exactly 1 PATCH. Red without the lock: got 4.
  • e2e: pool of 2, two distinct MRs together → both complete. Red without the holder cap: both hang.
  • unit: the lock polls and releases the connection between tries (3 acquires for False, False, True), degrades after the deadline, and holders are capped (peak 1 with 1 slot, red without the semaphore: 3 == 1).

Only the non-closing merge_request path needed this: pipeline and periodic_cleanup already serialise through update_all_messages_transactional's FOR UPDATE, and note/emoji go through pending_mr_refresh with SKIP LOCKED.

Success metric, unchanged from the last two changes: share of PATCH /api/v1/message ≥ 5 s in activity-api's access logs, 17.5 % over the last 7 days.

GitLab fans every merge request event out through the three group hooks
and, on some projects, a project hook too. Each delivery patched every
card of the merge request, and the per-message fingerprint check could
not stop it: the deliveries arrive within a second and all read the
previous fingerprint before any of them writes. The same Teams activity
was updated 3-4 times concurrently, and Teams answers exactly one of
those with a fixed ~10s delay, which is where most of the 10s client
timeouts came from.

Handlers of one merge request now run under a per-MR advisory lock.
The lock is transaction-scoped because the pool never resets sessions,
and it is polled with try-lock so a waiter never pins a pool connection.
Holders are capped at pool // 2 - 1 through a semaphore on the pool
handler: a handler holds its lock connection for its whole body and
needs one more for the body itself, and without the cap handlers of
distinct merge requests deadlock the pool once there are as many as it
has connections. Pool acquisition gets a timeout so any residual
starvation fails instead of hanging. A lock not acquired within 30s
runs unlocked rather than dropping the event.
@babs
babs merged commit c98f097 into master Sep 15, 2026
1 check passed
@babs
babs deleted the fix/serialise-mr-deliveries branch September 15, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant