fix: patch each Teams card once per event, not once per hook - #48
Merged
Merged
Conversation
GitLab fans every merge request event out through the three group hooks and, on some projects, a project hook too. Each delivery patched every card of the merge request, and the per-message fingerprint check could not stop it: the deliveries arrive within a second and all read the previous fingerprint before any of them writes. The same Teams activity was updated 3-4 times concurrently, and Teams answers exactly one of those with a fixed ~10s delay, which is where most of the 10s client timeouts came from. Handlers of one merge request now run under a per-MR advisory lock. The lock is transaction-scoped because the pool never resets sessions, and it is polled with try-lock so a waiter never pins a pool connection. Holders are capped at pool // 2 - 1 through a semaphore on the pool handler: a handler holds its lock connection for its whole body and needs one more for the body itself, and without the cap handlers of distinct merge requests deadlock the pool once there are as many as it has connections. Pool acquisition gets a timeout so any residual starvation fails instead of hanging. A lock not acquired within 30s runs unlocked rather than dropping the event.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitLab fans every merge request event out through the three
skeepersgroup hooks and, on some projects, a project hook too. Each delivery patched every card of the merge request, and the per-message fingerprint check could not stop it: the deliveries arrive within a second and all read the previous fingerprint before any of them writes. The same Teams activity was updated 3–4 times concurrently, and Teams answers exactly one of those with a fixed ~10.6 s delay — measured on production this morning: in each burst of 4 concurrent PUTs on one activity, three took 0.5–2 s and one took 11 s. That single slow PUT is what pushed activity-api past gitlab-mr-api's 10 s client timeout on 17.5 % of PATCHes over the last week, and each of those became a failed webhook delivery from GitLab's point of view.Change
mr_lockinwebhook/merge_request.py: handlers of one merge request run under a per-MR advisory lock, so the 2nd–4th deliveries wait for the first and then skip on the fingerprint it stored. Transaction-scoped because the pool never resets sessions (NoResetConnection); polled withpg_try_advisory_xact_lockso a waiter never pins a pool connection.handler_slotsonDatabaseLifecycleHandler: holders capped atpool // 2 - 1. A handler holds its lock connection for its whole body and needs one more for the body itself; without the cap, handlers of distinct merge requests deadlock the pool as soon as there are as many as it has connections, and/healthzshares that pool.DATABASE_ACQUIRE_TIMEOUT_SECONDS(default 30): pool acquisition times out instead of hanging.Guards
updatearrives 4 times concurrently → exactly 1 PATCH. Red without the lock:got 4.False, False, True), degrades after the deadline, and holders are capped (peak 1 with 1 slot, red without the semaphore:3 == 1).Only the non-closing
merge_requestpath needed this:pipelineandperiodic_cleanupalready serialise throughupdate_all_messages_transactional'sFOR UPDATE, andnote/emojigo throughpending_mr_refreshwithSKIP LOCKED.Success metric, unchanged from the last two changes: share of
PATCH /api/v1/message≥ 5 s in activity-api's access logs, 17.5 % over the last 7 days.