feat(integrations): record deployments from GitLab and Flux webhooks - #209
Open
TartanLeGrand wants to merge 22 commits into
Open
TartanLeGrand wants to merge 22 commits into
TartanLeGrand wants to merge 22 commits into
Conversation
Read INTEGRATION_GITLAB_SIGNING_TOKEN, INTEGRATION_GITLAB_SECRET_TOKEN, INTEGRATION_FLUX_HMAC_KEY, INTEGRATION_WEBHOOK_TOLERANCE and INTEGRATION_ENVIRONMENTS at startup. An invalid value stops the server; secrets never appear in errors or logs. Refs BananaOps#208
Standard Webhooks v1 signatures (HMAC-SHA256 over id.timestamp.body) with a freshness window, and the legacy X-Gitlab-Token compared in constant time. Refs BananaOps#208
Accept X-Signature with sha224, sha256, sha384 or sha512, compared with hmac.Equal over the raw body. Refs BananaOps#208
…malization Observation is the normalized form of a GitLab or Flux notification; Rank orders statuses for equal timestamps; NormalizeRepoURL makes SSH and HTTP repository URLs comparable. Refs BananaOps#208
Deployment Hook payloads become observations keyed by instance host and deployment id, with the environment mapping, the status table, the title and the message of the Tracker event. Refs BananaOps#208
Kustomization and HelmRelease events become observations keyed by environment, object and revision, with the severity and reason table, the freshness check on the signed timestamp and the short revision. Refs BananaOps#208
Correlates webhook notifications with Tracker events. Claim relies on the unique _id, Advance is a conditional FindOneAndUpdate on lastEventAt and rank, Revert restores the previous state. Indexes on eventId and a 90 day TTL on updatedAt. Refs BananaOps#208
CreateEvent, UpdateEvent, CreateLock and UpdateLock keep authz.Authorize as their first statement and delegate to unexported cores (createEvent, updateEvent, createLock, updateLock). createEvent gains an observe mode that records a lock conflict instead of failing, and updateEvent measures the duration at an explicit instant. Characterization tests pin the RPC behaviour. Behaviour change: an event that takes a lock now counts one authorization decision in tracker_auth_requests_total instead of three, because the nested lock calls go through the cores. Refs BananaOps#208
IntegrationProcessor claims the correlation key, creates the Tracker event once and updates it afterwards through the unauthorized cores, in observe mode for locks: a conflict is recorded as a changelog comment and never fails the request. Terminal statuses release only the lock of their own event. Service names come from a 60 second snapshot of the catalog. Refs BananaOps#208
Revert's filter now also matches the status and rank of the Advance being undone, in addition to lastEventAt, so it can never clobber a newer state that shares the same instant but carries a higher rank. Refs BananaOps#208
After every successful event write for a key, the processor re-reads the claim and, if a concurrent observation has moved it further, re-applies the claim's own status and end date through the same unauthorized update core, up to 3 times, without any per-key mutex or Mongo lease; the reported outcome always stays the request's own. The lock taken on a start transition is now created with its event id in the same write instead of a separate attach step, is released immediately if the claim has already gone terminal by the time it is taken, and is released by its id if the post-take re-read itself fails. SetEventID now retries up to 3 times (50/100/200ms) after createEvent succeeds; if it still fails, the created event and any lock attached to it are removed before the claim is dropped, instead of leaving them orphaned. The service resolver reloads the catalog against a context detached from the caller, single-flighted so concurrent callers never wait on the reload or block on the exclusive lock, and retries a failed reload after 5s instead of the full 60s ttl. Refs BananaOps#208
Rank now gives waiting_approval 0 instead of 1, tied with start: a later start always takes over an approval, matching the processor's create/update flow. Refs BananaOps#208
The observe-mode create path now creates the event first, then takes the lock with the event id already set in the same write, reusing the round 1 observeLock path instead of a separate attach step; a conflict is recorded as a changelog comment via a follow-up write, and any other failure taking the lock is only logged. After taking the lock, the claim is re-read and the lock released immediately if it has already gone terminal, same as the update path. In update, takeLock is now derived from the claim state read before Advance (prev), never from the event's current status, since converge can rewrite it concurrently. Tests: every lock assertion in the concurrent and out-of-order processor tests now checks no lock exists at all for the service/environment, not only by event id; added an update-path race test (blocked, then running/success concurrently, 20 iterations) and a create-path lock-conflict test. Refs BananaOps#208
Adds two processor tests: waiting_approval and start sharing the same status_changed_at second still lets start take the lock (higher rank wins the tie), and start followed by a later waiting_approval is rejected as stale, leaving the event and its lock untouched. Comments only otherwise: the Rank doc comment now states that a later waiting_approval never moves an event back from start, and the observeLock comment no longer describes the round 1 separate-attach behaviour removed in round 2. Refs BananaOps#208
POST /api/v1alpha1/integrations/gitlab/webhook and /api/v1alpha1/integrations/flux/webhook are registered only when their source is configured. The signature is the authentication (1 MiB body limit, check before any JSON decoding); responses are 200 recorded, 202 ignored, 400, 401, 413 or 500. Adds tracker_integration_webhooks_total{source,result}.
Refs BananaOps#208
New INTEGRATIONS.md (GitLab webhook, Flux Provider and Alert, locks, troubleshooting), INTEGRATION_* variables in CONFIGURATION.md, gitlab and flux sources in EVENTS.md, link from the README. Refs BananaOps#208
… payload X-Gitlab-Instance is not part of the Standard Webhooks signed content (webhook-id.webhook-timestamp.body). Keying the correlation on it let a captured, still-valid signed delivery be replayed within the tolerance window with a different instance header to mint a new key, defeating the replay window's idempotency guarantee. ParseGitLab now derives the key host from project.web_url only, the field the signature actually covers, and no longer takes an instance header parameter. A web_url with no parsable host is rejected the same way as any other missing required field. Refs BananaOps#208
The event correlated with a claim can be deleted (DeleteEvents RPC or UI) while notifications for its deployment keep arriving. Because that is a business condition rather than a storage failure, the next update turned it into a permanent 500: the claim kept pointing at the deleted event id for the rest of its 90-day TTL, so every following notification (and every manual resend) failed the same way, and GitLab disables a webhook after repeated failures. The processor now recognizes mongo.ErrNoDocuments from the event store read, reverts the claim to its state before this Advance exactly as a genuine failure would, and reports a new ignored outcome (reason "event deleted") instead of an error, so the handler answers 202. Also fixes a related overwrite: SetEventID only takes effect while the claim's eventId is still empty. Without that guard, a claim abandoned and recreated after 30 seconds could have its second, legitimate SetEventID clobbered by a slow first write, orphaning the second event. The zero-matched case is treated exactly like the existing SetEventID failure: compensate and drop the claim. Refs BananaOps#208
…elivery ids Every 500 response from the webhook handler logged only the key, never the underlying error: Claim, Advance, createEvent and SetEventID failures (and abandoned claims) left an operator with nothing to act on, unlike the 401/400 paths, which already carry a reason. Both Process error branches now log "reason", err.Error(); none of these errors (Mongo, processor) ever carries a secret, header or body. Delivery identifiers logged before the request is authenticated (Idempotency-Key, X-Gitlab-Event-UUID, webhook-id) are now truncated to 64 bytes so an anonymous caller cannot inflate log volume by sending an oversized header, and X-Gitlab-Instance joins them: it no longer feeds the correlation key but is still useful for debugging. Refs BananaOps#208
Characterization tests for the event.go RPC paths the refactor moved but left unmodified, none of it changed by this commit: CreateEvent's error text for an unknown related_id and the positive duration it sets for a known one, UpdateEvent's error text for an unknown slack id, no lock release through the RPC on a non-terminal status, and the updateEvent changelog rules (ticket linked, priority updated, title updated, an owner-only change recorded as an approval, and the generic update entry for anything else) with their exact field/old/ new values. Refs BananaOps#208
NormalizeRepoURL now drops an explicit default port (:443 on https, :80 on http) and strips a .git suffix case-insensitively, so a catalog repository URL written with either form still matches the normalized URLs built from a webhook payload. parseGitLabTime now returns (time.Time, bool) instead of building an error the caller only used to decide whether parsing failed and then discarded; behaviour is unchanged. Refs BananaOps#208
The "sender should retry" note was only true for Flux. Checked against GitLab's current docs: a failed delivery is not retried automatically; GitLab only disables the webhook after repeated consecutive failures (temporarily, then permanently past 40), and a lost terminal notification has to be resent by hand from the webhook's Recent events (or the equivalent resend API). Documented both paths under a new Retries section, linked from the 500 row. Also documents the event deleted 202 reason, the missing URL: <environment_external_url> line in the GitLab message layout, and how to find and release a lock orphaned by a crash between event creation and correlation. Source: https://docs.gitlab.com/user/project/integrations/webhooks/ Refs BananaOps#208
TartanLeGrand
force-pushed
the
feat/deploy-integrations
branch
from
October 2, 2026 14:01
ebffca2 to
79ef516
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Record deployments in Tracker from what the deployment tooling already emits, without any extra CI job:
POST /api/v1alpha1/integrations/gitlab/webhook: GitLab "Deployment events" (signing token with Standard Webhooks signatures, or the legacy secret token).POST /api/v1alpha1/integrations/flux/webhook: Flux notification-controllergeneric-hmacprovider (Kustomization and HelmRelease).deploymentevent per deployment, created on start and updated afterwards; retries, replays and out of order deliveries have no effect (newintegration_deploymentscollection, 90 day TTL).tracker_auth_requests_totalinstead of three. A negative event duration from clock skew is now clamped to 0. The "lock not found for event to update" warning log no longer exists on the RPC path.tracker_integration_webhooks_total{source,result}.Configuration
Routes exist only when their source is configured:
INTEGRATION_GITLAB_SIGNING_TOKEN(orINTEGRATION_GITLAB_SECRET_TOKEN),INTEGRATION_FLUX_HMAC_KEY,INTEGRATION_WEBHOOK_TOLERANCE(default5m),INTEGRATION_ENVIRONMENTS(defaultproduction=production,staging=preproduction). Seedocs/INTEGRATIONS.mdanddocs/CONFIGURATION.md.Checks
go test ./...with MongoDB 7: green, 392 tests passed across 15 packages.go test -race ./internal/... ./server/...: green, no data race.go test ./internal/integrations/ -cover: green, 98.8% statement coverage.golangci-lint run: 2 issues (baseline 2, pre-existing in untrackedpkg/flatted/flatted.go, unrelated to this change).gosec -exclude=G103,G115 -exclude-generated ./...: 0 issues (baseline 0).govulncheck ./...: no vulnerability in called code; one pre-existing module-level advisory (golang.org/x/cryptoopenpgp, GO-2026-5932) that this branch does not call and does not introduce (go.mod/go.sumuntouched).tracker serv: GitLab running then success gives one event and a lock taken then released, replay answers 202 duplicate, an invalid signature answers 401 with nothing written, areview/*environment answers 202 ignored, Flux Progressing then ReconciliationSucceeded in the same second gives one successful event, a bad Flux signature answers 401, both routes answer 404 without configuration, no secret in the logs,tracker_integration_webhooks_totalexposed on:8081/metricswithsource/resultlabels.Notes for reviewers
CreateEventthat takes a lock now counts one authorization decision instead of three intracker_auth_requests_total, and a negative event duration caused by clock skew is now clamped to 0.log.Fatallnin the event and lock storeCreateis untouched (out of scope).project.web_urlonly (the signed payload), notX-Gitlab-Instance: that header is not part of the Standard Webhooks signed content, so keying on it let a captured, still-valid signed delivery be replayed within the tolerance window with a different instance header to mint a new key and bypass the replay protection.X-Gitlab-Instanceis still logged (truncated), just no longer trusted for correlation.webhook-timestamp. GitLab documents only that theIdempotency-Keyis kept. If the original timestamp is reused, a resend older thanINTEGRATION_WEBHOOK_TOLERANCEis refused with 401 in signing-token mode, and the recovery note indocs/INTEGRATIONS.mdneeds adjusting.Revert); covering it in the processor needs an injectable event store. Follow-up.Out of scope
deployment_status.EventStoreClient.CreateandLockStoreClient.Createstill stop the process on an insert error (existing behaviour).createLockchecks then inserts without a unique index (pre-existing); two concurrent deployments of the same service can both hold a lock; follow-up issue.Closes #208