Skip to content

feat(integrations): record deployments from GitLab and Flux webhooks - #209

Open
TartanLeGrand wants to merge 22 commits into
BananaOps:mainfrom
TartanLeGrand:feat/deploy-integrations
Open

TartanLeGrand wants to merge 22 commits into
BananaOps:mainfrom
TartanLeGrand:feat/deploy-integrations

Conversation

@TartanLeGrand

Copy link
Copy Markdown
Contributor

What

Record deployments in Tracker from what the deployment tooling already emits, without any extra CI job:

  • POST /api/v1alpha1/integrations/gitlab/webhook: GitLab "Deployment events" (signing token with Standard Webhooks signatures, or the legacy secret token).
  • POST /api/v1alpha1/integrations/flux/webhook: Flux notification-controller generic-hmac provider (Kustomization and HelmRelease).
  • One Tracker deployment event per deployment, created on start and updated afterwards; retries, replays and out of order deliveries have no effect (new integration_deployments collection, 90 day TTL).
  • The deployment lock is taken like a manual deployment; a service locked by someone else never makes the webhook fail, the event carries a changelog comment instead.
  • Event and lock RPCs now delegate to unauthorized cores. Behaviour change: an event that takes a lock counts one decision in tracker_auth_requests_total instead of three. A negative event duration from clock skew is now clamped to 0. The "lock not found for event to update" warning log no longer exists on the RPC path.
  • New metric tracker_integration_webhooks_total{source,result}.

Configuration

Routes exist only when their source is configured: INTEGRATION_GITLAB_SIGNING_TOKEN (or INTEGRATION_GITLAB_SECRET_TOKEN), INTEGRATION_FLUX_HMAC_KEY, INTEGRATION_WEBHOOK_TOLERANCE (default 5m), INTEGRATION_ENVIRONMENTS (default production=production,staging=preproduction). See docs/INTEGRATIONS.md and docs/CONFIGURATION.md.

Checks

  • go test ./... with MongoDB 7: green, 392 tests passed across 15 packages.
  • go test -race ./internal/... ./server/...: green, no data race.
  • go test ./internal/integrations/ -cover: green, 98.8% statement coverage.
  • golangci-lint run: 2 issues (baseline 2, pre-existing in untracked pkg/flatted/flatted.go, unrelated to this change).
  • gosec -exclude=G103,G115 -exclude-generated ./...: 0 issues (baseline 0).
  • govulncheck ./...: no vulnerability in called code; one pre-existing module-level advisory (golang.org/x/crypto openpgp, GO-2026-5932) that this branch does not call and does not introduce (go.mod/go.sum untouched).
  • End to end against tracker serv: GitLab running then success gives one event and a lock taken then released, replay answers 202 duplicate, an invalid signature answers 401 with nothing written, a review/* environment answers 202 ignored, Flux Progressing then ReconciliationSucceeded in the same second gives one successful event, a bad Flux signature answers 401, both routes answer 404 without configuration, no secret in the logs, tracker_integration_webhooks_total exposed on :8081/metrics with source/result labels.

Notes for reviewers

  • The event/lock core refactor keeps RPC behaviour, with two observable changes: a CreateEvent that takes a lock now counts one authorization decision instead of three in tracker_auth_requests_total, and a negative event duration caused by clock skew is now clamped to 0.
  • The "lock not found for event to update" warning log no longer exists on the RPC path.
  • The pre-existing log.Fatalln in the event and lock store Create is untouched (out of scope).
  • The GitLab correlation key's host now comes from project.web_url only (the signed payload), not X-Gitlab-Instance: that header is not part of the Standard Webhooks signed content, so keying on it let a captured, still-valid signed delivery be replayed within the tolerance window with a different instance header to mint a new key and bypass the replay protection. X-Gitlab-Instance is still logged (truncated), just no longer trusted for correlation.
  • To confirm on a GitLab instance: whether "Resend request" re-signs with a fresh webhook-timestamp. GitLab documents only that the Idempotency-Key is kept. If the original timestamp is reused, a resend older than INTEGRATION_WEBHOOK_TOLERANCE is refused with 401 in signing-token mode, and the recovery note in docs/INTEGRATIONS.md needs adjusting.
  • Test gap: the processor path "event update fails for a storage reason other than not found, then Revert, then 500" is covered only at the store level (Revert); covering it in the processor needs an injectable event store. Follow-up.

Out of scope

  • Pipeline and job events (drift), merge request enrichment, GitLab API polling, GitHub deployment_status.
  • Helm chart wiring of the new secrets (tracked separately).
  • UI changes.
  • EventStoreClient.Create and LockStoreClient.Create still stop the process on an insert error (existing behaviour).
  • createLock checks then inserts without a unique index (pre-existing); two concurrent deployments of the same service can both hold a lock; follow-up issue.

Closes #208

Read INTEGRATION_GITLAB_SIGNING_TOKEN, INTEGRATION_GITLAB_SECRET_TOKEN, INTEGRATION_FLUX_HMAC_KEY, INTEGRATION_WEBHOOK_TOLERANCE and INTEGRATION_ENVIRONMENTS at startup. An invalid value stops the server; secrets never appear in errors or logs.

Refs BananaOps#208
Standard Webhooks v1 signatures (HMAC-SHA256 over id.timestamp.body) with a freshness window, and the legacy X-Gitlab-Token compared in constant time.

Refs BananaOps#208
Accept X-Signature with sha224, sha256, sha384 or sha512, compared with hmac.Equal over the raw body.

Refs BananaOps#208
…malization

Observation is the normalized form of a GitLab or Flux notification; Rank orders statuses for equal timestamps; NormalizeRepoURL makes SSH and HTTP repository URLs comparable.

Refs BananaOps#208
Deployment Hook payloads become observations keyed by instance host and deployment id, with the environment mapping, the status table, the title and the message of the Tracker event.

Refs BananaOps#208
Kustomization and HelmRelease events become observations keyed by environment, object and revision, with the severity and reason table, the freshness check on the signed timestamp and the short revision.

Refs BananaOps#208
Correlates webhook notifications with Tracker events. Claim relies on the unique _id, Advance is a conditional FindOneAndUpdate on lastEventAt and rank, Revert restores the previous state. Indexes on eventId and a 90 day TTL on updatedAt.

Refs BananaOps#208
CreateEvent, UpdateEvent, CreateLock and UpdateLock keep authz.Authorize as their first statement and delegate to unexported cores (createEvent, updateEvent, createLock, updateLock). createEvent gains an observe mode that records a lock conflict instead of failing, and updateEvent measures the duration at an explicit instant. Characterization tests pin the RPC behaviour.

Behaviour change: an event that takes a lock now counts one authorization decision in tracker_auth_requests_total instead of three, because the nested lock calls go through the cores.

Refs BananaOps#208
IntegrationProcessor claims the correlation key, creates the Tracker event once and updates it afterwards through the unauthorized cores, in observe mode for locks: a conflict is recorded as a changelog comment and never fails the request. Terminal statuses release only the lock of their own event. Service names come from a 60 second snapshot of the catalog.

Refs BananaOps#208
Revert's filter now also matches the status and rank of the Advance being undone, in addition to lastEventAt, so it can never clobber a newer state that shares the same instant but carries a higher rank.

Refs BananaOps#208
After every successful event write for a key, the processor re-reads the claim and, if a concurrent observation has moved it further, re-applies the claim's own status and end date through the same unauthorized update core, up to 3 times, without any per-key mutex or Mongo lease; the reported outcome always stays the request's own.

The lock taken on a start transition is now created with its event id in the same write instead of a separate attach step, is released immediately if the claim has already gone terminal by the time it is taken, and is released by its id if the post-take re-read itself fails.

SetEventID now retries up to 3 times (50/100/200ms) after createEvent succeeds; if it still fails, the created event and any lock attached to it are removed before the claim is dropped, instead of leaving them orphaned.

The service resolver reloads the catalog against a context detached from the caller, single-flighted so concurrent callers never wait on the reload or block on the exclusive lock, and retries a failed reload after 5s instead of the full 60s ttl.

Refs BananaOps#208
Rank now gives waiting_approval 0 instead of 1, tied with start: a later start always takes over an approval, matching the processor's create/update flow.

Refs BananaOps#208
The observe-mode create path now creates the event first, then takes the lock with the event id already set in the same write, reusing the round 1 observeLock path instead of a separate attach step; a conflict is recorded as a changelog comment via a follow-up write, and any other failure taking the lock is only logged. After taking the lock, the claim is re-read and the lock released immediately if it has already gone terminal, same as the update path.

In update, takeLock is now derived from the claim state read before Advance (prev), never from the event's current status, since converge can rewrite it concurrently.

Tests: every lock assertion in the concurrent and out-of-order processor tests now checks no lock exists at all for the service/environment, not only by event id; added an update-path race test (blocked, then running/success concurrently, 20 iterations) and a create-path lock-conflict test.

Refs BananaOps#208
Adds two processor tests: waiting_approval and start sharing the same status_changed_at second still lets start take the lock (higher rank wins the tie), and start followed by a later waiting_approval is rejected as stale, leaving the event and its lock untouched.

Comments only otherwise: the Rank doc comment now states that a later waiting_approval never moves an event back from start, and the observeLock comment no longer describes the round 1 separate-attach behaviour removed in round 2.

Refs BananaOps#208
POST /api/v1alpha1/integrations/gitlab/webhook and /api/v1alpha1/integrations/flux/webhook are registered only when their source is configured. The signature is the authentication (1 MiB body limit, check before any JSON decoding); responses are 200 recorded, 202 ignored, 400, 401, 413 or 500. Adds tracker_integration_webhooks_total{source,result}.

Refs BananaOps#208
New INTEGRATIONS.md (GitLab webhook, Flux Provider and Alert, locks, troubleshooting), INTEGRATION_* variables in CONFIGURATION.md, gitlab and flux sources in EVENTS.md, link from the README.

Refs BananaOps#208
… payload

X-Gitlab-Instance is not part of the Standard Webhooks signed content
(webhook-id.webhook-timestamp.body). Keying the correlation on it let a
captured, still-valid signed delivery be replayed within the tolerance
window with a different instance header to mint a new key, defeating
the replay window's idempotency guarantee.

ParseGitLab now derives the key host from project.web_url only, the
field the signature actually covers, and no longer takes an instance
header parameter. A web_url with no parsable host is rejected the same
way as any other missing required field.

Refs BananaOps#208
The event correlated with a claim can be deleted (DeleteEvents RPC or
UI) while notifications for its deployment keep arriving. Because that
is a business condition rather than a storage failure, the next update
turned it into a permanent 500: the claim kept pointing at the deleted
event id for the rest of its 90-day TTL, so every following
notification (and every manual resend) failed the same way, and
GitLab disables a webhook after repeated failures.

The processor now recognizes mongo.ErrNoDocuments from the event store
read, reverts the claim to its state before this Advance exactly as a
genuine failure would, and reports a new ignored outcome (reason
"event deleted") instead of an error, so the handler answers 202.

Also fixes a related overwrite: SetEventID only takes effect while the
claim's eventId is still empty. Without that guard, a claim abandoned
and recreated after 30 seconds could have its second, legitimate
SetEventID clobbered by a slow first write, orphaning the second
event. The zero-matched case is treated exactly like the existing
SetEventID failure: compensate and drop the claim.

Refs BananaOps#208
…elivery ids

Every 500 response from the webhook handler logged only the key, never
the underlying error: Claim, Advance, createEvent and SetEventID
failures (and abandoned claims) left an operator with nothing to act
on, unlike the 401/400 paths, which already carry a reason. Both
Process error branches now log "reason", err.Error(); none of these
errors (Mongo, processor) ever carries a secret, header or body.

Delivery identifiers logged before the request is authenticated
(Idempotency-Key, X-Gitlab-Event-UUID, webhook-id) are now truncated
to 64 bytes so an anonymous caller cannot inflate log volume by
sending an oversized header, and X-Gitlab-Instance joins them: it no
longer feeds the correlation key but is still useful for debugging.

Refs BananaOps#208
Characterization tests for the event.go RPC paths the refactor moved
but left unmodified, none of it changed by this commit: CreateEvent's
error text for an unknown related_id and the positive duration it
sets for a known one, UpdateEvent's error text for an unknown slack
id, no lock release through the RPC on a non-terminal status, and the
updateEvent changelog rules (ticket linked, priority updated, title
updated, an owner-only change recorded as an approval, and the
generic update entry for anything else) with their exact field/old/
new values.

Refs BananaOps#208
NormalizeRepoURL now drops an explicit default port (:443 on https,
:80 on http) and strips a .git suffix case-insensitively, so a
catalog repository URL written with either form still matches the
normalized URLs built from a webhook payload.

parseGitLabTime now returns (time.Time, bool) instead of building an
error the caller only used to decide whether parsing failed and then
discarded; behaviour is unchanged.

Refs BananaOps#208
The "sender should retry" note was only true for Flux. Checked
against GitLab's current docs: a failed delivery is not retried
automatically; GitLab only disables the webhook after repeated
consecutive failures (temporarily, then permanently past 40), and a
lost terminal notification has to be resent by hand from the
webhook's Recent events (or the equivalent resend API). Documented
both paths under a new Retries section, linked from the 500 row.

Also documents the event deleted 202 reason, the missing
URL: <environment_external_url> line in the GitLab message layout,
and how to find and release a lock orphaned by a crash between event
creation and correlation.

Source: https://docs.gitlab.com/user/project/integrations/webhooks/

Refs BananaOps#208

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record deployments from GitLab deployment events and Flux notifications

1 participant