Skip to content

fix(providers): key and match StatefulSet replicas as one workload - #577

Open
Smana wants to merge 1 commit into
mainfrom
fix/statefulset-ordinal-identity
Open

Smana wants to merge 1 commit into
mainfrom
fix/statefulset-ordinal-identity

Conversation

@Smana

@Smana Smana commented Sep 21, 2026

Copy link
Copy Markdown
Owner

Fixes #513.

Why

Instant recall had never been accepted on a live deployment: 56 evaluations over 7 days, 0 accepted, 38 of them no_resource_match. The KB held observability/vmagent-vmagent-0 and -1 as two entries saying the same thing, and an incident on -2 matched neither.

NormalizeWorkloadName folds a Deployment pod's hash and a CronJob's run stamp, but a StatefulSet pod is <name>-<ordinal> and no rule matched it. The same function backs curator.IncidentKey / DupFingerprint and the recall gate, so the corpus fragmented per replica on write and a sibling could not match on read.

The judgement call the issue asked for

The fold does not go into NormalizeWorkloadName. That function takes a bare string, and on a kind-less name a short numeric tail is the name: aurora-serverless-postgres-old-1 and -2 are two databases, ip-10-20-0-144 is a node. TestNormalizeWorkloadName already pins that, deliberately. Only the Kind licenses the fold, so it lives at the seam that has one.

This lands on both sides, not recall-only: an alert on any replica of a StatefulSet now keys, dedupes and recalls as that StatefulSet, exactly as pods of a Deployment already do. A per-replica incident (one pod on a bad node) still investigates per pod; only the identity the knowledge is filed and looked up under is the workload. If you would rather keep replica identity in the dedup key, the write-side half is the normalizeResourceName change in internal/curator and can be reverted on its own.

What

  • providers.NormalizePodName folds ONE trailing ordinal after the existing rules, for a name known to be a Pod's. One, not a fixed point, so a StatefulSet named sts-0 keeps its own name for pods sts-0-N.
  • Workload.IdentityName / Workload.ResourceID apply it when Kind == "Pod". Ingestion still keeps the raw pod name: normalization stays a comparison-time concern, as workload_identity_test pins.
  • Workload.AgreesWithEntryName reads a stored entry both as written and folded once, so an entry filed under a sibling replica before this change still matches, and one filed under the StatefulSet after it does too.
  • The curator's normalizeResourceName takes the Workload now, so IncidentKey, DupFingerprint and the drafted entry's resource: read one rule; the recall gate reads it through refsAgree.

Kind-less requests are untouched: they keep their ordinal and match only themselves.

Migration

One-off, same shape as the CronJob fold: a StatefulSet pod's IncidentKey changes bytes, so its recurrence count restarts once on the first firing after upgrade. Existing per-replica KB entries keep matching (the gate reads them folded); the next curation on that workload files under the StatefulSet.

Verified

TDD: four new test files failed first (providers did not build; 7 behavioural cases failed for the expected reason), then passed. Full gate: go build, go vet, go test ./..., gofmt -l clean, hack/lint.sh reports 0 issues.

Found while working out why instant recall had never been accepted on a live
deployment: 56 recall evaluations over 7 days, 0 accepted, 38 of them
no_resource_match. The KB held observability/vmagent-vmagent-0 and -1 as two
entries saying the same thing, and an incident on -2 could match neither.

NormalizeWorkloadName folds a Deployment pod's hash and a CronJob's run stamp,
but a StatefulSet pod is <name>-<ordinal>, and no rule matched it. Since the
same function backs curator.IncidentKey/DupFingerprint and the recall gate,
the corpus fragmented per replica on write and a sibling could not match on
read.

The fold cannot go into NormalizeWorkloadName itself: that function takes a
bare string, and on a kind-less name a short numeric tail is the name —
aurora-serverless-postgres-old-1 and -2 are two databases, ip-10-20-0-144 is a
node — which TestNormalizeWorkloadName already pins. Only the Kind licenses
the fold, so it lives at the seam that has one:

- providers.NormalizePodName folds ONE trailing ordinal after the existing
  rules, for a name known to be a Pod's. One, not a fixed point, so a
  StatefulSet named sts-0 keeps its own name for pods sts-0-N.
- Workload.IdentityName / Workload.ResourceID apply it when Kind is "Pod".
  Ingestion still keeps the raw pod name — normalization stays a
  comparison-time concern, as workload_identity_test pins.
- Workload.AgreesWithEntryName reads a stored entry both as written and
  folded once, so an entry filed under a sibling replica before this change
  still matches, and one filed under the StatefulSet after it does too.
- The curator's normalizeResourceName now takes the Workload, so
  IncidentKey, DupFingerprint and the drafted entry's resource read the same
  rule; the recall gate reads it through refsAgree.

A kind-less request is untouched: it keeps its ordinal and matches only
itself. A pod-scoped alert on any replica of a StatefulSet now keys, dedupes
and recalls as that StatefulSet, exactly as pods of a Deployment already did.

Fixes #513
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StatefulSet ordinals are not normalized, so recall fragments per replica and cannot match a sibling

1 participant