Conversation
Found while working out why instant recall had never been accepted on a live deployment: 56 recall evaluations over 7 days, 0 accepted, 38 of them no_resource_match. The KB held observability/vmagent-vmagent-0 and -1 as two entries saying the same thing, and an incident on -2 could match neither. NormalizeWorkloadName folds a Deployment pod's hash and a CronJob's run stamp, but a StatefulSet pod is <name>-<ordinal>, and no rule matched it. Since the same function backs curator.IncidentKey/DupFingerprint and the recall gate, the corpus fragmented per replica on write and a sibling could not match on read. The fold cannot go into NormalizeWorkloadName itself: that function takes a bare string, and on a kind-less name a short numeric tail is the name — aurora-serverless-postgres-old-1 and -2 are two databases, ip-10-20-0-144 is a node — which TestNormalizeWorkloadName already pins. Only the Kind licenses the fold, so it lives at the seam that has one: - providers.NormalizePodName folds ONE trailing ordinal after the existing rules, for a name known to be a Pod's. One, not a fixed point, so a StatefulSet named sts-0 keeps its own name for pods sts-0-N. - Workload.IdentityName / Workload.ResourceID apply it when Kind is "Pod". Ingestion still keeps the raw pod name — normalization stays a comparison-time concern, as workload_identity_test pins. - Workload.AgreesWithEntryName reads a stored entry both as written and folded once, so an entry filed under a sibling replica before this change still matches, and one filed under the StatefulSet after it does too. - The curator's normalizeResourceName now takes the Workload, so IncidentKey, DupFingerprint and the drafted entry's resource read the same rule; the recall gate reads it through refsAgree. A kind-less request is untouched: it keeps its ordinal and matches only itself. A pod-scoped alert on any replica of a StatefulSet now keys, dedupes and recalls as that StatefulSet, exactly as pods of a Deployment already did. Fixes #513
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #513.
Why
Instant recall had never been accepted on a live deployment: 56 evaluations over 7 days, 0 accepted, 38 of them
no_resource_match. The KB heldobservability/vmagent-vmagent-0and-1as two entries saying the same thing, and an incident on-2matched neither.NormalizeWorkloadNamefolds a Deployment pod's hash and a CronJob's run stamp, but a StatefulSet pod is<name>-<ordinal>and no rule matched it. The same function backscurator.IncidentKey/DupFingerprintand the recall gate, so the corpus fragmented per replica on write and a sibling could not match on read.The judgement call the issue asked for
The fold does not go into
NormalizeWorkloadName. That function takes a bare string, and on a kind-less name a short numeric tail is the name:aurora-serverless-postgres-old-1and-2are two databases,ip-10-20-0-144is a node.TestNormalizeWorkloadNamealready pins that, deliberately. Only the Kind licenses the fold, so it lives at the seam that has one.This lands on both sides, not recall-only: an alert on any replica of a StatefulSet now keys, dedupes and recalls as that StatefulSet, exactly as pods of a Deployment already do. A per-replica incident (one pod on a bad node) still investigates per pod; only the identity the knowledge is filed and looked up under is the workload. If you would rather keep replica identity in the dedup key, the write-side half is the
normalizeResourceNamechange ininternal/curatorand can be reverted on its own.What
providers.NormalizePodNamefolds ONE trailing ordinal after the existing rules, for a name known to be a Pod's. One, not a fixed point, so a StatefulSet namedsts-0keeps its own name for podssts-0-N.Workload.IdentityName/Workload.ResourceIDapply it whenKind == "Pod". Ingestion still keeps the raw pod name: normalization stays a comparison-time concern, asworkload_identity_testpins.Workload.AgreesWithEntryNamereads a stored entry both as written and folded once, so an entry filed under a sibling replica before this change still matches, and one filed under the StatefulSet after it does too.normalizeResourceNametakes theWorkloadnow, soIncidentKey,DupFingerprintand the drafted entry'sresource:read one rule; the recall gate reads it throughrefsAgree.Kind-less requests are untouched: they keep their ordinal and match only themselves.
Migration
One-off, same shape as the CronJob fold: a StatefulSet pod's
IncidentKeychanges bytes, so its recurrence count restarts once on the first firing after upgrade. Existing per-replica KB entries keep matching (the gate reads them folded); the next curation on that workload files under the StatefulSet.Verified
TDD: four new test files failed first (providers did not build; 7 behavioural cases failed for the expected reason), then passed. Full gate:
go build,go vet,go test ./...,gofmt -lclean,hack/lint.shreports 0 issues.