Skip to content

Kubescape scan records outlive deleted workloads, so the scan set is not an existence test #3697

Description

@devantler

🤖 Generated by the Agentic Engineer

Why

Kubescape config-scan records are not pruned when the scanned object is deleted, so the scan set
retains entries for workloads that no longer exist. That makes presence in the scan set unusable as an
existence test — which is exactly how a reader naturally uses it.

Measured on the live cluster 2026-09-09 while verifying C-0013 for #3223: 7 tofu/tf-runner-named
scan records persist whose live counterparts are all absent.
tofu-controller has no Deployment,
StatefulSet, DaemonSet or HelmRelease, and its ServiceAccounts, Role, RoleBinding and cache-encryption
Secret were each individually verified gone (its retirement landed via #3480). The scan records outlived
all of it.

Impact

This is a fail-open in the direction of "still there", and it lands on exactly the reasoning security
work depends on:

  1. A sweep that answers "does workload X still exist?" from scan records concludes YES for a workload
    that was deleted.
    On Six flux-system controllers fail C-0013 on a missing runAsGroup, blocking the C-0013 narrowing #3223 the whole blocker was "is tofu-controller gone?" — a scan-record-based
    check would have answered wrongly and kept a Security issue parked. It stayed parked for 22 days on a
    blocker whose dependency had in fact shipped.
  2. It inflates finding counts against phantom workloads. Any posture or compliance number computed
    over the scan set counts controls for objects that cannot be remediated, because there is nothing
    left to remediate. Work aimed at driving those numbers to 100% then chases records rather than
    workloads.
  3. It compounds the existing trap in this surface. A LIST of workloadconfigurationscans already
    returns .spec.controls as null, so only a single-object get populates controls; and a config
    scan on this cluster was once dead for 53 days while every surface read healthy. This adds a third
    way for a confident-looking read of the same surface to be wrong — and unlike the other two, it is
    invisible to a liveness check, because the stale records sit inside an otherwise healthy scan.

⚠️ Note this is orthogonal to scan liveness. At the time of measurement the scan was LIVE — last
success 2026-09-08T09:12:36Z, three consecutive daily successes, objects still being written. A fresh
scan does not remove a dead record.

What

Reconcile the scan set against live objects so a record cannot outlive its subject. Options, cheapest
first — the point is that something must own deletion:

  • Prefer a scanner-side setting if one exists to prune or expire records for absent objects.
  • Otherwise a reconciling sweep that deletes scan records whose owning object no longer resolves,
    keyed on the object reference the record already carries. Deleting a record is not a posture change,
    so this is safe in a way that changing a control is not.
  • Failing both, stop treating the scan set as an object inventory: derive existence from the live API
    and use scan records only for the verdicts of objects independently confirmed to exist. This is the
    fallback, not the fix — it leaves the inflated counts in place.

Acceptance criteria

  • A scan record for a deleted workload does not survive reconciliation, proved by deleting a throwaway
    workload and observing its record go away.
  • Posture and compliance counts are computed only over workloads that currently exist, so a deletion
    reduces the denominator rather than leaving a permanently-unremediable finding.
  • Whatever mechanism is chosen, a negative control shows the sweep does not delete a record whose
    object still exists.
  • The 7 known tofu/tf-runner orphans are gone.

Rough size: S–M.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions