[bug] Deleting/redistributing a target leaves it orphaned (still running) on the gNMIc pod that previously held it

[vuthinh0301]

[bug] Deleting/redistributing a target leaves it orphaned (still running) on the gNMIc pod that previously held it

Summary

When a Target is removed (deleted directly, or removed from a TargetSource inventory), the operator deletes the Target CR and recomputes distribution — but a pod that ends up with 0 assigned targets is skipped during config apply, so it never receives a pruning config/apply. The gNMIc pod keeps running the removed target: an open gNMI subscription to a device that is no longer in the desired state.

For production (especially telecom dial-in), this means a decommissioned/removed router keeps being polled invisibly — wasted gNMI sessions on the device control plane, stale data flowing to outputs, and a risk of collecting from a wrong device if the address is later reused.

Environment

• operator built from main @ commit 63a262b1a56a8897af9a3e055b99db2b30aeed6b (2026-07-05)
• kind cluster, 3-replica Cluster, gNMIc pods ghcr.io/openconfig/gnmic:latest
• targets fed via a TargetSource (http provider) and also a directly-created Target

Reproduce

1. Run a Cluster with replicas: 3 and a few targets so that pods each hold ≥1 target (e.g. srl-a→pod-1, srl-direct→pod-2, srl-b→pod-0).
2. Remove srl-b (delete the Target CR, or remove it from the TargetSource HTTP inventory).
3. Observe: the Target CR is deleted correctly, and distribution recomputes so pod-0 now has 0 assigned targets.

$ kubectl -n telemetry get targets
NAME                 ADDRESS            PROFILE
db-inventory-srl-a   172.22.0.6:57400   default-profile
srl-direct           172.22.0.6:57400   default-profile
# (db-inventory-srl-b is gone from CRs — correct)

# but the pod that held it still runs it, minutes later:
$ kubectl -n telemetry exec gnmic-telemetry-cluster-0 -- \
    wget -qO- http://localhost:7890/api/v1/config/targets
{"telemetry/db-inventory-srl-b": { ... }}   # connection-state READY, still polling

Expected vs actual

• Expected: after srl-b is removed, the pod that ran it stops collecting it (target pruned from that pod).
• Actual: srl-b keeps running on pod-0 indefinitely (verified 5+ minutes later). Only pods that still have ≥1 assigned target get re-applied config; a pod that drops to 0 targets is never touched.

Root cause

internal/gnmic/distribute.go — PerPodPlans is populated only from newAssignment, i.e. only pods that receive at least one target. A pod that ends up with 0 targets gets no entry:

result := make(map[int]*ApplyPlan)
for podIndex, targets := range newAssignment {   // no entry for empty pods
    result[podIndex] = &ApplyPlan{ ... }
}
return &DistributeResult{PerPodPlans: result, ...}

internal/controller/cluster_controller.go — applyConfigToPods() then continues past any pod not present in PerPodPlans, so it never sends that pod a config/apply:

for podIndex := 0; podIndex < numPods; podIndex++ {
    podPlan, ok := distResult.PerPodPlans[podIndex]
    if !ok {
        continue   // pod that dropped to 0 targets is skipped → stale targets never pruned
    }
    url := fmt.Sprintf("%s://%s:%d/api/v1/config/apply", scheme, podDNS, restPort)
    r.sendApplyRequest(ctx, url, podPlan, httpClient)
}

Since gNMIc's POST /api/v1/config/apply prunes resources not present in the request, a pod that transitions from N→0 targets needs an (empty) apply to have its previous targets removed. Because that apply is never sent, the removed target stays running.

Suggested fix

In applyConfigToPods, iterate all pods 0..numPods and send an empty ApplyPlan to pods missing from PerPodPlans (instead of continue), so their stale targets/subscriptions get pruned. For example, fall back to an empty plan carrying the shared subscriptions/outputs but no targets:

podPlan, ok := distResult.PerPodPlans[podIndex]
if !ok {
    podPlan = &gnmic.ApplyPlan{Targets: map[string]*gapi.TargetConfig{}} // prune this pod
}

(or have DistributeTargets emit an empty ApplyPlan for every pod index in 0..numPods.)

Impact

High for dial-in production: removing a device from the source of truth does not stop collection on the pod that held it, leaving orphaned gNMI sessions and stale telemetry until that pod is restarted or happens to be reassigned a target again.

[karimra]

TargetSource is not a released feature.