enhancement: The sliced working-dir GC deletes live partition ownership records, and reclaim then destroys the partitions

[thxCode]

What would you like to be added:

Stop the logical-slicing working-directory GC from deleting the hardware-partition ownership
records that live in the same per-Pod tree. Today it removes the whole <OperatorPodsDir>/<podUID>
directory, and a partition whose record is gone is indistinguishable from an abandoned one — so the
vendor reclaim loop destroys it while its Pod is alive.

Note: this reads as a defect rather than a feature; it is filed as an enhancement per the reporter's
preference. It is pre-existing on main and affects every partitioning manufacturer — NVIDIA,
T-Head and Hygon alike.

Why is this needed:

The two surfaces share one directory tree

A hardware partition's ownership record is written under the per-Pod tree:

Manufacturer	Record path
NVIDIA	deviceplugin.OperatorPodsDir/<podUID>/c-<container>/… (nvidia/mig.go#L622)
T-Head	same tree (thead/mig_visibility.go#L101)
Hygon	same tree (hygon/mig.go, migMarkerPath)

The logical-slicing server owns a GC over that same root:

// pkg/deviceplugin/server.go#L117-L119
var gc *podDirGC
if s.AllocationMode == workercore.DeviceAllocationModeSliced {
    gc = newPodDirGC(OperatorPodsDir)
}

and it reclaims by removing the whole Pod directory:

// pkg/deviceplugin/gc.go#L64
if err := os.RemoveAll(filepath.Join(g.dir, uid)); err == nil {

The Sliced server runs even where no logical slice can ever be served. It is constructed on
!opts.NoSliced alone, independent of whether the node's cards advertise any sliced capability — so
on a Hygon node in MIG mode, where hygon.com/dcu.sliced is 0, the GC is running over the tree
that holds every live partition's record.

The failure

podDirGC.reconcile is driven per broadcast from notifyListeners, which fires on every reconcile
and synchronously from reserveDevices/releaseReservation. A freshly admitted Pod is not in
lastLivePodUIDs until the reconcile that observes its allocation annotation, so a burst of
admissions can deliver several broadcasts in a row that do not name it. Three consecutive misses is
the threshold (podDirGCMaxMisses = 3), and at broadcast rate that is well under a second.

Once the record is gone:

1. scanMigMarkers finds nothing for that Pod, so the partition is in no seen set.
2. The grace window does not help. It guards the record-names-a-dead-Pod path only; a partition
   with no record at all goes to the orphan sweep, which has no age to check.
3. destroyIfStillUnclaimed re-reads the records under the accelerator lock — and still finds
   nothing, because the record was deleted rather than merely unseen. It destroys the partition.
4. The driver refuses only if a process already holds the device. A container that has been granted
   but has not yet opened /dev/kfd is not protected.

The container then starts against a registry file that no longer exists, and the container runtime
creates a directory at that path — which permanently poisons that instance id, since
INSUFFICIENT_RESOURCES then follows the id forever. That secondary failure was observed on
hardware: two ids became permanently uncreatable on a Hygon node.

Why the existing mitigations do not cover it

0d0d1e5-era work added migReclaimGrace and the under-lock re-read specifically because a
stale scan was destroying just-granted partitions. Both fixes assume the record is on disk.
Neither can see a record that was deleted out from under them.

A second, weaker path to the same outcome

RunReclaimLoop's broadcast channel is deliberately lossy (buffered, non-blocking send), and the
60 s resync reuses lastLive from the most recent broadcast that was actually received
(reclaim.go#L44-L60).
Under sustained saturation lastLive can stay stale past the two-minute grace, at which point the
record-names-a-dead-Pod path destroys a live partition on its own. This needs continuous drops for
the whole grace window, so it is far less likely than the GC path above — but it has the same blast
radius and would be fixed in the same neighbourhood.

Suggested fix

Preferred: scope the GC to what it owns. The sliced artifacts live at a known depth under the Pod
directory (c-<container>/etc/vdev/..., .../nvidia/...), so the GC can remove those rather than the
Pod root, and remove the Pod root only when nothing else remains in it.

Cheaper alternative: have podDirGC skip any Pod directory that contains a partition ownership
record. That leaks an empty directory for a genuinely dead Pod's partition until the vendor reclaim
removes the record, which is the safe direction.

Either way the reclaim's own decision would be improved by treating "no record at all" as
inconclusive when the record tree could have been touched — but scoping the GC is what removes the
cause.

Provenance

Found by an independent cross-model review of the Hygon partitioning branch, then confirmed against
the source. Not introduced by that branch — the same tree, the same GC and the same reclaim shape
predate it on NVIDIA and T-Head.

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

A unit test on podDirGC.reconcile is the natural regression guard: a Pod directory holding a
partition ownership record must survive podDirGCMaxMisses consecutive misses.

[thxCode]

A note for whoever picks this up, because a change that just merged looks like it covers this issue
and does not.

#499 fixes a different way the reclaim loop destroys a live pod's partition, and its symptom is the
same sentence as this issue's: the vendor reclaim loop destroys a partition while its Pod is alive.
The root causes are disjoint, and the guard #499 adds is structurally unable to help here.

What #499 fixed. The record survives, but the loop could not tell the partition it recorded from a
successor created at the same placement. Placement selection is the lowest free slot and a MIG-device
identifier is name-based, so the recorded id, the placement and the identifier are all reproduced by
whoever takes the slot next. The fix re-scans the markers inside the accelerator lock and refuses to
destroy a gpu instance some other pod's marker claims.

Why that guard cannot fire in this issue's scenario. It lives in the path that acts on a pod's
markers, and it decides by asking the marker set who claims the id. Here the markers are gone -- the
working-dir GC removed the whole per-Pod tree -- so:

• the per-pod destroy path is never entered for the affected pod at all, because there is no record to
  act on; and
• destroyOrphans is entered instead, and its own gate reads
  len(ownedGiIDsOnCard(entries, card)) > 0 || ownershipUnknownOnCard(corrupt, card). A deleted
  record is neither an entry nor a corrupt path. On an accelerator whose only ownership record was the
  deleted one, both terms are false and the gate opens on a live pod's partition.

The corrupt term is worth calling out, because it is the near miss: this codebase already fails
closed on a record it cannot read. A record that was removed leaves nothing to fail closed on. So
the existing conservatism covers unreadable records and not absent ones, and this issue is the absent
case.

What this means for the fix here. Closing this still needs what the description says -- the sliced
GC must stop removing partition ownership records -- and no amount of hardening inside the reclaim
loop substitutes for it, because from inside that loop a deleted record and a partition that was never
recorded are the same observation.

One thing that did change, with a caveat. The e2e side now reports WHERE a failed partition request
died rather than only counting admission errors, because a partition destroyed under a starting
container surfaces as a container-create failure and raises no UnexpectedAdmissionError -- measured.
Whether this issue produces that same shape is NOT measured: the record deleted here belongs to a pod
that is already running, so the partition may well be destroyed out from under a container that has
already started, which would surface differently again. Worth checking against before assuming either
classification covers it.