Skip to content

fix(atlas-lib): prune stale LVM device entries on force-released teardown - #545

Closed
boddumanohar wants to merge 1 commit into
mainfrom
fix/lvm-devices-hygiene
Closed

boddumanohar wants to merge 1 commit into
mainfrom
fix/lvm-devices-hygiene

Conversation

@boddumanohar

Copy link
Copy Markdown
Member

Summary

Q5 in design-issue-277-client-side-compression.md: the node's LVM devices file (/etc/lvm/devices/system.devices, which restricts LVM's default device visibility) never gets pruned after a device disappears, so a node accumulates one stale entry per failure cycle over its lifetime.

lvmVolumeGroup.Release (atlas-lib/volstack/layers/lvmvolumegroup.go) already has the right force path for this: when vgchange -an fails because the backing device vanished without a clean unstage, it falls back to removing the orphaned device-mapper nodes directly (Manager.RemoveOrphanedDMNodes). That's exactly where the stale system.devices entry is also known to be stale, so this adds the pruning step there.

Changes

  • Manager.ForgetDevice (atlas-lib/lvm/devices.go): a new primitive, lvmdevices --deldev <path>, scoped to the device the same way CreatePhysicalVolume/RemovePhysicalVolume are.
  • lvmVolumeGroup.Release now calls it, once per device it was handed, after falling back to the device-mapper cleanup. It's best-effort: a failure is logged (warnf, matching atlas-lib/lvm/vdo's own Logger convention, introduced here for volstack/layers since it didn't have one yet) and does not fail Release — this is hygiene, and it must never turn a routine unstage into a failed one.

Testing

Followed this repo's red-first workflow: added TestManager_ForgetDevice and TestLVMVolumeGroupReleaseForgetsDevicesOnTheForcePath first, confirmed both failed (the first on a missing method, the second because the layer's Release didn't call it yet), then implemented.

  • go test -race ./lvm/... ./volstack/... — all green, including a new test that Release still succeeds when ForgetDevice itself errors.
  • golangci-lint run ./lvm/... ./volstack/... — 0 issues.
  • make test fails locally, but on an unrelated, pre-existing issue: its vet target pulls in a buf-generated dependency chain that tries to fetch github.com/google/cel-go and the environment here can't reach it. Not something this change touches.

One thing worth flagging before merge: I don't have lvm2/lvmdevices available in this environment, so I could not verify against a real system what lvmdevices --deldev actually prints when a device isn't in the devices file. ForgetDevice doesn't special-case that (no isNoXxx-style error matching, unlike this package's other idempotency helpers), and the caller in Release treats any failure as best-effort/logged rather than fatal, specifically to sidestep needing to know that exact text. Worth a quick live check before this ships, the way this package's other isNoXxx helpers were confirmed against a real cluster.

Out of scope, flagged separately

While reading fabric.go for this, Observe returns StateAbsent with an empty Artifact{} when a device has vanished from the kernel entirely (not just unreachable). lvmPhysicalVolume.Observe and lvmVolumeGroup.Observe both hard-error on zero devices below them ("the layer below exposes no device[s] to..."), and Down's survey() aborts on the first Observe error. That would mean a total, complete device loss on a volume with an LVM stack fails teardown outright, rather than reaching the force path this PR extends — which seems to work against design-node-volume-stack.md §7.4's stated goal that bring-down "tolerates a dead foundation." Not verified against a live cluster, not fixed here — a separate investigation.

🤖 Generated with Claude Code

…down

When a volume group's backing device disappears without a clean
unstage, lvmVolumeGroup.Release already falls back to removing the
orphaned device-mapper nodes directly. Nothing prunes the matching
entry in this host's LVM devices file (/etc/lvm/devices/system.devices),
so a node accumulates one stale entry per failure cycle over its
lifetime.

Add Manager.ForgetDevice (lvmdevices --deldev, scoped to the device)
and call it from the same force path, once per device the layer was
handed. It's best-effort: a failure is logged, not propagated, since
this is hygiene and must never turn a routine unstage into a failed
one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
boddumanohar added a commit that referenced this pull request Sep 16, 2026
…#277)

Implements design-issue-277-client-side-compression.md against current main
rather than rebasing the abandoned PR #402 branch, which predates the
csi-driver restructuring (#497), the v1alpha2 CRD redesign, and the
SimplyblockDriver-managed DaemonSet (#513). Built on atlas-lib/volstack's
three LVM layers per the user's request, in place of PR #402's flat
atlas-lib/lvm/vdo package, which retires here with zero importers.

- atlas-lib/lvm: a built-in "vdo" VolumeProvisioning handler, registered by
  the package's own init (the registry existed with nothing real registered
  against it — CreateLogicalVolume silently dropped compression/deduplication
  before this). Manager.ForgetDevice/HasOrphanedDMNodes, new lvmdevices/
  dmsetup primitives.
- atlas-lib/volstack/layers: lvmPhysicalVolume, lvmVolumeGroup, and
  lvmLogicalVolume now tolerate total path loss (the member device gone
  entirely, not merely unreadable) without erroring the whole Down walk;
  lvmVolumeGroup.Release forgets stale system.devices entries (folds in the
  still-open PR #545).
- atlas-lib/kube: client_compression/client_deduplication StorageClass
  parameters, storage.simplyblock.io/vdo-capable label and its managed-by
  annotation, the shared vdo-capable marker path.
- operator: VolumeDefaults.EnableClientCompression/EnableClientDeduplication
  (v1alpha2), threaded through ClassParameters and the v1alpha1 conversion's
  hub-only stash; the csi-node DaemonSet's postStart hook probes dm-vdo
  alongside its existing nvme-tcp/nvme-rdma modprobes (no hostPID needed —
  the existing probes already prove that), with RBAC to self-label.
- csi-driver: vdoCapableSegment (twin of dhchapAllowedNodeSegment) pins PV
  nodeAffinity from CreateVolume, deliberately not via StorageClass
  AllowedTopologies — that mechanism was found broken for DHCHAP and removed
  in PR #484, for a reason that applies identically to a self-probed
  capability label. A rawDeviceLayer adapter lets the three LVM layers run
  through the real volstack.Runner without adopting fabric/filesystem
  (Phase 1, unrelated, unwired work), wired into NodeStageVolume/
  NodeUnstageVolume/restageVolume/NodeExpandVolume only for volumes that
  request either parameter.
- Dockerfile_base: the vdo package (vdoformat), x86_64 only.

Deviations from the merged design are called out inline in the doc's
2026-09-16 revision notes (§5, §7.1, §4.1, Q9).

Not implemented, matching the design's own already-open items: the
zero-capable-node pool event (§5.1, Q13), CSI metrics (§13, Q6), the
SetFeatures live-toggle path (Q1), the PVC-size floor admission webhook (Q2,
despite the doc's stale "Resolved" note — no such webhook exists on main),
and periodic capability re-checking (Q12).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@boddumanohar

Copy link
Copy Markdown
Member Author

Folded into #402 directly (Manager.ForgetDevice in atlas-lib/lvm/devices.go, wired into lvmVolumeGroup.Release's force path, same behavior covered by tests there). Closing in favor of #402.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant