docs: design doc for client-side compression (VDO/dedup) — issue #277 by boddumanohar · Pull Request #398 · simplyblock/simplyblock-operator

boddumanohar · 2026-08-05T11:37:30Z

Summary

Design document for issue #277 (client-side compression and deduplication). Most of the mechanism was implemented and live-verified end-to-end on a real cluster; the node-capability probe was reworked during review (§4.1) to drop the RHEL-specific install path in favor of a single modprobe dm-vdo check, which has not yet been re-verified live.

Covers: node-capability detection and advertisement (plus an operator-override mechanism for a golden-image node), topology-based scheduling gate, StorageClass/CRD changes (compression and deduplication as two independent, separately-switchable parameters), VDO device management (create/reattach/grow), performance characteristics, re-provisioning/failure handling, and a compatibility review against existing CSI driver/operator features (Guardian, VolumeMigration, snapshots/clones, encryption).

Implements the design in design-issue-277-client-side-compression.md (PR #398): new clientCompression/clientDeduplication Pool StorageClass params, VDO-capable node topology gating, a new csi-driver/pkg/util/vdo.go managing per-volume VDO stacks over LVM, and nodeserver.go wiring to create/reattach/grow/remove VDO devices across stage/unstage/restage/expand. Not yet covered (tracked as follow-ups): unit tests, deliberate exercise of clone/snapshot VDO resolution, multi-instance+reboot, XFS-on-VDO, and crash-consistency of async write policy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

noctarius · 2026-09-09T11:06:12Z

+independent step:
+
+```bash
+nsenter -t 1 -m -u -n -i -- sh -c 'rpm -q kmod-kvdo vdo >/dev/null 2>&1 || dnf install -y kmod-kvdo vdo'


I think we need to guard this. Not everything is Redhat.

dnf is not required altogether. we can first run modprobe dm-vdo if it succeeds we label vdo-capable=true. And nothing related to dnf is required.

noctarius · 2026-09-09T11:09:17Z

+
+The `lvm/vdo` package's public surface is the lifecycle of one volume's stack:
+
+```go


Nice split. Much of that should absorb cleanly into the volume stacks.

yeah, will be part of lvmLogicalVolume layer

noctarius · 2026-09-09T11:10:02Z

+Keeping the wrappers means `nodeserver.go` never assembles an `lvm.Manager` of
+its own, and the primitive stays usable outside a Kubernetes context.
+
+### 7.2 Device Creation


Same here. Much of this is already prepared on the volume stacks. VDO will become a flag of the creation.

as we'll have seen on the call today, LVMLogicalVolumeConfig.Definition already carries Compression and Deduplication variables.

noctarius · 2026-09-09T11:10:50Z

+identical across every ha-mode volume in the cluster. The model field carries
+the lvol's UUID instead.
+
+- **The NVMe-oF and udev layer.** SPDK sets the NVMe namespace UUID equal to the


Ideally, this should already consider the atlas-based resolution of the devices.

initiator.go still resolves devices with its own by-id glob waitForDeviceReady. The switch is not done yet.

noctarius · 2026-09-09T11:12:12Z

+  and `pvscan --cache` rediscovers its PV.
+- **`NodeExpandVolume`** calls `GrowVDO` before the existing filesystem resize,
+  per §9.
+- **`stageVolume`** skips `xfsStripeOptions` whenever VDO is in play. Those


Oh, good one. This needs to be supported in volume stacks.

atlas-lib/volstack's Geometry field already supports this.

noctarius · 2026-09-09T11:20:17Z

+| Q6  | **CSI-side metrics:** §13's metrics need a Prometheus endpoint the CSI driver does not have. Whether to add one for this feature or wait for a driver-wide decision is open                                                                                                                                                                                 | Operator team |
+| Q7  | **`lvm2` VDO segtype detection:** the capability probe checks the kernel module and not whether `lvm segtypes` lists `vdo`. Whether the segtype check belongs alongside `modprobe` is undecided                                                                                                                                                             | Operator team |
+| Q8  | **Non-RHEL and `aarch64` nodes:** both are non-goals today (§2), and P0-4 makes `aarch64` a packaging problem rather than a design one. Whether either becomes supported depends on demand                                                                                                                                                                  | Product       |
+| Q10 | **Where the install runs:** the module install rides the `csi-node` `postStart` hook (§4.1), which puts `dnf` in the node plugin's readiness path. Whether a short-lived pod, started when a node needs it, should own the install instead is undecided                                                                                                     | Operator team |


I think a separate installer may be better. Can be the same image but different pod.

dnf is not required altogether. we can first run modprobe dm-vdo if it succeeds we label vdo-capable=true. And nothing related to dnf is required.

noctarius · 2026-09-09T11:20:45Z

+| Q7  | **`lvm2` VDO segtype detection:** the capability probe checks the kernel module and not whether `lvm segtypes` lists `vdo`. Whether the segtype check belongs alongside `modprobe` is undecided                                                                                                                                                             | Operator team |
+| Q8  | **Non-RHEL and `aarch64` nodes:** both are non-goals today (§2), and P0-4 makes `aarch64` a packaging problem rather than a design one. Whether either becomes supported depends on demand                                                                                                                                                                  | Product       |
+| Q10 | **Where the install runs:** the module install rides the `csi-node` `postStart` hook (§4.1), which puts `dnf` in the node plugin's readiness path. Whether a short-lived pod, started when a node needs it, should own the install instead is undecided                                                                                                     | Operator team |
+| Q11 | **Which kernel the target distributions ship:** whether the install path or the in-tree `dm-vdo` path is the common case depends on the default kernel of each supported node OS, and OpenShift, Rancher, and K3s have not been surveyed against the 6.9 line (§4.1)                                                                                        | Operator team |


I would limit it to kernels that have lvm-vdo support already.

yeah, we can first run modprobe dm-vdo if it succeeds we label vdo-capable=true

noctarius · 2026-09-09T11:21:10Z

+| Q8  | **Non-RHEL and `aarch64` nodes:** both are non-goals today (§2), and P0-4 makes `aarch64` a packaging problem rather than a design one. Whether either becomes supported depends on demand                                                                                                                                                                  | Product       |
+| Q10 | **Where the install runs:** the module install rides the `csi-node` `postStart` hook (§4.1), which puts `dnf` in the node plugin's readiness path. Whether a short-lived pod, started when a node needs it, should own the install instead is undecided                                                                                                     | Operator team |
+| Q11 | **Which kernel the target distributions ship:** whether the install path or the in-tree `dm-vdo` path is the common case depends on the default kernel of each supported node OS, and OpenShift, Rancher, and K3s have not been surveyed against the 6.9 line (§4.1)                                                                                        | Operator team |
+| Q12 | **Re-checking capability:** the probe runs at `csi-node` pod start and never again, so a node that becomes capable without a restart stays labeled `false`, and an install slower than the probe's five-minute wait leaves that label behind durably (§4.3). Whether the probe should re-check on an interval, and at what cost in API writes, is undecided | Operator team |


It definitely has to run again after the install step.

Agreed, updated the doc. It has to re-check, not just run once at start

noctarius · 2026-09-09T11:21:37Z

+| Q10 | **Where the install runs:** the module install rides the `csi-node` `postStart` hook (§4.1), which puts `dnf` in the node plugin's readiness path. Whether a short-lived pod, started when a node needs it, should own the install instead is undecided                                                                                                     | Operator team |
+| Q11 | **Which kernel the target distributions ship:** whether the install path or the in-tree `dm-vdo` path is the common case depends on the default kernel of each supported node OS, and OpenShift, Rancher, and K3s have not been surveyed against the 6.9 line (§4.1)                                                                                        | Operator team |
+| Q12 | **Re-checking capability:** the probe runs at `csi-node` pod start and never again, so a node that becomes capable without a restart stays labeled `false`, and an install slower than the probe's five-minute wait leaves that label behind durably (§4.3). Whether the probe should re-check on an interval, and at what cost in API writes, is undecided | Operator team |
+| Q13 | **Reporting an unsatisfiable pool:** §5.1's capable-node count and its `VDOPoolUnsatisfiable` event are not implemented. Whether the pool's view refreshes through a node watch rather than the reconciler's requeue, and whether the fact earns a durable status field of its own, is undecided                                                            | Operator team |


Yes, needs to reverify the claim.

noctarius · 2026-09-09T11:22:08Z

+| Q11 | **Which kernel the target distributions ship:** whether the install path or the in-tree `dm-vdo` path is the common case depends on the default kernel of each supported node OS, and OpenShift, Rancher, and K3s have not been surveyed against the 6.9 line (§4.1)                                                                                        | Operator team |
+| Q12 | **Re-checking capability:** the probe runs at `csi-node` pod start and never again, so a node that becomes capable without a restart stays labeled `false`, and an install slower than the probe's five-minute wait leaves that label behind durably (§4.3). Whether the probe should re-check on an interval, and at what cost in API writes, is undecided | Operator team |
+| Q13 | **Reporting an unsatisfiable pool:** §5.1's capable-node count and its `VDOPoolUnsatisfiable` event are not implemented. Whether the pool's view refreshes through a node watch rather than the reconciler's requeue, and whether the fact earns a durable status field of its own, is undecided                                                            | Operator team |
+| Q9  | ~~**Where the VDO and LVM primitives live**~~ **Resolved.** They are `atlas-lib` packages, `lvm` and `lvm/vdo`, and the CSI driver holds only the wrappers in `csi-driver/pkg/util/vdo.go` (§7.1)                                                                                                                                                           | Resolved      |


It'll be absorbed into the volume stacks. For now, this layout is good enough.

…277 Design document and companion test plan for client-side compression and deduplication: a VDO device between the NVMe-oF multipath device and the filesystem mount, assembled on the node that consumes the volume. Covers node capability detection and auto-install with an operator override, the topology scheduling gate, the StorageClass and CRD parameters as two independent switches, VDO device management (create, reattach, clone identity resolution, grow), re-provisioning and failure handling, and a compatibility review against Guardian, VolumeMigration, snapshots and clones, and encryption. Review feedback addressed: - Report a pool that no node in the cluster can serve, rather than leaving its PVCs Pending behind a scheduler message that names neither the label nor the feature (§5.1, Q13). - Align the capability label with the storage.simplyblock.io/ guideline. - Drop the standalone vdo CLI non-goal, since LVM-integrated VDO is the interface this design uses. - Replace the golden-image airgapped path with a mirrored BaseOS repository, and narrow the non-goal to a node that can reach no repository at all. - Say why the install rides the node plugin's own lifecycle, and name its cost (Q10). - Record that each distribution's default kernel decides whether the install path or in-tree dm-vdo is the common case (Q11). - Record that the capability probe is triggered by pod start alone, including the five-minute wait that turns a slow install into a durable wrong answer (§4.3, Q12). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Resolves the review comments that had a concrete answer, and reconciles the doc where one resolution's fallout changed another: - Guard the postStart install step so it does not assume RHEL. - Q1: SetFeatures is a live lvchange, not a reformat. - Q2: a PVC-admission webhook rejects a sub-floor size at creation. - Q4: the operator will not apply the out-of-service taint. - Q5: verified against the code, not actually handled by the volume stacks yet, corrected rather than assumed. - Q6: csilink already exists as the route for CSI-side metrics. - Q8: aarch64 is a goal for final release, and non-RHEL support resolves for free once Q11 drops the RHEL-only install path. - Q10/Q11: drop the dnf/kmod-kvdo install path entirely in favor of a single modprobe dm-vdo probe; Q10's separate installer pod is superseded, since there is no longer a slow install step to move out of the readiness path. Q3 is absorbed into Q11 the same way. - Reconcile every section that described the dropped install path: §1.1, §4.1 (renamed "Module Load and Detection"), the architecture diagram, §4.3, the P0 table, and §2's Non-Goals. - Note where §7.1 and §7.6 already have generalized counterparts in atlas-lib/volstack, per design-node-volume-stack.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Brings the VolumeGroupSnapshot work (#504), lblk device support (#437), the client-side compression design (#398), and two openapi.json syncs (#541, #549) onto the branch carrying the CRD redesign. Thirteen files conflicted. The eight generated ones — the deepcopy, the StorageNode CRD in its four copies, the manager role, the chart's webhook template, and install.yaml — were resolved by regenerating rather than by editing, since a generated file has no side to prefer. Three were modify-and-delete, where the lblk work edited the v1alpha1 controllers this branch retired: storagenode_controller.go, the per-node ConfigMap writer, and the StorageNodeSet unit test. The deletions stand. Their replacements under internal/controllers/node are what the operator runs, and restoring a v1alpha1 controller to hold a feature would reinstate the model the redesign removed. cmd/main.go took both import sides. One test came across orphaned. TestBuildStorageNodeSetDaemonSet_ConfigGenerator- MountsDevAndSys guards the /dev and /sys mounts node_configure.py's lblk eligibility check needs, and it was written against the v1alpha1 builder; it is rewritten against BuildStorageNodeDaemonSet, which is the builder this operator runs and which carries those mounts. The assertion is about the container rather than about which kind describes it. lblk arrives half-wired, and this commit does not finish it. What works is the cluster half and the workload half: StorageCluster.spec.deviceClass still resolves to sbcli's device_mode, and the DaemonSet's init script still assembles --lblk, --blk-names, --blk-names-exclude, --blk-serials, --jm-percent, and --force-format. What is missing is the field half. The six selectors exist on v1alpha1's StorageNodeSet and StorageNode, which this operator no longer reads, and have no counterpart on v1alpha2; the per-node ConfigMap writer under internal/controllers/node therefore emits none of the LBLK or BLK_ keys the init script reads, so every flag resolves empty. Selecting devices by lblk is inert on this branch until v1alpha2 carries the fields, which is its own change rather than a conflict resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

boddumanohar marked this pull request as draft August 5, 2026 11:54

boddumanohar force-pushed the design-issue-277-client-side-compression branch from b89fa3b to 8a74d82 Compare August 6, 2026 07:41

boddumanohar changed the title ~~docs: design doc for client-side compression (VDO) — issue #277~~ docs: design doc for client-side compression (VDO/dedup) — issue #277 Aug 6, 2026

boddumanohar marked this pull request as ready for review August 6, 2026 07:53

This was referenced Aug 6, 2026

bug: upsertStorageClass is create-only — Pool StorageClassParameters edits on an existing Pool silently no-op #401

Closed

feat: client-side compression/dedup via VDO (issue #277) #402

Open

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 2cd09f1 to 4f4de85 Compare August 21, 2026 12:01

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 4f4de85 to 86d6f32 Compare August 25, 2026 10:26

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 6c335f9 to 72e8057 Compare August 31, 2026 13:43

noctarius reviewed Sep 1, 2026

View reviewed changes

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 4fa3a57 to 14706d6 Compare September 1, 2026 11:06

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 14706d6 to 5e79313 Compare September 8, 2026 12:11

noctarius force-pushed the main branch 2 times, most recently from 60dceb7 to fbaabe4 Compare September 9, 2026 10:21

noctarius reviewed Sep 9, 2026

View reviewed changes

boddumanohar and others added 2 commits September 15, 2026 13:44

boddumanohar force-pushed the design-issue-277-client-side-compression branch from 5e79313 to 0371134 Compare September 15, 2026 15:35

noctarius approved these changes Sep 15, 2026

View reviewed changes

noctarius merged commit b10db7a into main Sep 15, 2026
22 checks passed

noctarius deleted the design-issue-277-client-side-compression branch September 15, 2026 15:53


		The `lvm/vdo` package's public surface is the lifecycle of one volume's stack:

		```go

Conversation

boddumanohar commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

boddumanohar commented Aug 5, 2026 •

edited

Loading