feat: Kubernetes-native VolumeGroupSnapshot on consistency groups (P0-4) - #504
Merged
geoffrey1330 merged 32 commits intoSep 16, 2026
Merged
Conversation
noctarius
force-pushed
the
main
branch
2 times, most recently
from
September 9, 2026 10:21
60dceb7 to
fbaabe4
Compare
…o internal/controllers/replication
…oups and VolumeGroupSnapshot, drop replication
…-time membership check, health precheck, migration exclusion, restore open question
… into design body
Implement the CSI GroupController so a VolumeGroupSnapshot snapshots an
existing, placement-pinned consistency group:
- controlplane: cluster-scoped client calls to resolve a volume's group,
read live membership, and take/get/delete a snapshot generation.
- groupsnapshot.go: Create verifies the source handles equal the group's
current membership (FAILED_PRECONDITION otherwise, §9.2) then takes one
generation; Delete/Get resolve the group from a self-contained group
snapshot id ({cluster}:{pool}:{groupUUID}:{seq}); a missing group or
generation is delete success (§9.3).
- register the GroupController service when the controller implements it;
advertise GROUP_CONTROLLER_SERVICE only when EnableGroupController is set,
so the sanity harness (arbitrary-volume group tests, incompatible with the
persistent-group model, §3/§11.6) is not offered them.
- tests: membership-equality (equal/extra/missing/non-member/two-group),
delete-missing-success, get round-trip, backend-error surfaced, capability
gating. csi-test, csi-lint, and house-style all green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…Migration (design §9.4, §9.5) Two validating webhooks, admission-only (no reconcile): - VolumeGroupSnapshot (§9.4): label check fail-closed (every selected PVC shares one non-empty storage.simplyblock.io/consistency-group value) and membership check fail-open (resolve the group by name, compare the selected lvols to its current members, admit on a backend blip). Rejects a selector that spans groups, matches an unlabeled PVC, matches nothing, or does not equal the group's membership, at kubectl apply instead of at snapshot time. - VolumeMigration (§9.5): resolve the target PV to its backing lvol and refuse the create when the volume, or any sibling sharing its NVMe subsystem, is a consistency-group member (pinned placement, §8.4). Fail-open; the backend refusal is the backstop. - webapi: GetConsistencyGroupByName, GetConsistencyGroupMembers, and group_id on VolumeInfo. Add the external-snapshotter v8 client for the VGS type; register both webhooks in main.go; manifests + helm-sync regenerated. - tests: U-12..U-19 (VGS), U-21..U-24 (migration). operator-test, operator-lint, and house-style green (pre-existing rebalancing.go debt outside the diff aside). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ign §9) Turn on the Kubernetes group-snapshot path the GroupController and webhooks already implement: - ship the external-snapshotter v8.2.0 VolumeGroupSnapshot, VolumeGroupSnapshotContent, and VolumeGroupSnapshotClass CRDs (guarded and resource-policy: keep, like the per-volume snapshot CRDs). - enable the CSIVolumeGroupSnapshot feature gate on the csi-snapshotter sidecar and the snapshot-controller (both already v8.2.0). - add a default VolumeGroupSnapshotClass for csi.simplyblock.io. - grant the csi-snapshotter sidecar SA the volumegroupsnapshotcontents / classes RBAC (the snapshot-controller RBAC already had the group rules). helm lint + template render clean (114 docs, 6 CRDs, the class, both feature gates, sidecar RBAC); house-style green; helm-sync leaves these chart-native files untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
design §4.1 states the cap and its rationale (bounded freeze window), §12 adds the over-cap failure mode, and the test-plan group-size axis notes the cap boundary. Matches MAX_CONSISTENCY_GROUP_MEMBERS in sbcli. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The provisioner never sent consistency_group, so labeled PVCs were created as ungrouped volumes: group_id stayed empty and CreateVolumeGroupSnapshot rejected them with FAILED_PRECONDITION 'not a consistency-group member' (design §4.1, P0-3 — the missing provisioner half of the group-join path). - CreateLVolData carries consistency_group (omitempty). - fetchPVCAnnotations becomes fetchPVCMeta, returning labels too from the one Get; prepareCreateVolumeReq reads storage.simplyblock.io/consistency-group and forwards it. - tests: a labeled PVC forwards the group; an unlabeled one omits it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…erialize VolumeGroupSnapshot members
geoffrey1330
changed the title
…se 3), resolving open question 4
…ign §7.4, Phase 3): CRD with CEL-immutable spec, ref-resolving webhook, statemachine-backed reconciler, unit coverage U-25…U-33
…lone body (design §7.2)
…es Implemented, P0 shipped, §7.4 ownership label documented, unit Test columns filled (31 of 33)
…he storagepool and volumegroupsnapshotops entries, breaking helm lint
geoffrey1330
requested review from
boddumanohar and
noctarius
and removed request for
noctarius
September 14, 2026 09:07
…the 5 QPS default stalled group-member status writes for ~50 minutes at scale
noctarius
reviewed
Sep 16, 2026
…ller to controllers/consistencygroup
boddumanohar
approved these changes
Sep 16, 2026
geoffrey1330
deleted the
support_volume_group_snapshot_consistenct_group
branch
September 16, 2026 15:14
noctarius
added a commit
that referenced
this pull request
Sep 17, 2026
Brings the VolumeGroupSnapshot work (#504), lblk device support (#437), the client-side compression design (#398), and two openapi.json syncs (#541, #549) onto the branch carrying the CRD redesign. Thirteen files conflicted. The eight generated ones — the deepcopy, the StorageNode CRD in its four copies, the manager role, the chart's webhook template, and install.yaml — were resolved by regenerating rather than by editing, since a generated file has no side to prefer. Three were modify-and-delete, where the lblk work edited the v1alpha1 controllers this branch retired: storagenode_controller.go, the per-node ConfigMap writer, and the StorageNodeSet unit test. The deletions stand. Their replacements under internal/controllers/node are what the operator runs, and restoring a v1alpha1 controller to hold a feature would reinstate the model the redesign removed. cmd/main.go took both import sides. One test came across orphaned. TestBuildStorageNodeSetDaemonSet_ConfigGenerator- MountsDevAndSys guards the /dev and /sys mounts node_configure.py's lblk eligibility check needs, and it was written against the v1alpha1 builder; it is rewritten against BuildStorageNodeDaemonSet, which is the builder this operator runs and which carries those mounts. The assertion is about the container rather than about which kind describes it. lblk arrives half-wired, and this commit does not finish it. What works is the cluster half and the workload half: StorageCluster.spec.deviceClass still resolves to sbcli's device_mode, and the DaemonSet's init script still assembles --lblk, --blk-names, --blk-names-exclude, --blk-serials, --jm-percent, and --force-format. What is missing is the field half. The six selectors exist on v1alpha1's StorageNodeSet and StorageNode, which this operator no longer reads, and have no counterpart on v1alpha2; the per-node ConfigMap writer under internal/controllers/node therefore emits none of the LBLK or BLK_ keys the init script reads, so every flag resolves empty. Selecting devices by lblk is inert on this branch until v1alpha2 carries the fields, which is its own change rather than a conflict resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kubernetes-native VolumeGroupSnapshot on consistency groups
Summary
This PR wires simplyblock's standalone consistency groups into the Kubernetes
VolumeGroupSnapshotAPI (groupsnapshot.storage.k8s.io/v1beta1): PVCs labeledstorage.simplyblock.io/consistency-groupjoin one placement-pinned backend group atprovisioning time, and a
VolumeGroupSnapshotover them materializes as onecrash-consistent backend generation with one member
VolumeSnapshotper PVC, restorablethrough the ordinary per-member snapshot path.
The design document and test plan ship in the PR
(
operator/docs/designs/design-consistency-groups.md,operator/docs/tests/test-plan-consistency-groups.md).What's in this PR
CSI driver
storage.simplyblock.io/consistency-grouplabel asconsistency_groupin thevolume-create body, so the control plane joins the volume to the named group and pins
it to the group's node/LVS before placement runs.
restore PVCs carrying the label birth a new placement-pinned group from the clones.
CreateVolumeGroupSnapshotresolves the group fromthe source volume handles, verifies the requested set equals the group's current
membership (backstop for the admission window), and takes one generation.
GetVolumeGroupSnapshotandDeleteVolumeGroupSnapshotoperate on the{cluster}:{pool}:{group}:{seq}group-snapshot id; a missing group or generationdeletes as success (§9.3).
GROUP_CONTROLLER_SERVICE.Operator
VolumeGroupSnapshotOps(§7.4, new kind): a one-shot, Job-analogous operationwith
action: Restore— onePersistentVolumeClaimper member snapshot of thetarget's generation, named
<namePrefix>-<source PVC>, waited to bind, with typedphase/step status, CEL-immutable spec, and an admission webhook that rejects an
unresolvable
volumeGroupSnapshotRefat apply.restore.consistencyGrouplabels theclones so they form a new group;
restore.enablePartialRestoreopts into restoringan incomplete generation, which otherwise fails naming the missing members.
does not resolve to exactly one group's current membership.
consistency-group member; the placement pin is load-bearing for crash consistency.
CSIVolumeGroupSnapshotfeature gate on thecsi-snapshotter sidecar and the
groupsnapshot.storage.k8s.ioRBAC live in theSimplyblockDriverreconciler (the chart no longer templates the CSI workloads).Helm chart
CSIVolumeGroupSnapshotfeature gate, the webhook manifests, and the newVolumeGroupSnapshotOpsCRD. ASimplyblockDriverCR is seeded inoperator_customresources.yaml(its name derives the adopted workload names).VolumeGroupSnapshotClassships: classes are the user's to author(the regression script creates its own).
createonvolumesnapshots(matching upstream external-snapshotter v8.2.0) — the group flowis the one place the controller creates member
VolumeSnapshotobjects itself.Companion control-plane PR (sbcli
cg-standalone-impl)The backend half: standalone
ConsistencyGroupmodel, join-at-create and join-at-clonewith placement pinning, the group-snapshot generation endpoints, and the
sbctl cgcommands — plus the fixes this PR's live validation surfaced: group records written
outside their FDB keyspace, a controller import cycle, the
ns_iddefault leaking intoclone placement, detached members of a never-snapshotted group counted forever, and
controller refusals surfacing as opaque 500s instead of a 422 with the reason.
Validation
Go tests: GroupController create/get/delete against a mock control plane, admission
tables for all three webhooks, label forwarding on create AND clone, and the
VolumeGroupSnapshotOpscontroller (claim derivation, group labels, partial-restoregate, collision handling, readiness waiting, bind-to-Succeeded, terminal inertness).
Live cluster,
regression_test/20/test_volume_group_snapshot.sh:group_id)VolumeGroupSnapshotgoes ReadyToUse; content carries the handle; one memberVolumeSnapshotper PVCVolumeMigrationof a member rejected at apply (§9.5 webhook)VolumeGroupSnapshotOpsrestores a generation (bound claim per member); unresolvable ref rejected at applyrestore.consistencyGroupforms a new pinned backend group from the clonesBIG_FILL_PCTof each member's size), a continuous random-overwrite workload across every member, generations takenBIG_SNAP_INTERVAL(30s) apart — reporting ReadyToUse latency, the max single-I/O stall (the LVS freeze window), and the CoW churn the timings were measured againstO_DIRECT|O_DSYNC, depth 1); 10 mid-workload generations plus a back-to-back pair each verified as a contiguous prefix{0..M}(present + superseded = M + 1); staggered per-volume snapshots flaggedVIOLATIONas the negative controlConsistencyGroupJoinRefusedon the PVCNotes and limitations
CSIVolumeGroupSnapshot; thechart enables both.
freezes for one
bdev_lvol_snapshot_groupcall.ensure_group's read-then-create is notyet transactional, so a burst of first-member creates against multiple API replicas
can still race duplicate groups; tracked for a follow-up.
Test results