fix(csi-driver): never format a device the plugin could not read

[boddumanohar]

The bug

NodeStageVolume hands every cold-staged device to mount-utils'
SafeFormatAndMount, which probes with blkid and cannot tell a device carrying
no filesystem from one whose reads failed. blkid exits 2 for both,
getDiskFormat maps that single exit code to ("", nil), and mkfs.ext4 -F -m0
runs. A volume behind a degraded NVMe-oF path — reads timing out under
nvme_core.io_timeout, or a controller past its ctrl_loss_tmo — is therefore
reformatted rather than staged. That is how a production cluster lost a volume.

The repair paths were already written against this hazard (restageVolume and
healVolumeBeforePublish use a plain Mount on purpose). Only the cold stage
was exposed, which is every node reboot, pod reschedule, failover clone, and
migrated volume.

Reproduced, not argued

On a live host, an ext4 filesystem under a dm-flakey error_reads table —
reads fail, writes still land, exactly what a stalled path does:

=== BEFORE ===
UUID=5098687b-37de-4a39-9e13-06c2f5210fa5  TYPE=ext4
lost+found  payload.bin  precious.txt

=== reads start failing ===
blkid: exit=2 output=[]
getDiskFormat() -> ("", nil)  ==>  existingFormat == ""  ==>  FORMAT

=== mkfs.ext4 -F -m0 runs ===
mkfs exit=0

=== AFTER ===
UUID=a9fa37da-9fad-45c2-94fe-797d7207a309  TYPE=ext4
lost+found

With reads still failing at mkfs time the outcome is worse than a clean
reformat: the discard lands, mkfs then fails partway, and the device is left
with no valid filesystem at all — an unmountable volume whose data is already
gone, presenting as a mount failure rather than as a reformat.

Both recipes are in operator/docs/tests/test-plan-node-stage-format.md.

The fix

Take the decision away from mount-utils. atlas-lib/blockfs reads the first
128 KiB of the device and reports what it found, keeping "read successfully,
found nothing" apart from "could not read":

Probe result	formatAndMount
ext / XFS / Btrfs signature	mount it, never format
LUKS, LVM2, swap, partition table	refuse: data, even if unmountable
all zeros	format — the only safe case
readable, unrecognized, non-zero	format, with a warning (unchanged behavior)
could not be read	refuse; kubelet retries

mount-utils still performs the format for a device proven blank, so the mkfs
flags, the XFS feature pinning, and the stripe geometry are untouched. Only its
decision is replaced.

Details worth knowing:

• A signature found in a partially read device counts as formatted — the data
  is there whatever became of the rest. Zeros from a partial read never conclude
  blank.
• The probe is bounded at 20s, under the kernel's 30s nvme_core.io_timeout, so
  a stalled path resolves as unreadable instead of hanging the stage. The read
  runs on its own goroutine because a read against a dead path is not
  interruptible.
• A device holding a filesystem is mounted with a plain Mount, matching what
  the restage path already does. ext4 and XFS each replay their own journal.

Refusing to stage is the deliberate trade: an outage is recoverable and a wiped
volume is not.

Relationship to #481

@noctarius's #481 addresses the same incident from the other end, and the two are
complementary rather than alternatives — its per-claim on-disk-filesystem
annotation is a real feature and a genuine second line of defense. They will
conflict textually in nodeserver.go (both add ctx to stageVolume, and both
make the same house-style prose fixes); happy to rebase onto whichever lands
first.

One thing worth checking together, and the reason I did not simply add to #481:
its getDiskFormat keeps upstream's exit.ExitCode() == 2 -> return "", nil, so
an unreadable device still classifies as blank. #481 catches that case through
the claim annotation instead, which covers a volume this driver has staged since
the change but leaves two paths open — a volume with no annotation yet (anything
staged only by an older driver), and any stage where the claim cannot be read,
since annotatedFilesystem treats every error as "no annotation" and continues
to the format. Reading the device closes it without needing API access or RBAC.

Testing

Regression tests proven red first — mkfs.ext4 -F -m0 ran on both a device
holding a filesystem and a device that could not be read — then green.

The format decision lives in mount-utils' mount_linux.go, so these tests only
exercise anything on Linux; on macOS FormatAndMountSensitive mounts without
probing and they pass vacuously. Build and run them on Linux:

CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go test -c -o /tmp/spdk.test ./pkg/spdk/
/tmp/spdk.test -test.run TestStageVolume -test.v

• 16 probe cases in atlas-lib/blockfs, including EIO, timeout, cancellation, and
  partial reads, under -race.
• 6 decision cases in csi-driver/pkg/spdk, covering both refusals, the blank
  device that must still be formatted, the mismatched filesystem, and raw block.
• Full pkg/spdk on Linux: 36 passed, 0 failed, CSI sanity suite included.
  csi-driver and atlas-lib suites green under -race; make lint clean in
  both; all 9 house-style gates pass.

Live, on a real volume

Against a real simplyblock ext4 volume on a four-node K3s cluster, with blkid
reporting nothing (TestLive*, env-gated and skipped by default):

TestLiveStageVolumeDoesNotReformatARealVolume
  commands issued while staging: []
  staged filesystem contains: [lost+found payload.bin payload.md5 precious.txt]
  precious.txt survived: PRODUCTION DATA - MUST SURVIVE A RESTAGE

TestLiveUpstreamFormatDecisionOnARealVolume
  commands mount-utils chose for that same device:
    [blkid -p -s TYPE -s PTTYPE -o export /dev/nvme0n1] [mkfs.ext4 -F -m0 /dev/nvme0n1]

TestLiveProbeSeesAFreshVolumeAsBlank
  probe of /dev/nvme2n1: state=Blank

The fixed path issued no commands at all — it read the device, saw ext4, and
mounted it. A freshly provisioned lvol still reads as blank, so first-time
formatting is unaffected.

Not yet deployed end-to-end. Everything live ran as a cross-compiled test
binary; no cluster runs this image yet.

Upstream

kubernetes/kubernetes#140376
— same defect, reached through an fsck that corrupted the primary superblock,
open and untriaged since 2026-07-09 with no fix PR. Neither approach proposed
there covers a device that will not answer a read, so this is ours to fix.
mount-utils itself has issues disabled; it is a publishing mirror.

🤖 Generated with Claude Code

[boddumanohar]

Pushed two commits that support the annotation mechanism in #481 rather than the probe in this PR, because both are needed for that mechanism to work on a real cluster and neither is in #481 today. Happy to move them there instead — say the word and I'll open them as a separate PR against blkid-workaournd.

The node plugin cannot write the annotation. simplyblock-csi-node-role grants get, list, watch on persistentvolumeclaims and nothing else, so the merge patch in recordOnDiskFilesystem is refused. The refusal is logged and swallowed — correctly, since a volume must still mount when the API is unreachable — which means the annotation silently never appears and the volume it protects stays exposed. On the test cluster the live role reads ["get","list","watch","patch","patch"], so the grant was added there by hand while testing; it is not in the chart source or in #481's diff, which is why this wasn't caught.

Both charts that grant the node plugin claim access now grant patch too, with the justification on the rule. csi-driver/deploy/kubernetes/node-rbac.yaml grants no claim access at all, so that path cannot use the annotation either way and I left it alone.

Nothing back-fills existing volumes. csi-driver/scripts/backfill-on-disk-filesystem.sh annotates the fleet in one pass, so the upgrade window closes immediately instead of one volume at a time. It writes only for a claim a running pod is using — a volume mounted into a running pod has been formatted, whereas a merely bound claim may never have been staged — and records the filesystem its StorageClass asks for. Raw block volumes, unbound claims, other provisioners, and claims that already record a filesystem are skipped. Dry run unless given --apply.

Verified against a four-node cluster: 6 simplyblock claims in use identified with the right filesystem, 8 local-path claims, one pending claim, and one raw block volume skipped for the right reason; a throwaway claim annotated, re-read, and skipped on a rerun.

Correction to what I wrote earlier in this PR

I said the annotation "leaves two paths open," which understated it in one direction and overstated it in another, and I have since tested both.

It is better than I said: the plugin writes the annotation on every successful stage, not only for newly provisioned volumes. An existing ext4 volume whose blkid works gets annotated on its first stage under the new driver, so a cluster back-fills itself as it runs, even without the script.

It is also narrower than I implied. The exposure is the window between deploying #481 and each volume's next successful stage — not the fleet indefinitely. What stays open is a volume that meets an unreadable device on its first stage after the upgrade (a node reboot is both the commonest cause of a cold stage and a moment when paths flap, so this ordering is not contrived), a claim recreated or restored without its annotation, and any stage where the claim cannot be read.

Tested on this branch, against a real ext4 filesystem under a dm-flakey error_reads table with the real blkid:

claim annotation	#481 result
absent	reformatted, data lost
ext4	mounted, data intact

and the same device, same failing reads, no annotation anywhere, on this PR:

refusing to stage: it could not be read (input/output error),
and a device that will not answer a read must not be assumed empty
→ UUID unchanged, precious.txt and payload.bin intact

The two remain complementary. The annotation is a real feature and a cheap second opinion; the probe is what holds when no record exists yet.