Skip to content

fix(image): the takeover proof waits for its target and takes back what a failed proof left - #41

Merged
rducom merged 1 commit into
mainfrom
fix/takeover-proof-eagain-and-clean
Nov 12, 2026
Merged

rducom merged 1 commit into
mainfrom
fix/takeover-proof-eagain-and-clean

Conversation

@rducom

@rducom rducom commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

spdk-takeover-proof had three defects, met on a node in a rolling upgrade. One of them held that node out of every upgrade until someone stepped in by hand.

  • A not-yet read as a verdict. A disk's start or recovery right after ublk_create_target can answer -EAGAIN, because the poll groups set up their buffers on their own threads after the target answered (0058). The return phase took that for a failure. Starts and recoveries now wait up to 10 s while the target answers -EAGAIN.
  • A failed phase left disks behind. It stopped only the disks its process served; those it had left to the kernel stayed quiesced, holding the proof's ids. It now takes them back and stops them too, best effort.
  • The clean could not clean. It took back a leftover disk over a 64 MiB null bdev, and the recovery refuses a bdev of another size than the disk's (0070). So since 0070, every proof after a failed one failed at clean, saying "another process serves it", which nothing did. It now uses a bdev of the disk's own size, read from /sys/block/ublkb<id>/size, or the proof's lvol size if unreadable.

ublk-functional.sh adds the case: a disk the create phase left to the kernel is taken back and stopped by the next clean.

Not changed: ublk_create_target still answers before its poll groups are ready, and starts and recoveries still answer -EAGAIN until then (0058's contract). Making the target's creation wait would turn the RPC asynchronous across the later ublk patches. The callers that start disks long after the target's creation never meet the window, and the proof now waits for it.

Test: ublk-functional.sh (the proof's phases against this image, then the leftover case). The tool builds with -Wall -Wextra -Werror.

…at a failed proof left

Three defects of spdk-takeover-proof, met on a node:

- A disk's start or recovery right after ublk_create_target can answer
  -EAGAIN: the poll groups set up their buffers on their own threads
  after the target answered (0058). The return phase took that
  not-yet for a verdict. Starts and recoveries now wait up to 10 s
  while the target answers -EAGAIN.
- A failed phase stopped only the disks its process served. Those it
  had left to the kernel stayed quiesced, holding the proof's ids. It
  now takes them back and stops them too, best effort.
- The clean phase took back a leftover disk over a 64 MiB null bdev.
  The recovery refuses a bdev of another size than the disk's (0070),
  so since 0070 the clean could not clean anything, and said "another
  process serves it". It now uses a bdev of the disk's size, read from
  /sys/block/ublkb<id>/size (the proof's lvol size if unreadable).

ublk-functional.sh adds the case: a disk the create phase left to the
kernel is taken back and stopped by the next clean.
@rducom
rducom requested a review from a team as a code owner November 12, 2026 12:21
@rducom
rducom merged commit ea7d4ce into main Nov 12, 2026
13 of 22 checks passed
@rducom
rducom deleted the fix/takeover-proof-eagain-and-clean branch November 12, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant