Better define when a crucible volume can indicate all regions in a region set are valid

[leftwo]

Currently we use the "activation" of a crucible upstairs as an indication that all three of the downstairs in the region set have been made consistent with each other. We use this state to determine that a region replacement has completed and a newly provided downstairs has been made consistent with the other existing downstairs.

When we activate with 2/3 downstairs (see PR oxidecomputer/crucible#1733), that will mean that "activation" is no longer a guarantee that all three downstairs have been made consistent.

We need another solution that the control plane can use to know when a region replacement has reached a state that it is considered completed.

[mkeeter]

Thanks to some pointers from @leftwo , here's some more notes.

Region replacement is handled by region_replacement_drive.rs.

There are two modes of operation:

• If the volume is attached to a VM, then we send a replacement request, then wait for it to finish
• Otherwise, we attach the volume to the Crucible pantry, which performs reconciliation on startup

Let's go into more detail for each of those cases.

Attached to a VM

If the disk is attached to a running vm, we use the DriveAction::Propolis { .. } action, which calls execute_propolis_drive_action. This uses the instance_issue_crucible_vcr_request endpoint in Propolis, which goes through several abstraction layers:

• reconfigure_crucible_volume (in Propolis)
• CrucibleBackend::vcr_replace (in Propolis)
• Volume::target_replace (in Crucible)

The endpoint is idempotent, so it serves as both a "start replacement" and a "check replacement status" request. We eventually get back a ReplaceResult.

If the ReplaceResult is Started, StartedAlready, or CompletedAlready, then we're fine: the replacement is either running or has completed normally. We either continue waiting, or know that the replacement is successful.

Unfortunately, we could also get ReplaceResult::VcrMatches, indicating that we're now talking to a different
Propolis instance and its VCR matches the new requested state. In this case, Propolis may be in the process of performing reconciliation on the Volume to bring it into compliance. We call disk_volume_status to see if the volume is active, which eventually bottoms out at BlockIO::query_is_active. This works in existing code because volumes do not become active until reconciliation is complete. It will not work moving forward.

Not attached to a VM

If the volume isn't attached to a VM, then we attach to the Crucible Pantry. When it is activated, it goes through reconciliation, which brings the Downstairs in sync with each other.

More specifically, we start a background job with attach_activate_background in the Crucible Pantry API, which calls PantryEntry::activate in a spawned task. Down in Crucible, we check Volume::query_is_active then call Volume::activate.

When polled, the saga calls check_from_previous_pantry_step, which calls is_job_finished in the pantry API to check whether the background job has finished. If the job has finished, we call volume_status (also in the pantry API) to check whether the pantry has ever seen the Volume finish activation (which indicates that reconciliation happened).

This has the same problem as the ReplaceResult::VcrMatches case above: activation could finish with only 2/3 downstairs, and will no longer guarantee that reconciliation is complete.

What's to be done?

First, we don't need to worry about the volume replacement path; that doesn't rely on activation to determine when volumes are reconciled.

For the unattached case, one option would be to add a new activate_with_quorum endpoint, which takes a new enum Quorum { Min, Full }. The pantry could then activate with Quorum::Full, so that activation still guarantees that all 3 Downstairs are in sync.

The hard part is handling ReplaceResult::VcrMatches. Even if we add activate_with_quorum as suggested above, we don't want to change Propolis to require full-quorum activation in the typical case, and making it sometimes do full-quorum activation would be awkward. Instead, we could add a new member to the api::VolumeStatus in the Propolis API – something like full_quorum_ready: bool – then use that instead of active: bool.

Do we need to worry about API versioning?

If this lands before we need to care about API versioning, then life is easier: we can change Omicron / Propolis / Crucible Pantry simultaneously. Otherwise, we would need to roll out the new APIs in Propolis + Crucible Pantry first, then start using them in a subsequent update (or gracefully ignore them).

When can we actually deploy min-quorum reconciliation?

We should make the changes to Omicron / Propolis / Crucible Pantry first, then actually change Crucible's behavior to support min-quorum reconciliation. This means github.com/oxidecomputer/crucible/pull/1733 will continue to float for a while.