Add first-class ActorSnapshot lifecycle APIs

[EItanya]

Summary

Substrate already creates an external snapshot whenever it suspends an Actor, but that snapshot is only reachable through the Actor's mutable latest_snapshot_info. A caller cannot address old snapshots, inspect or delete them, or create another Actor from one.

Every successful durable suspension should create an immutable ActorSnapshot resource and make the Actor reference it as its latest snapshot. Add the minimum APIs needed to:

1. return the snapshot created by SuspendActor;
2. get, list, and delete snapshots; and
3. create a new Actor from a snapshot.

Snapshot garbage collection is a separate policy. This proposal only defines the snapshot resource, how Actors refer to it, and when it is safe to delete.

Motivation

Control planes built on Substrate need to preserve an Actor at a known state and later create an independent Actor from that state. A higher-level system may also bind the Actor snapshot to application history or other metadata that Substrate does not own.

Today SuspendActor is sufficient for ordinary suspend/resume, where only the Actor's latest state matters. It is insufficient when an older state must remain independently addressable because a later suspension replaces Actor.latest_snapshot_info, and CreateActor can start only from an ActorTemplate.

This cannot safely be implemented above Substrate by copying Storage fields or object-storage prefixes. That would bypass Substrate's compatibility checks, authorization boundary, ownership of storage layout, and reference tracking.

Current behavior

As of upstream main on 2026-07-24:

• SuspendActor creates an external snapshot and stores it in Actor.latest_snapshot_info.
• ResumeActor restores the Actor's latest external snapshot or its ActorTemplate golden snapshot.
• PauseActor uses node-local snapshots.
• ActorTemplate.spec.snapshotsConfig selects full or data snapshots.
• ActorTemplate.status.goldenSnapshot records the template's initial snapshot.
• The Control API has no snapshot resource or snapshot RPCs.
• Snapshot garbage collection after Actor deletion is not implemented.

The stored snapshot already has a unique location. What is missing is a public resource that callers can safely find, use, and delete.

Goals

• Represent every successful external suspension snapshot as an ActorSnapshot.
• Keep multiple snapshots of one Actor independently addressable.
• Create an independent Actor from either a full or data snapshot.
• Keep physical storage locations private to Substrate.
• Prevent snapshot deletion from breaking an Actor.
• Keep the existing suspend and create operations while replacing the public snapshot field.

Non-goals

• Defining automatic snapshot GC or retention policy in lifecycle RPCs.
• In-place rollback of an existing Actor.
• Snapshotting an Actor while it continues running.
• Cloning node-local pause snapshots.
• Cross-atespace snapshot sharing or Actor creation in the first version.
• Snapshotting application history or state owned by external services.

Proposed API

Names and field layout are illustrative. The important parts are when snapshots are created, what refers to them, and what happens after a failed request.

ActorSnapshot

enum SnapshotScope {
  SNAPSHOT_SCOPE_UNSPECIFIED = 0;
  SNAPSHOT_SCOPE_FULL = 1;
  SNAPSHOT_SCOPE_DATA = 2;
}

message ActorSnapshot {
  ResourceMetadata metadata = 1;

  // Keep the source UID because an Actor name may be reused after deletion.
  ObjectRef source_actor = 2;
  string source_actor_uid = 3;

  // The Actor version accepted before the suspension transition began.
  int64 source_actor_version = 4;

  // Identity of the exact launch definition used by the source Actor.
  string actor_template_namespace = 5;
  string actor_template_name = 6;
  string actor_template_uid = 7;

  SnapshotScope scope = 8;
}

ActorSnapshots do not have human-chosen names. Substrate assigns an opaque ID, such as a UUID, and places it in metadata.name so existing ObjectRef APIs can address the resource. metadata.uid continues to distinguish a resource if an ID is ever reused.

ActorSnapshot is immutable after creation. Substrate retains the private storage reference, compatibility metadata, and integrity data needed to restore it. The object-storage URI is not part of the public resource.

The ActorTemplate identity records provenance. A FULL snapshot is coupled to the exact workload and sandbox state that produced it, so Substrate must retain enough immutable launch metadata to validate and reconstruct that state. A DATA snapshot contains only supported durable-volume contents and may initialize a different compatible ActorTemplate.

Actor reference

An Actor should refer to the first-class resource instead of exposing storage details as its durable identity:

message Actor {
  // Existing fields omitted.
  reserved 10;
  reserved "latest_snapshot_info";
  ObjectRef latest_snapshot = 13;
}

Remove latest_snapshot_info; callers should not receive object-storage locations. Its field number and name remain reserved so they cannot be reused accidentally. Substrate must keep a snapshot while an Actor points to it through latest_snapshot.

SuspendActor always returns a snapshot

message SuspendActorRequest {
  ObjectRef actor = 1;

  // Optional optimistic-concurrency preconditions. Exact-state callers
  // should provide both fields.
  string expected_actor_uid = 2;
  int64 expected_actor_version = 3;
}

message SuspendActorResponse {
  Actor actor = 1;
  ActorSnapshot snapshot = 2;
}

Every successful external suspension returns an ActorSnapshot; callers do not opt into resource creation or retention through this RPC.

Saving a snapshot touches both object storage and Substrate metadata, so it cannot be one storage transaction. Callers should still see one consistent result:

• Substrate stores the snapshot bytes and manifest before exposing the resource;
• Substrate publishes the Actor update and ActorSnapshot together in its metadata;
• success means the suspended Actor already points to the returned snapshot;
• a failed upload may leave unused bytes for later cleanup, but callers never see a usable resource for an incomplete snapshot; and
• retrying continues or recovers the same suspension operation.

Substrate assigns an opaque snapshot ID when suspension begins and stores it in Actor.in_progress_snapshot. A retry continues that operation. After success, Actor.latest_snapshot exposes the same ID, so a caller can recover even if the response was lost.

Suggested state behavior:

Actor state	Result
RUNNING	Create an external snapshot and ActorSnapshot, then suspend the Actor and reference it.
SUSPENDED with latest_snapshot	Return the Actor and its existing latest snapshot without writing another snapshot.
Newly created SUSPENDED Actor with no Actor snapshot	FAILED_PRECONDITION; its golden snapshot belongs to the ActorTemplate, not the Actor.
PAUSED	FAILED_PRECONDITION; node-local state is not a durable ActorSnapshot.
Other transitional or failed states	Preserve existing SuspendActor validation.

Both scopes can initialize an Actor using the existing restore semantics. FULL restores process memory, the rootfs delta, and durable volumes. DATA starts the target ActorTemplate's workload process fresh and restores only supported durable volumes.

Get, list, and delete

rpc GetActorSnapshot(GetActorSnapshotRequest) returns (ActorSnapshot) {}
rpc ListActorSnapshots(ListActorSnapshotsRequest)
    returns (ListActorSnapshotsResponse) {}
rpc DeleteActorSnapshot(DeleteActorSnapshotRequest)
    returns (ActorSnapshot) {}

message GetActorSnapshotRequest {
  ObjectRef snapshot = 1;
}

message ListActorSnapshotsRequest {
  string atespace = 1;
  int32 page_size = 2;
  string page_token = 3;
}

message ListActorSnapshotsResponse {
  repeated ActorSnapshot snapshots = 1;
  string next_page_token = 2;
}

message DeleteActorSnapshotRequest {
  ObjectRef snapshot = 1;
  DeleteOptions options = 2;
}

List pagination follows the existing Actor and Atespace conventions. Source-Actor filtering is not required in the first version.

Deletion returns FAILED_PRECONDITION while an Actor or ActorTemplate points to the snapshot. After the final reference is gone, deleting the resource also allows Substrate to delete the stored data.

Create an Actor from a snapshot

Extend the existing creation RPC rather than add a parallel CloneActor lifecycle:

message CreateActorRequest {
  // Desired target Actor, including the ActorTemplate to instantiate.
  Actor actor = 1;

  // Optional initialization source. When omitted, creation continues to use
  // the ActorTemplate golden snapshot.
  ObjectRef source_snapshot = 2;
}

When source_snapshot is present:

• the target Actor must be in the same atespace as the snapshot;
• the target Actor still names the ActorTemplate to instantiate;
• for FULL, the target template and sandbox must be compatible with the exact process and filesystem state captured by the snapshot;
• for DATA, the target template may differ from the source template and receives the snapshot's compatible durable-volume contents;
• the target is created SUSPENDED, as it is today; and
• duplicate target Actor names return ALREADY_EXISTS, following existing CreateActor semantics.

Creating an Actor and deleting its source snapshot may happen at the same time. Substrate must produce one of two results: the Actor is created and keeps the snapshot alive, or deletion wins and CreateActor returns NOT_FOUND.

An explicit CloneActor RPC would add another Actor-creation path without adding capability. It should be introduced only if CreateActor cannot represent the required source semantics cleanly.

References and deletion

An Actor's latest_snapshot and an ActorTemplate's golden snapshot keep that snapshot alive.

• Deleting an ActorSnapshot fails while any Actor refers to it as latest_snapshot.
• Deleting an ActorSnapshot also fails while an ActorTemplate uses it as its golden snapshot.
• Deleting the source Actor does not automatically delete older snapshot resources.
• After an Actor produces a newer snapshot, its previous snapshot may be deleted if nothing else refers to it.
• Substrate may delete the stored bytes only after the snapshot resource and every Actor or ActorTemplate reference are gone.

Substrate may implement this with reference counts, copies, or copy-on-write storage. Callers only depend on the behavior above.

Separate snapshot GC policy

Automatic cleanup should be designed separately from SuspendActor, CreateActor, and the snapshot CRUD methods.

A separate policy may delete snapshots that no Actor or ActorTemplate uses after a grace period. Its scope, default age, deletion protection, and limits need their own design. SuspendActorRequest should not contain GC or retention settings.

Consistency boundary

An ActorSnapshot guarantees a consistent Substrate Actor image. It does not include a caller's application history or state owned by external services.

A higher-level control plane must stop new work, wait for current work to finish, and save any history it owns before suspension. It may associate that history position with the returned ActorSnapshot, but Substrate does not manage that combined checkpoint.

Compatibility

• Existing SuspendActor requests keep the same operation but receive an ActorSnapshot in the response.
• Existing CreateActor requests omit source_snapshot and continue from the ActorTemplate golden snapshot.
• Apart from removing latest_snapshot_info, the new request fields and RPCs are additive.
• Actor.latest_snapshot_info is removed and its protobuf name and field number are reserved. Clients must regenerate against the updated API.

Expected errors

Condition	gRPC status
Actor or snapshot does not exist	NOT_FOUND
Target Actor name already exists	ALREADY_EXISTS
Expected Actor UID or version is stale	ABORTED
Actor is paused or has no Actor-owned external snapshot	FAILED_PRECONDITION
Snapshot is referenced during deletion	FAILED_PRECONDITION
Snapshot and target are in different atespaces	INVALID_ARGUMENT
Snapshot is incomplete or incompatible with the target runtime	FAILED_PRECONDITION
Snapshot manifest or data is missing or corrupt	DATA_LOSS

Acceptance criteria

• Every successful external suspension returns a first-class ActorSnapshot.
• The suspended Actor references the returned snapshot as latest_snapshot.
• Retrying an interrupted suspension recovers the same in-progress operation.
• UID and version preconditions prevent capturing a replaced or changed Actor.
• Multiple snapshots of one Actor remain independently addressable.
• Get and paginated list enforce atespace authorization.
• An Actor created from a full snapshot resumes with captured process memory, rootfs delta, and durable volumes.
• An Actor created from a data snapshot may use a different compatible ActorTemplate and starts its process fresh with supported durable volumes restored.
• The source and created Actor can advance independently.
• Deleting the source Actor does not delete an independently addressable snapshot.
• Deleting a referenced snapshot is rejected.
• Concurrent snapshot deletion and Actor creation cannot produce a dangling Actor reference.
• Paused Actors and cross-atespace creation are rejected.
• Existing suspend, resume, and create workflows continue after clients regenerate against the updated API.
• Automatic GC policy is not coupled to lifecycle RPC request fields.

Open decisions

1. Can CreateActorRequest.source_snapshot be added cleanly, or is an explicit CloneActor RPC preferable in the implementation?
2. What compatibility rules apply between each snapshot scope and the target ActorTemplate?
3. What independent policy should govern dangling-snapshot garbage collection?

References

• ateapi.proto
• ActorTemplate snapshot configuration
• Substrate architecture