Do ActorTemplates still need mandatory golden snapshots?

[EItanya]

Summary

Every ActorTemplate currently creates a hidden “golden Actor,” boots it, waits for initialization, suspends it, and stores its snapshot URI in the ActorTemplate status. By default, new Actors start from that golden snapshot.

This gives new Actors a warm starting point, but it also creates a second, special snapshot lifecycle alongside normal Actors. With first-class ActorSnapshots, is that special path still worth its complexity?

Would it be simpler to use this model instead?

1. an ActorTemplate becomes ready after its configuration is accepted;
2. an Actor with no snapshot boots directly from its ActorTemplate;
3. ResumeActor continues to wait for the Actor's configured readyz probes, as it does today;
4. SuspendActor explicitly creates the Actor's first ActorSnapshot; and
5. callers that need a prewarmed starting point create an Actor, wait for readiness, suspend it, and pass that snapshot to CreateActor.

This would give Substrate one Actor and snapshot lifecycle. Prewarming would remain possible, but callers would choose when they need it.

Current complexity

The ActorTemplate controller currently:

1. creates the reserved golden-Actor atespace;
2. creates a hidden Actor;
3. resumes it from scratch;
4. waits for readyz, or sleeps for 20 seconds when a container lacks a probe;
5. suspends the Actor;
6. reads the private snapshot URI from the Actor;
7. writes that URI into ActorTemplate.status.goldenSnapshot; and
8. marks the ActorTemplate ready.

This introduces:

• special golden Actor phases and status fields;
• a reserved atespace and hidden Actor lifecycle;
• a fixed 20-second initialization guess when readyz is missing;
• another owner for snapshot storage and garbage collection;
• failure recovery that differs from ordinary Actor recovery; and
• a raw snapshot URI in the ActorTemplate API.

The controller already contains TODOs for leaked workers and recovery when golden suspension succeeds but reconciliation fails.

Alternative to consider

ActorTemplate readiness

ActorTemplate readiness could mean that Substrate accepted the immutable template and can use it to boot an Actor, without creating or snapshotting a hidden Actor.

That would allow us to remove the golden-specific status fields and phases:

• GoldenActorID;
• TakeGoldenSnapshotAt;
• GoldenSnapshot;
• ResumeGoldenActor; and
• WaitGoldenActor.

First Actor start

CreateActor still creates a suspended Actor. When that Actor has no latest_snapshot, ResumeActor boots it from the selected ActorTemplate.

This readiness behavior already exists. Both gVisor RunWorkload and RestoreWorkload call readyz.WaitAll, and the call path back to ResumeActor blocks until every configured probe returns HTTP 200. Containers without a probe are skipped.

Could we rely on that existing signal instead? Templates without readyz would receive no application-readiness guarantee, but Substrate would no longer replace a missing signal with a fixed sleep.

Explicit snapshots

SuspendActor creates the Actor's first-class ActorSnapshot and sets Actor.latest_snapshot. Later resumes restore that snapshot using its declared FULL or DATA semantics.

This is the same lifecycle used for every Actor:

CreateActor
    -> ResumeActor
    -> wait for readyz
    -> SuspendActor
    -> ActorSnapshot

Optional prewarming

If mandatory golden snapshots were removed, a caller that needs faster startup could still prewarm through the normal APIs:

CreateActor(template)
    -> ResumeActor
    -> wait for readyz
    -> SuspendActor => prewarmed ActorSnapshot

CreateActor(template, source_snapshot = prewarmed snapshot)

That snapshot would be visible, would follow normal ownership and deletion rules, and could be refreshed like any other ActorSnapshot. Would this make the hidden golden snapshot type unnecessary?

A higher-level API later

If prewarming becomes a common managed workflow, it could return later as a higher-level API rather than remain implicit in every core ActorTemplate. A provisional GoldenActorTemplate resource could compose an ordinary ActorTemplate with the normal Actor lifecycle and add features such as refresh policy, health status, rollout, retention, and prewarming metrics.

That resource would not require a second snapshot mechanism. It would create an Actor, wait for readiness, suspend it to an ActorSnapshot, and publish that snapshot as a managed warm starting point. Core ActorTemplate, CreateActor, ResumeActor, and SuspendActor semantics would remain simple.

This is a possible extension, not part of the change being considered here. It should be introduced only when concrete prewarming requirements justify it.

Tradeoffs to evaluate

The first start from an ActorTemplate would become a cold start. Workloads with expensive initialization might take longer to become ready.

We should compare this cost with the work currently performed for every template. If prewarming is valuable, could it be an explicit higher-level policy built from normal Actors and ActorSnapshots?

Boot and readiness failures would also move from ActorTemplate reconciliation to the first Actor that uses the template. Systems that want earlier validation could explicitly create and resume a smoke-test Actor without making that Actor's snapshot part of every template's lifecycle.

readyz does not provide the speed benefit of a golden snapshot. It provides the correctness signal that the current 20-second delay is trying to approximate.

Relationship to first-class ActorSnapshots

This alternative becomes possible with the snapshot lifecycle discussed in agent-substrate/substrate#529:

• every durable suspension returns an ActorSnapshot;
• Actors refer to snapshots by opaque ID;
• CreateActor can use an ActorSnapshot as its initial state; and
• snapshot garbage collection is governed separately from Actor lifecycle RPCs.

Decision criteria

Before changing the design, we should establish:

• cold-start and prewarmed-start latency for representative workloads;
• how often ActorTemplates are created but never instantiated;
• whether boot validation at template reconciliation catches failures that readyz on the first Actor would catch too late;
• whether explicit prewarming can meet the same operational needs;
• whether workloads without readyz need another readiness signal; and
• how much controller, recovery, storage, and garbage-collection complexity the golden path adds.

Open questions

1. Is mandatory prewarming a core Substrate responsibility, or should callers opt into it?
2. Should readyz be required for every workload container, or remain optional with weaker readiness semantics?
3. Does removing golden snapshots make ResumeActorRequest.boot unnecessary?
4. Should optional prewarming remain entirely outside Substrate, or eventually become a policy built on ordinary ActorSnapshots?
5. If managed prewarming becomes necessary, should it be represented by a higher-level resource such as GoldenActorTemplate?

References

• ActorTemplate controller
• ActorTemplate API
• Resume workflow
• First-class ActorSnapshot issue