Counter demo: in-memory state not preserved across suspend/resume (gVisor, kind), despite `onPause: Full`

[mcftira]

Environment: kind v0.30.0 (k8s v1.36.1) on WSL2/Ubuntu 24.04, substrate@main, deployed via hack/create-kind-cluster.sh + hack/install-ate-kind.sh.

Repro: deploy counter demo, create atespace + actor, POST 3x (memory count 3, file count 3), kubectl ate suspend, POST again.

Expected: per the API docs, SnapshotScopeFull "captures process memory plus the entire filesystem delta", so the resumed actor should respond preserved memory count: 4.

Actual: memory count resets to 1 while the DurableDir file counter correctly continues to 4. Reproducible across cycles (3 -> suspend -> 1 twice; file counter 3 -> 4, 6 -> 7).

Logs: atelet shows the Checkpoint RPC succeed with pages.img.zstd/pages_meta.img.zstd uploaded, and the Restore RPC succeed (~800ms), with no warnings or errors. The memory snapshot is captured and uploaded, but appears not to be restored (or silently falls back to a fresh start).

Is memory restore expected to work on the kind/gVisor path, or is this a known limitation? Happy to collect further diagnostics.

[BenTheElder]

Running in kind is expected to be the same as in any other cluster for this purpose. We run e2e tests with kind for now, which include the counter demo, but are not super exhaustive yet.

This definitely sounds like a bug.

[amolr]

I can pick this up

[mcftira]

Thanks for confirming. I'll dig into the restore path and try to find where the memory portion gets dropped on resume. If I can pin it down, I'd be happy to take a shot at the fix - just say the word.

[mjlshen]

I think this is mostly an issue with the demo specification defaulting to onCommit: Data - which according to the docs only persists DurableDir data, not process memory. I think it's natural to want to demo suspend/resume with process memory restored as well, so it's just a matter of updating the demo to default to onCommit: Full :D

[mcftira]

Agreed, that matches what I found - with onCommit: Data the suspend checkpoint takes the runsc fscheckpoint branch and memory is never captured. I'll send a PR switching the demo template to onCommit: Full, verified end-to-end on kind (memory counter continuing across suspend/resume).

Two smaller follow-ups for later, if you're interested: recording the capture scope in the snapshot manifest (restore currently re-derives scope from the current template, so capture and restore scope can silently disagree), and a loud log when cold-booting a snapshot that contains memory images.

[mjlshen]

Ah I think I beat you to it 😅 it's a dupe of #520

Two smaller follow-ups for later, if you're interested: recording the capture scope in the snapshot manifest (restore currently re-derives scope from the current template, so capture and restore scope can silently disagree), and a loud log when cold-booting a snapshot that contains memory images.

I'm understanding you correctly, you're pointing out that a snapshot does not record as Data of Full in sandboxAssetsRecord, so that during restore time, restoring a Full snapshot in Data mode (or vice versa) is confusing and unexpected. I'm not sure whether there's a good use case for allowing this.

I think this is being discussed more in #16

[mcftira]

Exactly. And I don't see a legit use case for the mismatch either: Full-as-Data silently drops memory (this issue), Data-as-Full looks for memory that was never captured. Recording the capture scope in sandboxAssetsRecord and validating it at restore time would turn a silent footgun into an explicit error. Happy to write it up as a proper proposal issue if you think it's worth pursuing.