Skip to content

Restore: read the state as fast as the disk, or only what the guest touches #276

Open

Description

@aledbf

A checkpoint restored to the first login is ~160 ms once its state is in memory. From the runner's gp3 volume it is ~840 ms, of which ~680 is reading the ~163 MiB state (#275, labs 44542990433 and 44542999076). QEMU reads it ~250 ms slower than one sequential pass of the same file does. spin's hosts read from local NVMe, so the disk share there is smaller, but neither the read nor the copy into guest memory is free. Experiments, each a row of boot:restore-first, measured on the bench once it has local NVMe (spin-stack/ci#21):

  1. Read the state the way the disk can. posix_fadvise(POSIX_FADV_SEQUENTIAL) and a large readahead on the file restore hands QEMU, or QEMU's file migration reading in larger blocks. Up to ~250 ms on gp3; on NVMe, whatever is left of it. Fingerprint unchanged.
  2. Read only what the guest touches (restore on demand). Guest RAM backed by a file (memory-backend-file, share=on), saved with x-ignore-shared: the checkpoint holds only device state, and the restore maps the RAM file and faults pages in as the guest touches them. The first login touches a fraction of 163 MiB. To find out:
    • where the RAM file lives while the workspace is suspended (tmpfs keeps host memory, a disk takes the guest's writes);
    • how it composes with virtio-mem and the balloon's free page reporting;
    • it is a change to Spec.Contract, so the fingerprint moves (CLAUDE.md: allowed when it serves the goal; the release notes say so).
  3. A smaller state. Most of 163 MiB in a 2 GiB guest is likely its page cache. Dropping clean caches, or reporting free pages, right before the save shrinks what is read. The save is spin's to trigger, so this is a contract with spin: what the guest is asked to do before a checkpoint.

Already rejected: mapped-ram with multifd (#246, 385 vs 386 ms, measured with the disk share unseparated; worth one row again once 1 says what the read costs).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions