Repository navigation
Description
Summary
The nightly full run of dev at c473e70d2 failed in Package nvx crate: the build of the packed nvx crate with every feature panicked in its build script, because the release that nvx-rs/artifacts.json pins, v0.1.0-dev.4f9a3214f09b, declares microVM ABI 2 in its SOURCE-MANIFEST.json. #1189 raised the crate's microVM ABI to 3, and #1191 extended that version, but artifacts.json still pins the release that #1174 chose for time ABI 2.
Every bundled build of dev that downloads the pinned release fails the same way, so crates.io gets no new version of nvx after 0.1.0-alpha.9, and the nightly stays red, until a pull request repins a compatible release, as The nvx crate says.
Incident
-
Workflow: CI run 41583294959, attempt 1,
scheduleondev, atc473e70d2358dc5978565be001543ccdecd0fe33, the merge of openvmm: give each microVM virtio slot its own IOAPIC input #1191. Nightly failure report: ci: the nightly full run is failing #1234. -
Failed job:
Package nvx crate, stepPackage, check, and build the nvx crate, onubuntu-latestwith Rust 1.95.0. Sincenvx-rs/changed aftere92bfb576236, from which0.1.0-alpha.9was packed, the job packed0.1.0-alpha.10, and then built it with every feature, bundling the KVM release:warning: nvx@0.1.0-alpha.10: nvx: downloading https://github.com/microsoft/nvx/releases/download/v0.1.0-dev.4f9a3214f09b/nvx-0.1.0-linux-kvm.tar.gz thread 'main' (3132) panicked at build.rs:354:9: nvx bundled artifacts: .../nvx-0.1.0-linux-kvm/SOURCE-MANIFEST.json does not declare the compatible OpenVMM contracts (microvm_abi_version: expected 3, found 2) -
Downstream:
Publish nvx cratewas skipped. The run's other 35 jobs passed, and the rest of its skipped jobs run only for pushes, pull requests or merge groups. -
Not superseded: this is the newest full run.
devhas since taken a[skip ci]baseline commit and kernel: generate the configuration from a feature inventory #1200, at304f8e168, which leavenvx-rs/and the OpenVMM pin unchanged, so the next nightly fails the same way.
History
- This is the third stale pin of
bundled. Track aci_edge_sandboxes bugs: uncancellable async exec, stale bundled pin, lock-file leak, and signed CIDR prefixes #364 found the first after the move to thenvx-microvm-v2-control-v2control contract, and aci_edge_sandboxes: fix four bugs from the crate's bug hunt #367 repinned. nvx-rs: artifacts.json pins a release from before time ABI 2, so bundled builds and the nightlyPackage nvx cratefail #1173 found the second after time ABI: publish the host's clock correction with each time sample #1018 raised the time ABI and the restore packet, and nvx-rs: bundle a release that declares time ABI 2 #1174 repinnedv0.1.0-dev.4f9a3214f09b. - openvmm: drop access-platform and the console size in microVM ABI 3 #1189, merged on November 3 at 01:33 UTC as
630a7a5b9, raisedMICROVM_ABI_VERSIONfrom 2 to 3, andBOOT_CONTRACT_VERSIONfrom 3 to 4 with it, in628fb78cd. Its description expected this failure: a bundled build that downloads the pinned release "would fail untilartifacts.jsonpins a development release that declares ABI 3, asc4acffbffdid for time ABI 2. That follow-up belongs after the stack lands, whendevpublishes such a release." - openvmm: give each microVM virtio slot its own IOAPIC input #1191, merged at 04:20 UTC as
c473e70d2, extends ABI 3: each virtio slot gets an IOAPIC input of its own. Its landing notes ask to pin "a development release that declares microVM ABI 3 innvx-rs/artifacts.json, before the nightly bundled build runs". The nightly started at 05:20 UTC, before any pull request repinned it. - kernel: generate the configuration from a feature inventory #1200, merged at 05:39 UTC as
304f8e168, adds no version, and Support nested virtualization in NVX microVMs #256 is closed as not planned, so no further layer of ABI 3 is pending. - nvx-rs slimming 4.2.3: the old configs, the old launch code and
bundledare gone #1122 deletesbundled, its build script andartifacts.json, which ends this kind of failure, but none of its pull requests merges before the slimming's API-ready commit.
Root cause
For the five fields that the build script checks, the pinned release's SOURCE-MANIFEST.json declares:
{"microvm_abi_version": 2, "time_abi_version": 2, "restore_packet_version": 5, "control_session_protocol_version": 1, "control_contract_revision": "nvx-microvm-v2-control-v2"}At c473e70d2, contract::manifest_mismatches requires microvm_abi_version 3, so stage() in nvx-rs/build.rs refuses the release. No pull request job builds bundled against the pin: the --all-features builds of check-nvx-rs set NVX_BUNDLE=skip, and its bundling test stages a synthetic release from the commit's own SOURCE-MANIFEST.json, which #1189 moved to ABI 3. A pull request that changes a contract cannot repin either, since no release declares the new version before it merges, so only the nightly Package nvx crate job finds the stale pin.
Reproduction
On Windows, with Rust 1.95.0:
- The
SOURCE-MANIFEST.jsonof each package, read from the release assets themselves: the pinned release'slinux-kvmpackage declares ABI 2, while those ofv0.1.0-dev.630a7a5b9025,v0.1.0-dev.c473e70d2358andv0.1.0-dev.304f8e1686b4, and thelinux-mshvpackage of the last, declare ABI 3 and the four other values that the crate expects. cargo check --locked --features bundledat304f8e168, whosenvx-rs/is that ofc473e70d2, withoutNVX_BUNDLE*: the build script downloads the pinnedwindows-whppackage and panics atbuild.rs:354:9with the same mismatch. Withartifacts.jsonpinningv0.1.0-dev.304f8e1686b4, in a fresh target directory, it downloads that release's package, which matches its digest, and passes.
Fix
Repin nvx-rs/artifacts.json to v0.1.0-dev.304f8e1686b4, the release of 304f8e168, the head of dev. Its nvx-rs/ and OpenVMM pin, 35e76265f, are those of the nightly's commit, and its SOURCE-MANIFEST.json declares every contract version of nvx::contract. The GitHub release API gives these digests:
| Package | SHA-256 |
|---|---|
linux-kvm |
4aa4a4fa786ca5f952bc5c8bdd6de3539dfe9c47f233d62a9786cf4ea07998a8 |
linux-mshv |
6c075d05cc5e9e07bf3c632192214f1bf3e4e22687b56e7fab1dc5d227dae02a |
windows-whp |
6e17261f3e8808d846d7599c979922d0eeaa72f222ca83e57103f8840c7a9175 |
Not v0.1.0-dev.630a7a5b9025, the release of #1189: it declares ABI 3, but its OpenVMM, b9b1f8f6d, predates the IOAPIC inputs that #1191 added to ABI 3, and the build script compares only the version numbers.
Then dispatch CI on dev, or wait for the next nightly, so that Package nvx crate and Publish nvx crate publish 0.1.0-alpha.10, and a passing nightly closes #1234.
Activity
nvx-rs: bundle a release that declares microVM ABI 3