Skip to content

ICMP: enforce ICMP egress rules end to end and make ICMP usable in the guest (NVX 6b485c82 / OpenVMM c32c57d1, Windows WHP) #1250

Closed

Description

Summary

A downstream SDK consumer of aci_edge_sandboxes needs usable ICMP inside the guest and enforced ICMP policy rules (allow and deny). With the runtime bundle below, neither works. We'd like to know what NVX/OpenVMM support at this pin and what guest configuration is intended, so that the ACI layer can be aligned.

Related: #277 (ICMP-only egress rules, closed 2026-10-09, before this pin).

Versions

Component Version
Rust crates (crates.io) aci_edge_sandboxes 0.1.1, aci_edge_sandboxes_model 0.1.1, nvx =0.1.0-alpha.6
Runtime bundle Azure/aci-edge-sandbox v0.1.0-build188642429, asset aci-edge-sandboxes-0.1.0-build188642429.zip, SHA-256 cad4db7c574a73aaa1c5a07e7a740cd2b01d5b83c62c81b2196d63bad9ecc126 (the same bundle as crate 0.1.0)
NVX in the bundle 6b485c82c9fdf4f592e1221efc7d6d41ab9bc66c
OpenVMM in the bundle c32c57d129a36fc8c82308d6659f8d7ab5955474 (OpenVMM log: version="0.3.0+gc32c57d12")
Host Windows 11 x64, Windows Hypervisor Platform
Guest image mcr.microsoft.com/azurelinux/base/python:3

The bundle's agent library (aci_edge_agent.dll) is ACI's sandbox agent. It translates the crate's network policy into NVX/OpenVMM options in an ACI layer on top of nvx, so some behavior below comes from that ACI layer rather than from NVX. Each section says which.

Observed

1. ICMP policy rules are rejected (ACI layer, not NVX)

An egress rule with "protocol": "icmp" fails at provision:

Error { code: PolicyValidation, message: "OpenVMM cannot select ICMP by itself" }

This message comes from ACI's policy translation, not from NVX. As far as we can tell from source at this pin, NVX's lowering (nvx-rs/src/host/network/lowering.rs) emits CIDR:icmp rules, and OpenVMM's egress rule parser (vm/devices/net/net_backend_resources/src/egress.rs) accepts them. ACI will change its layer; we have not verified end-to-end enforcement by NVX/OpenVMM.

2. ICMP sockets fail in the guest even with egress allowed

With {"egress": {"default": "allow"}, "ingress": {"default": "deny"}}, the workload runs as uid 0 without CAP_NET_RAW, and unprivileged ping sockets are disabled:

uid 0
CapEff:     00000000200004fb
CapBnd:     00000000200004fb
NoNewPrivs: 1
ping_group_range 1  0
dgram errno 13 Permission denied     # socket(AF_INET, SOCK_DGRAM, IPPROTO_ICMP)
raw errno 1 Operation not permitted  # socket(AF_INET, SOCK_RAW, IPPROTO_ICMP)

We have not determined whether this capability set and ping_group_range come from NVX's guest init or from ACI's guest configuration. Because no ICMP socket could be opened, we couldn't test whether OpenVMM relays ICMP.

Reproducer

A minimal consumer of the published crate. With the agent-bundled feature, the build downloads and verifies the runtime bundle above.

Cargo.toml:

[package]
name = "nvx-net-repro"
version = "0.1.0"
edition = "2024"

[dependencies]
aci_edge_sandboxes = { version = "=0.1.1", default-features = false, features = ["agent-bundled"] }
serde_json = "1"

src/main.rs:

//! Usage: nvx-net-repro <state-root> <policy.json> (--python <script.py> | <guest command...>)
//! Set REPRO_SNAPSHOTS=1 to enable platform snapshots (templates and clones).
use std::sync::Arc;

use aci_edge_sandboxes::agent::{AgentBackend, BundledAgent, PlatformSnapshots, SandboxDefaults};
use aci_edge_sandboxes::{
    AciEdgeSandbox, ExecRequest, ImageSource, NetworkPolicy, ProvisionRequest, SandboxSpec,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut args = std::env::args().skip(1);
    let state = args.next().expect("state root");
    let policy: NetworkPolicy =
        serde_json::from_str(&std::fs::read_to_string(args.next().expect("policy.json"))?)?;
    let rest: Vec<String> = args.collect();
    let bundle = BundledAgent::discover()?;
    let mut setup = bundle.setup(state).with_defaults(SandboxDefaults::default().with_image(
        ImageSource::Reference("mcr.microsoft.com/azurelinux/base/python:3".into()),
    ));
    if std::env::var_os("REPRO_SNAPSHOTS").is_some() {
        setup = setup.with_platform_snapshots(PlatformSnapshots::new(2, 512));
    }
    let backend = Arc::new(AgentBackend::new(bundle.config(setup))?);
    let client = AciEdgeSandbox::from_shared(backend.clone());
    let request = ProvisionRequest::new().with_network(policy);
    let id = client.provision_with(&request, &SandboxSpec::new())?.sandbox_id;
    let started = client.start(&id)?;
    eprintln!("bootMode: {:?}", started.boot_mode());
    let exec = match rest.as_slice() {
        [flag, script] if flag == "--python" => {
            ExecRequest::argv(["python3".to_owned(), "-c".to_owned(), std::fs::read_to_string(script)?])
        }
        _ => ExecRequest::command_line(rest.join(" ")),
    };
    let output = client.exec(&id, &exec)?.wait_with_output()?;
    println!("outcome: {}", output.outcome);
    print!("{}{}", String::from_utf8_lossy(&output.stdout), String::from_utf8_lossy(&output.stderr));
    client.stop(&id)?;
    client.deprovision(&id)?;
    Ok(())
}

Run it with cargo run --release -- <state-root> <policy.json> --python <probe.py>, using a short state root such as C:\nvxr.

Policies and probe:

// icmp-rule.json: expect a rejection today
{ "egress": { "default": "deny", "allow": [ { "to": [ { "cidr": "1.1.1.1/32" } ], "ports": [ { "protocol": "icmp" } ] } ] }, "ingress": { "default": "deny" } }
// egress-allow.json
{ "egress": { "default": "allow" }, "ingress": { "default": "deny" } }
# icmp_probe.py
import socket, os
print("uid", os.getuid())
for line in open("/proc/self/status"):
    if line.startswith(("CapEff", "CapBnd", "NoNewPrivs")): print(line.strip())
print("ping_group_range", open("/proc/sys/net/ipv4/ping_group_range").read().strip())
for name, kind in (("dgram", socket.SOCK_DGRAM), ("raw", socket.SOCK_RAW)):
    try: socket.socket(socket.AF_INET, kind, socket.IPPROTO_ICMP).close(); print(name, "ok")
    except OSError as e: print(name, "errno", e.errno, e.strerror)

Expected (downstream requirement)

  • ICMP egress rules, ICMP-only as in Support additional egress protocol combinations #277, are enforced: allowed destinations are reachable by ICMP, and others are blocked, including by deny rules.
  • A workload can use ICMP, for example echo, without the consumer depending on a particular ping implementation. We don't assume this requires unrestricted raw sockets.

Questions

  1. At NVX 6b485c82 / OpenVMM c32c57d1 on Windows WHP, is ICMP (and ICMPv6) policy enforcement supported end to end, including relaying echo to allowed destinations? Any limits, such as echo only?
  2. What guest configuration does NVX intend for usable ICMP: for example, net.ipv4.ping_group_range covering the workload's group for SOCK_DGRAM ICMP, or an opt-in capability? Is that controlled by NVX's guest init, or left to the integrator?
  3. Any interaction with published listeners or other policies that would restrict this?

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions