Skip to content

feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in) - #144

Draft
luthermonson wants to merge 1 commit into
mainfrom
feat/windows-l2bridge-egress
Draft

feat(networking): Windows L2Bridge egress with router-safe VFP ACLs (opt-in)#144
luthermonson wants to merge 1 commit into
mainfrom
feat/windows-l2bridge-egress

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Problem

Windows container egress cannot be software-filtered on the current HNS NAT
network. VFP (the Virtual Filter Platform) does not engage on a NAT network, so
the per-endpoint Switch ACLs ephemerd already applies are inert — enforcement
there rests entirely on the Hyper-V firewall. We need an enforcement point where
software ACLs actually bite.

Fix (opt-in, NAT stays default)

Add an L2Bridge networking path, gated behind a per-pool flag
(network.l2bridge_egress). On L2Bridge the container gets a VFP-managed
vSwitch port
, so per-endpoint ACLs enforce. This was proven on metal
(node mfl-win-amd64-1, Server 2025): with the correct ACL model, a
Hyper-V-isolated container reached 1.1.1.1 / public HTTPS while
container→Grafana/Incus/Proxmox (RFC1918) was blocked.

Merging this flips nothing — NAT remains the default and the NAT init/setup/
ACL path is untouched. A pool only reaches L2Bridge by setting
l2bridge_egress = true.

The router-safe ACL model (the load-bearing change)

VFP is default-DENY the moment any ACL is present — a pure-Block set (today's
buildEgressBlockPolicies) would black-hole the internet too. buildL2BridgeEgressACLPolicies
(a pure, unit-tested function) emits this precedence ladder as RuleType=Switch
VFP ACLs (lower Priority = higher precedence, first match wins):

Priority Action Dir Remote / Ports Proto Purpose
90 Allow Out RemotePorts 67,68 UDP (17) DHCP lease/renew survives the block
90 Allow In LocalPorts 67,68 UDP (17) DHCP reply
95 Allow Out extra-allowed CIDR 256 operator carve-outs (future use; default none)
100 Block Out 10/8, 172.16/12, 192.168/16, 169.254/16 256 RFC1918 + link-local, whole supernets
65500 Allow Out 0.0.0.0/0 256 internet
65500 Allow In 0.0.0.0/0 256 TCP return traffic (SYN-ACK)

This matches the Linux end-state (firewall_linux.go): block all of RFC1918
including the LAN router, permit only the internet.

  • Whole supernets, no carve-out. No gateway/own-subnet exclusion (unlike NAT).
    On L2Bridge the container is a LAN peer, so carving out the subnet would expose
    the same-subnet management plane and the router. This is the fix for the
    same-subnet management-plane bug.
  • Both allow-any rules are mandatory. Without the low-precedence pair the port
    default-denies everything; the inbound allow is the non-obvious one — drop it
    and every outbound TCP connect fails when the SYN-ACK returns.
  • DNS → public resolvers. The endpoint's Dns.ServerList is set to a
    configurable public resolver (default 1.1.1.1, 8.8.8.8), so DNS rides the
    0.0.0.0/0 allow and the container never needs the LAN router. No allow to
    the gateway/router.
  • No container-to-container allow (unlike Linux) — on L2Bridge that is LAN
    access, which we block.

Fail-closed, no post-lease window

The ladder is static — it does not depend on the leased gateway or DNS (DNS is
public, the router gets no allow) — so the ACLs are applied at endpoint creation,
before the container starts
. Any ApplyPolicy error tears down the endpoint and
refuses the job.

Network shape

L2Bridge network is hcn.L2Bridge bound to the host NIC via a hcn.NetAdapterName
network policy (recipe §1). IPAM = DHCP: the network declares no static subnet/
routes and the endpoint carries no IpConfigurations, so the container leases its
IP, gateway, and default route from the LAN DHCP server. Network name is
ephemerd-l2bridge (distinct from the NAT ephemerd).

Config surface (what mayfly must render)

New [network] keys (per-pool → rendered into the per-VM ephemerd config.toml,
same flow as the existing [vm.linux] toggle and windows_runner sizing):

  • l2bridge_egress (bool, default false) — the opt-in.
  • host_nic (string) — required when l2bridge_egress = true; the host adapter
    name to bridge onto. No default (host-specific; do not hardcode "Ethernet 2",
    the spike's hot-added test NIC). Missing → init fails closed.
  • public_dns ([]string, default ["1.1.1.1","8.8.8.8"]).
  • extra_allowed_destinations ([]string, default []) — future use.

Wired through networking.Config (L2BridgeEgress, HostNIC, PublicDNS,
ExtraAllowedCIDRs) at both networking.New call sites in cmd/ephemerd/main.go.
Documented in config.example.toml. mayfly rendering is a follow-up (not in
this PR): add the four keys to the per-pool Windows config schema and render them
into [network]; host_nic should be a required field when the pool opts in.

Honest caveats

  • DHCP-on-L2Bridge was NOT validated on metal. The spike proved L2Bridge + VFP
    enforcement with a static IP. DHCP is the chosen v1 IPAM and will be validated
    at the gated cutover. If DHCP-on-L2Bridge misbehaves there, static-pool IPAM is
    the proven fallback
    (recipe §6 recommends static-first).
  • The 3-tier carve-out (allow-/32 above block) was not run end-to-end on metal
    the 2-tier (high-prio blocks over low-prio allow-any) was. The default posture here
    has no carve-outs (empty extra_allowed_destinations), so the default set is the
    proven shape plus the DHCP allow.
  • Anti-spoof: documented gap. hcsshim v0.14.0-rc.1's hcn does not expose a
    ready source-IP/MAC anti-spoof endpoint/port policy in a form I could apply without
    guessing the schema, so it is not implemented here rather than faked. On L2Bridge
    the container is a routable LAN peer that could source-spoof within the L2 domain;
    egress enforcement rests on the VFP ACLs. Follow-up: evaluate an HNS L2Bridge
    MAC/IP anti-spoof port policy.

Deviations from the recipe

  • Did not repurpose buildEgressBlockPolicies for NAT. Rewriting it into the
    ladder (or applying the ladder on NAT) would blackhole the NAT gateway 10.88.0.1
    (inside the whole-10/8 block, no carve-out) if VFP ever engaged. Kept the NAT
    block-only builder and path exactly as-is; the ladder is a new function on
    the L2Bridge path only. This better satisfies "NAT stays intact".
  • DHCP shape (no static IPAM, no endpoint IP) instead of the recipe's static
    IpConfigurations — per the user's DHCP v1 choice; see caveat above.

Tests

pkg/networking/network_windows_test.go (pure, no HCN calls):

  • TestL2BridgeEgressACLPolicies_LadderShape — both allow-any (Out+In), both DHCP
    allows, all four supernets blocked; exact protocols/priorities/directions.
  • TestL2BridgeEgressACLPolicies_NoGatewayOrSubnetCarveOut — no allow to any RFC1918
    address, blocks are plain CIDRs (no exclusion ranges leaked in).
  • TestL2BridgeEgressACLPolicies_Precedence — DHCP/extra < block < allow-any.
  • TestL2BridgeEgressACLPolicies_ExtraAllowed — carve-outs emitted Out, above block.
  • Existing TestBuildEgressBlockPolicies (NAT) unchanged and passing.

go build ./pkg/networking/... ./pkg/config/..., go vet ./pkg/networking/..., and
go test ./pkg/networking/... all clean (native GOOS=windows). (go build ./...
hits a pre-existing missing-embed-artifact error in pkg/vm, unrelated to this change.)

Not in this PR

No live node touched. The live cutover is a separate, gated step.

…opt-in)

NAT cannot software-filter Windows container egress — VFP does not engage on
an HNS NAT network, so per-endpoint Switch ACLs are inert there. This adds an
opt-in L2Bridge path that binds containers to a host NIC via a NetAdapterName
network policy, giving each container a VFP-managed vSwitch port where ACLs
actually enforce. Proven on metal (static IP); DHCP is the v1 IPAM choice.

The router-safe ACL ladder (buildL2BridgeEgressACLPolicies, a pure function)
matches the Linux end-state: block ALL of 10/8, 172.16/12, 192.168/16,
169.254/16 — whole supernets, no gateway or own-subnet carve-out — and permit
only the internet. VFP is default-DENY once any ACL is present, so the ladder
is, by precedence (lower number wins):

  90    Allow DHCP (UDP 67/68, Out+In) — lease/renew survives the block
  95    Allow extra-allowed CIDRs (future use; default none)
  100   Block the RFC1918 + link-local supernets, Out
  65500 Allow 0.0.0.0/0 Out AND In — both mandatory (the In allow keeps
        TCP SYN-ACK return traffic alive; without the pair the port
        default-denies everything, internet included)

DNS is set to public resolvers on the endpoint so the container never needs
the LAN router for name resolution. The rule set is static (independent of the
leased gateway/DNS) and applied at endpoint creation, before the container
starts — fail-closed, no post-lease window. Any ApplyPolicy error tears down
the endpoint and refuses the job.

NAT stays the default and is untouched: buildEgressBlockPolicies and the NAT
init/setup path are unchanged, and the new ladder is only reached when a pool
sets network.l2bridge_egress = true. The NAT block-only builder is deliberately
NOT repurposed — its whole-supernet block with no carve-out would blackhole the
NAT gateway (10.88.0.1, inside 10/8) if VFP ever engaged.

Config: network.l2bridge_egress (bool), network.host_nic (required when on),
network.public_dns (default 1.1.1.1/8.8.8.8), network.extra_allowed_destinations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant