Pin a pasta that keeps the guest's address on a host without IPv4 - #1059
Conversation
On a host with no IPv4 address and no IPv4 default route, a rootless VM
started with --health-check never turned healthy. fcvm passes pasta
`-a 10.0.2.100 -n 255.255.255.0 -g 10.0.2.2` and relies on pasta not
answering ARP for the -a address. The pinned passt (3f57f0382f6a) found
no IPv4 interface to copy its configuration from, entered local mode, and
replaced the configured address and gateway with 169.254.2.1 and
169.254.2.2. It then answered ARP for 10.0.2.100 with its own MAC, so the
namespace's probes to the guest reached pasta and were refused. The
mapping of the gateway to the host's loopback moved to 169.254.2.2 the
same way.
Upstream fixed local mode in 4e8aa70379a3 ("conf: Honour --address,
--gateway, --netmask in local mode as well"), 46 commits after the old
pin, which is an ancestor of it. The pin moves there: rootfs-config.toml,
scripts/build-passt.sh with the new archive checksum, and
tests/test_pasta_pin.rs, which ties the two together.
New test network::pasta::tests::pinned_pasta::
does_not_answer_arp_for_the_guest_address_in_local_mode runs the binary
`fcvm setup` built with the argument vector build_pasta_args produces. It
starts pasta from a scratch user and network namespace that has no IPv4,
so pasta is in local mode on any host, against a second scratch
namespace, and asserts that ARP for the gateway is answered and ARP for
the guest's address is not. Both address shapes of the vector run. No VM
is started and no privilege is needed. The test needs the binary setup
builds, so it is behind integration-fast: make test-fast and
make test-root run it, make test-unit does not.
A comment in build_pasta_args said pasta ignores NDP for the guest's
IPv6 address. passt answers every neighbour solicitation, at both
commits, and the comment now says so. Nothing in the namespace resolves
that address: the bridge carries IPv4 only.
Tested on a host with no IPv4 address and no IPv4 default route:
make _test-fast FILTER="-p fcvm --lib -E 'test(/does_not_answer_arp_for_the_guest_address_in_local_mode/)'"
old pin: FAILED for both address shapes, "ARP for the guest's address
10.0.2.100 was answered by 9a:55:9a:55:9a:55, which is also the
address pasta answers for the gateway with"
new pin: passed in 3 to 5 s, and as root through make _test-root
commit line alone reverted, then make build: FAILED again, one try
commit line reverted without make build: refused, "BLOCKED: the active
fcvm config does not carry this tree's pasta pin"
fcvm podman run --network rootless --health-check http://localhost/ nginx:alpine
old pin: not healthy in 150 s; the namespace's neighbour for
10.0.2.100 is 9a:55:9a:55:9a:55, pasta's address
new pin: healthy in 31 s; the neighbour is the guest's address and
the health URL returns 200 from the namespace
make _test-unit, limited to network::pasta::, setup:: and
tests/test_pasta_pin.rs: 110 passed
make lint: cargo fmt --check and cargo clippy --all-targets clean; cargo
audit and cargo deny did not run (the audit needs its advisory
database from the network)
make _test-root, one test at a time, new pin: test_sanity_rootless,
test_port_forward_rootless,
test_publish_reaches_loopback_and_stays_contained_rootless,
test_forward_localhost, test_egress_fresh_rootless,
test_clone_port_forward_rootless and test_dns_resolution_in_vm pass.
test_ipv6_egress_to_host fails on this host on both pins.
The archive checksum in scripts/build-passt.sh matches the upstream
snapshot, and the snapshot's content equals the git tree of the commit.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 48 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (11)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b6a8cb730
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The pinned passt (4e8aa70379a3) logs socket errors in udp_sock_errs()
through flow_dbg(), flow_perror_ratelimit() and flow_err_ratelimit(). Those
macros read the flow's state field before any level check, so they require a
non-NULL flow. udp_sock_fwd() calls udp_sock_errs() with FLOW_SIDX_NONE for
a listening socket, so uflow is NULL there, and pasta dereferences NULL when
such a socket reports an unqueued error, when SO_ERROR cannot be read, or
when EPOLLERR fires with nothing to report. fcvm starts pasta with -u for
every VM that publishes a UDP port, so the previous pin move could take the
network away from such a VM. The old pin logged and carried on.
scripts/passt-udp-sock-errs-null-flow.patch falls back to the non-flow
logging macros when there is no flow, matching udp_sock_recverr(), which
already logs its own no-flow cases that way. Both build paths apply it:
src/setup/pasta.rs embeds and git-applies it and hashes it into the
content-addressed binary's identity, so a host that built the unpatched pin
rebuilds; scripts/build-passt.sh applies it on the CI, AMI and runner
builds. tests/test_pasta_pin.rs pins the commit, the archive checksum and
the patch together. The same change is prepared as a standalone upstream
submission; drop the carried patch when the pin moves past its merge.
The run-time crash path is not exercised by a test. A published port's
listening socket is receive-only on the host side, so it cannot be driven
to a socket error from user space (ICMP errors attach only to a socket that
sent a matching datagram, and the datagram-sending flow sockets always carry
a valid flow); triggering udp_sock_errs() with a NULL flow would need kernel
fault injection. The patch is pinned at source level instead:
the_carried_patch_guards_the_null_flow_udp_error_sites fails if the patch,
which both build paths apply, stops guarding any of the three sites.
Test Plan:
make _test-unit (pin, setup and network::pasta unit tests): 111 passed,
including both tests in tests/test_pasta_pin.rs. With the guard removed from
the patch, the_carried_patch_guards_the_null_flow_udp_error_sites fails
("patch does not touch the unguarded flow call flow_perror_ratelimit(uflow,
now,") and both_pasta_builds_pin_the_same_upstream_commit_and_patch fails on
the pinned patch checksum; both pass with the patch restored.
The patch applies to the pinned archive with both git apply -p1 and
patch -p1. cargo clippy --all-targets -- -D warnings and cargo fmt clean.
fcvm setup computes the patched pin's new binary identity (sha
b67b2035da64, distinct from the unpatched a347ffe18c89) and enters the
build. The patched source, built from the pinned archive, compiles to a
working pasta with no warnings; the store binary was not rebuilt and the
ARP integration test not re-run because the build host could not reach
passt.top to fetch pasta at the time.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c2ad96b88
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
scripts/build-passt.sh took a directory for finished source as soon as it
held a Makefile. tar writes Makefile before the patch loop runs, so a run
killed in between left an unpatched tree under the fingerprinted name, and
the next run skipped the download and the patch and built and installed the
unpatched pin. Two runs at once shared that directory: the second saw the
first's Makefile while the first was still extracting or patching, and both
ran make in it.
Each run now builds in a directory of its own, made with mktemp and removed
on exit. The verified and patched source is still kept for later runs under
the fingerprinted name. It gets that name in one rename, after it has been
verified, patched and marked, and is never written to or built in
afterwards. A directory under that name without the mark is neither trusted
nor touched, and the run builds from its own copy. A killed run leaves only
directories under names of its own, which no run looks up. When two runs
fetch at once, the second rename fails and that run removes its copy. No
lock is involved. The step is a function, fill_build_dir, so that the tests
can call it.
The other build path, src/setup/pasta.rs, does not have the killed-run
case. It removes its build tree before every build and never reuses one,
and a build counts as complete only when the binary has been renamed onto
its content-addressed path, which happens after the patch applied and make
succeeded. Two builds in one kernel are kept apart by the store's flock.
Between VMs that share the store over FUSE, each guest kernel grants that
flock by itself, and the binary was staged under one fixed name beside its
final path, so the first builder to publish could rename a file the second
was still writing onto the final path. The staging name now carries a uuid,
as the Firecracker build's does, and a failed copy or rename removes it.
Tests. The three in tests/test_pasta_pin.rs run fill_build_dir, taken
verbatim out of the script, on a stand-in archive made from the carried
patch's own context lines, so the real patch applies to it and nothing is
downloaded, built or installed:
a_half_made_source_directory_is_never_built_from
a_kept_source_tree_is_patched_and_serves_a_later_run_without_a_download
an_archive_with_the_wrong_checksum_is_neither_extracted_nor_kept
setup::pasta::tests::two_builders_of_one_binary_stage_to_separate_files
The script before this commit has no function to call, so the red run used
its rules unchanged behind the same two entry points: a directory that
holds a Makefile is reused, and one staging name per binary.
make _test-unit, those rules:
a_half_made_source_directory_is_never_built_from: FAILED, "the next run
would build an unpatched tree: it took the half-made directory for a
finished one", 0 of 3 guards
two_builders_of_one_binary_stage_to_separate_files: FAILED, "the first
builder published the file the second builder was writing"
an_archive_with_the_wrong_checksum_is_neither_extracted_nor_kept:
FAILED, "a tree was kept from an unverified archive"
make _test-unit, this commit with the rename that keeps the source taken
out:
a_kept_source_tree_is_patched_and_serves_a_later_run_without_a_download:
FAILED, the second run found no kept tree
make _test-unit, this commit (network::pasta, setup, test_pasta_pin,
test_ami_hash_inputs): 131 passed.
The script itself, start to end, with the upstream archive served from disk
and stand-ins for curl, make and sudo, BUILD_DIR left as a killed run
leaves it:
before: exit 0, no download, make ran in that directory on a udp.c with 0
of 3 guards, then the install commands
this commit: downloaded, verified, patched, make ran in a directory of
its own on a udp.c with 3 of 3 guards, the half-made directory unchanged
Eight runs at once on a name nothing was kept under, ten rounds:
before: 57 of 80 failed, and 5 exited 0 after make ran on an unpatched
tree
this commit: 80 of 80 ran make on a patched tree, each in its own
directory, with one kept tree and nothing left beside it
cargo clippy --all-targets -- -D warnings and cargo fmt: clean.
Not run: the script against the network, and its install step.
|
@codex review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
ejc3
left a comment
There was a problem hiding this comment.
NOT-A-DEFECT: the review bodies on this pull request carry no finding of their own.
Each inline finding has its answer in its thread: the NULL flow in udp_sock_errs() (RED-VERIFIED: the_carried_patch_guards_the_null_flow_udp_error_sites) and the half-made source directory (RED-VERIFIED: a_half_made_source_directory_is_never_built_from). The last review of the current head found no major issues.
Pin a pasta that keeps the guest's address on a host without IPv4, and carry a
patch that keeps a published UDP port from crashing it.
This PR has three commits.
Commit 1: move the pin to 4e8aa70379a3
Problem
On a host with no IPv4 address and no IPv4 default route, a rootless VM started
with
--health-check http://localhost/never turns healthy. Every probe fromthe VM's namespace to the guest is refused, while the guest serves the same URL
to itself.
build_pasta_argspasses-a 10.0.2.100 -n 255.255.255.0 -g 10.0.2.2and relieson pasta not answering ARP for the
-aaddress. The old pin (3f57f0382f6a)finds no IPv4 interface to copy its configuration from on such a host and enters
local mode, where
conf_ip4_local()overwrites the configured address andgateway with 169.254.2.1 and 169.254.2.2.
arp.cdeclines ARP only forc->ip4.addr, so pasta answers ARP for 10.0.2.100 with its own MAC and thenamespace sends pasta every packet meant for the guest. The same overwrite moves
the mapping of the gateway to the host's loopback: pasta's startup output says
NAT to host 127.0.0.1: 169.254.2.2where fcvm passed 10.0.2.2.CI does not see it. Its runners have IPv4, so pasta copies from the host and
keeps
-a.Change
4e8aa70379a3("conf: Honour --address, --gateway,--netmask in local mode as well"). The old pin is an ancestor of it, 46 commits
earlier, so the addr_seen fix for test_clone_port_forward_stress_rootless: first clone's pre-storm curl RSTs (health-ready vs proxy-path-ready race; reproduces on main) #661 stays in.
rootfs-config.toml,scripts/build-passt.sh(commit and archive SHA-256) andtests/test_pasta_pin.rsmove together.network::pasta::tests::pinned_pasta::does_not_answer_arp_for_the_guest_address_in_local_mode.It runs the binary
fcvm setupbuilt, with the argument vectorbuild_pasta_argsproduces, from a scratch user and network namespace that hasno IPv4, so pasta is in local mode wherever the test runs. A second scratch
namespace is pasta's target. It brings pasta's device up with 10.0.2.1/24, makes
the namespace send ARP for the guest and then the gateway, and reads the
neighbour table: the gateway must resolve (the control), and the guest's entry
must exist with no link-layer address. Both address shapes run. No VM, no
privilege. It is behind
integration-fast:make test-fast,make test-rootand
make container-testrun it,make test-unitdoes not.build_pasta_argssaid pasta ignores NDP for the guest's IPv6address. passt answers every neighbour solicitation, at both commits. The
comment now says that, and that nothing in the namespace resolves that address.
Commit 2: guard udp_sock_errs() against a NULL flow
Problem
The new pin added flow-specific logging to
udp_sock_errs()inudp.c.flow_dbg(),flow_perror_ratelimit()andflow_err_ratelimit()read theflow's
statefield before any level check, so they require a non-NULL flow.udp_sock_fwd()callsudp_sock_errs()withFLOW_SIDX_NONEfor a listeningsocket, so
uflowis NULL there, and pasta dereferences NULL when such a socketreports an unqueued error, when
SO_ERRORcannot be read, or when EPOLLERR fireswith nothing to report. fcvm starts pasta with
-ufor every VM that publishes aUDP port, so the pin move alone could take the network away from such a VM. The
old pin logged and carried on.
Change
scripts/passt-udp-sock-errs-null-flow.patchfalls back to the non-flowlogging macros when there is no flow, matching
udp_sock_recverr(), whichalready logs its own no-flow cases that way.
src/setup/pasta.rsembeds andgit applys it andhashes it into the content-addressed binary's identity (so a host that built the
unpatched pin rebuilds), and
scripts/build-passt.shapplies it on the CI, AMIand runner builds.
tests/test_pasta_pin.rspins the commit, the archivechecksum and the patch together. The same change is prepared as a standalone
upstream submission; the carried patch is dropped once the pin moves past its
merge.
socket is receive-only on the host side, so it cannot be driven to a socket error
from user space (ICMP errors attach only to a socket that sent a matching
datagram, and the datagram-sending flow sockets always carry a valid flow);
triggering
udp_sock_errs()with a NULL flow would need kernel fault injection.The patch is pinned at source level instead:
the_carried_patch_guards_the_null_flow_udp_error_sitesfails if the patch,which both build paths apply to the tree they compile, stops guarding any of the
three sites.
Commit 3: never build pasta from a half-made source directory
Problem
scripts/build-passt.shtook a directory for finished source as soon as it helda
Makefile.tarwritesMakefilebefore the patch loop runs, so a run killedin between left an unpatched tree under the fingerprinted name. The next run
skipped the download and the patch, and built and installed the unpatched pin.
Two runs at once shared that directory: the second saw the first's
Makefilewhile the first was still extracting or patching, and both ran
makein it.Change
mktempand removed onexit. The verified and patched source is still kept for later runs under the
fingerprinted name. It gets that name in one rename, after it has been
verified, patched and marked, and is never written to or built in afterwards. A
directory under that name without the mark is neither trusted nor touched, and
the run builds from its own copy. A killed run leaves only directories under
names of its own, which no run looks up. When two runs fetch at once, the second
rename fails and that run removes its copy. No lock is involved.
src/setup/pasta.rsdoes not have the killed-run case. It removes its buildtree before every build and never reuses one, and a build counts as complete
only when the binary has been renamed onto its content-addressed path, after the
patch applied and
makesucceeded. Two builds in one kernel are kept apart bythe store's flock. Between VMs that share the store over FUSE, each guest kernel
grants that flock by itself, and the binary was staged under one fixed name, so
the first builder to publish could rename a file the second was still writing
onto the final path. The staging name now carries a uuid, as the Firecracker
build's does.
Test results
On a host with no IPv4 address and no IPv4 default route, under load from other
work:
3f57f0382f6a4e8aa70379a3make _test-fast FILTER=.../does_not_answer_arp_for_the_guest_address_in_local_mode)ARP for the guest's address 10.0.2.100 was answered by 9a:55:9a:55:9a:55, which is also the address pasta answers for the gateway withfcvm podman run --network rootless --health-check http://localhost/ nginx:alpineNAT to host 127.0.0.1: 169.254.2.2NAT to host 127.0.0.1: 10.0.2.2Commit 2:
make _test-unit(pin, setup andnetwork::pastaunit tests): 111 passed,including both tests in
tests/test_pasta_pin.rs.the_carried_patch_guards_the_null_flow_udp_error_sitesfails ("patch does nottouch the unguarded flow call flow_perror_ratelimit(uflow, now,") and
both_pasta_builds_pin_the_same_upstream_commit_and_patchfails on the pinnedpatch checksum; both pass with the patch restored.
git apply -p1andpatch -p1, and the patched source compiles to a working pasta under passt's-Wall -Wextra -pedanticwith no warnings.fcvm setupbuilds the patched pin into a new content-addressed binary(
pasta-b67b2035da64.bin, where the unpatched pin waspasta-a347ffe18c89.bin),and the ARP test passes against it.
cargo clippy --all-targets -- -D warningsandcargo fmtclean.Commit 3:
a_half_made_source_directory_is_never_built_from,a_kept_source_tree_is_patched_and_serves_a_later_run_without_a_downloadandan_archive_with_the_wrong_checksum_is_neither_extracted_nor_keptrun thescript's
fill_build_dir, taken verbatim out of the script, on a stand-inarchive made from the carried patch's own context lines. Nothing is downloaded,
built or installed.
setup::pasta::tests::two_builders_of_one_binary_stage_to_separate_filesstages two builds of one binary before either is published.
rules unchanged behind the same entry points (a directory that holds a
Makefileis reused; one staging name per binary):a_half_made_source_directory_is_never_built_fromFAILED ("the next run wouldbuild an unpatched tree: it took the half-made directory for a finished one", 0
of 3 guards),
two_builders_of_one_binary_stage_to_separate_filesFAILED ("thefirst builder published the file the second builder was writing"), and
an_archive_with_the_wrong_checksum_is_neither_extracted_nor_keptFAILED ("atree was kept from an unverified archive"). With the rename that keeps the
source taken out of the fixed script,
a_kept_source_tree_is_patched_and_serves_a_later_run_without_a_downloadFAILED.make _test-unit(network::pasta,setup,test_pasta_pin,test_ami_hash_inputs): 131 passed.stand-ins for
curl,makeandsudo:BUILD_DIRleft as a killed run leaves itmakeran in that directory on audp.cwith 0 of 3 guards, then the install commandsmakeran in a directory of its own on audp.cwith 3 of 3 guards; the half-made directory unchangedmakeran on an unpatched treemakeon a patched tree, each in its own directory, with one kept tree and nothing left beside itcargo clippy --all-targets -- -D warningsandcargo fmtclean.What the upstream commits change for fcvm
passes. On a host with IPv4 the changed function is not reached.
fcvm's log under
--quiet.build_pasta_argspasses parses the same.udp_sock_errs()for a published UDP port, is fixed by the carried patch incommit 2.
Not run
cargo auditandcargo deny(need the advisory database from the network); thefull unit, fast and root suites; bridged and routed tests; arm64; the container
targets; a host that has IPv4; and
scripts/build-passt.shagainst the networkwith its install step. The run-time path of the crash commit 2 fixes has no test,
for the reason given there. CI covers the rest.
Fixes #1056.