fix(control): pace the response poller by what a poll costs

[EnRaiha]

Problem

response_poller never reaches its own backoff. Its idle counter is cleared by any routed response, so as long as the cores keep answering — heartbeats on an otherwise idle node are enough — idle_iters never climbs to the first wait and the loop stays on its yield_now fast path for the life of the process.

The code-level defect is real, and the arithmetic is worse than it looks. A poll of this loop is not cheap: poll_and_route_responses() takes the dispatcher mutex and poll_responses() calls flush_wfq() per live core on every call, whether or not anything was routed (bridge/dispatch/dispatcher.rs:373-377). Instrumented on a live node, one fast-path iteration costs ~4.3 ms (debug build, 256 iterations in ~1.1 s), so 256 consecutive empty polls take about a second of wall clock — and the counter is cleared by work, not by the wait. Any response rate above roughly one per second therefore holds the loop below the first wait, and ~230 iterations/second at 4.3 ms each is a pinned worker.

What the measurement also showed

Instrumenting a live idle node (origin/main, two data-plane cores, no clients) produced a result worth recording, because it changes what this PR can claim:

measure	value
pinned worker	100.4% of a core; whole process ~115%; data-core-0/1 at 0.0%
response_poller on that node	0 responses routed in 150 s, 90.8 polls/s, settled in the 10 ms wait (global_max_idle 1338 → 14512)
the actual burner	nodedb-cluster/src/swim/bootstrap.rs:172 → FailureDetector::run: 1,920 ms CPU per 2,000 ms, 418,000 iterations/s

So on the node measured, the poller was idle and the pinned worker was the SWIM failure detector, whose select! arm on shutdown.changed() resolves with Err forever once every sender is dropped, and whose is_ok() guard turns that into "keep looping". That is a separate defect and is deliberately not patched here — see the scope note below.

The pacing defect in this PR stands on its own: it is what the issue describes, it is reachable, and it costs a core whenever the response rate is high enough.

Change

Pace the loop by what a poll costs instead of trusting the last poll.

• PollPacer::step(routed, cost) takes the measured duration of the poll that just ran. wait_for(cost) returns max(TICK, (FAST_POLLS + 1) * cost * (100 / MAX_DUTY_PERCENT - 1)), so a cycle of FAST_POLLS + 1 polls plus the wait stays inside a 10% duty budget — for every per-poll cost, not just the one measured here. A cheap poll still falls back to the 1 ms wait.
• idle keeps counting polls that routed nothing and widens the wait to 10 ms at the same 1024 threshold the loop already used, so a node that goes quiet settles on the timeline it had before.
• A response that lands just behind a drained batch still earns one poll without a wait.

A fixed wait cannot do this, which is exactly what review caught on the first attempt: with a fixed 1 ms wait and a 4.3 ms poll the loop sits at ~90% of a worker, so it would have been a 10-point improvement rather than a bound. The wait has to be derived from the cost, so it is.

The pacer lives in its own module (nodedb/src/bootstrap/poll_pacer.rs, 114 non-test lines) rather than inline, because the reasoning it carries does not fit the repo's 500-non-test-line cap alongside the spawner.

Evidence

step	run	exit	log	commit
red — same tests, base's pacing decisions behind the same API	nextest run -p nodedb --lib -E 'test(~poll_pacer::tests)'	100 (4 of 6 fail)	20260926T062215-pacer-module-red-base-base.log	c7515e926
green — with the cost-aware pacer	same filter	0 (6 passed)	20260926T063134-pacer-module-green-on-commit-fix.log	c7515e926
preflight	cargo fmt --all -- --check + repo preflight vs origin/main	0	20260926T064048-pacer-module-hygiene-on-commit.log	c7515e926
full nodedb lib suite	nextest run -p nodedb --lib	0 (7125 passed)	20260926T064102-nodedb-lib-full-suite-final-fix.log	c7515e926
CI re-run after the lint fix	Lint & Check	see PR checks	nextest/clippy on CI	c7515e926

The red arm is base's three inline tiers moved behind the pacer's API unchanged, so the tests are the only new code on that arm and the pricing decisions under test are byte-for-byte the ones being replaced. Red fails on the defect itself — a 10 us poll held 100% of a worker, fast path ran 10000 polls without a wait, trickle held the fast path for 10000 polls.

The duty bound is asserted across three orders of magnitude of poll cost (10 µs → 10 ms), because that is the invariant that must hold whatever a poll costs on a given build.

The first push failed CI's Lint & Check: clippy with -D warnings (a newer toolchain than the development box) flagged clippy::manual_repeat_n in the new test. Fixed by using repeat_n, which is stable and already used in-tree; the fix is one line in the test module and the production pacing logic is byte-identical (verified in review by hashing the non-test region of poll_pacer.rs across both commits).

Review

Independent Review 2: PASS, 0 blockers, on c7515e926.

The first review of the earlier commit (0681531a7) returned FAIL and was right: that version bounded the length of the fast run but not CPU, leaving the loop at ~89.6% of a worker in the worst case and ~81% for any rate above ~0.18/s. The wait is now derived from the measured cost, and the reviewer re-derived the bound independently — 2c / (2c + 18c) = exactly the 10% budget at 100 µs, 4.3 ms and 50 ms, with an adversarial search over routed patterns of period ≤ 4 finding nothing above it. Remaining notes are nits: the fast poll's cost is not charged separately in the model (the dominant per-poll cost is unconditional, so the two are ~equal), and busy-path routing latency moves to ~20× the poll cost by construction — that is the trade the issue asks for ("degrades to latency rather than to a pinned core").

Scope notes

• The pinned worker measured on our node is not fixed by this PR. It was the SWIM failure detector (96% of a core, 418,000 iterations/second), not the poller. The one-line shape of that fix — treat a gone shutdown sender as terminal — is written and measured, and it is held back deliberately: it deterministically breaks the repository's own multi-node integration tests. Same worktree, same target directory, back to back:

  arm	result	wall
  origin/main code	7/7 pass	128 s
  Err → break	2 passed / 5 failed	493 s
  park the shutdown arm	3 passed / 4 failed	496 s
  park the task (future::pending)	2 passed / 5 failed	494 s

  Failures are compulsory, not flaky (every failing test failed all 3 nextest retries), and the unpatched tree is faster, which a CPU-burning spin should not be — so the spin is load-bearing for those tests in a way that needs a maintainer's call on what a detector with no shutdown sender is supposed to do. Raising it separately rather than smuggling it in here.

• tokio is not implicated and is not modified. The vendored registry source is byte-identical to upstream tokio-1.53.1 (cloned and diffed; only cargo's publish normalisation differs). Upstream #7883 is a related but different timer livelock (fix PR #8146 open) and would not have caused either symptom here.

• Not changed: poll_responses() still calls flush_wfq() per core on every poll. That is the cost this pacer now paces around; making the poll itself cheap (or gating it on the cores' response notifier, the issue's third suggestion) would remove the polling entirely and is the better long-term shape. It is a dispatcher change with its own risk surface and is left for a follow-up.

Closes #376

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[EnRaiha]

Superseded by #410.

The head branch fix/issue376-poller-backoff lives in this repository, and this account has no push access to it (permissions: {"push": false}), so the rebased work cannot update this pull request. The branch now lives at EnRaiha/nodedb:pr/377-poller-backoff.

What changed in the replacement:

• Rebased onto aa6e91bbd. The branch was 71 commits behind main.
• One conflict, in nodedb/src/bootstrap/mod.rs, where both sides added a single pub mod line. Both are kept.
• The diff against the new base is unchanged: three files, +258/−13. cargo check -p nodedb exits 0.
• Evidence in the replacement body is re-pointed at the new head 3aa4f0515; the technical content, the measurement, the duty bound, and the scope notes are carried over unchanged.

No code decision is revisited here. The rebase is mechanical, and the one conflict has a no-judgement resolution.