Skip to content

refactor: route engine clocks and timers through the runtime - #218

Draft
greenhat616 wants to merge 5 commits into
fix/hardening-48-hung-connect-evidencefrom
refactor/hardening-49-runtime-timers
Draft

greenhat616 wants to merge 5 commits into
fix/hardening-48-hung-connect-evidencefrom
refactor/hardening-49-runtime-timers

Conversation

@greenhat616

Copy link
Copy Markdown
Owner

PR-6c of the hardening stack (T33; design D33, §14.34; crates/bolt-load/docs/runtime.md, docs/runner.md). Stacked on #217, which is its base.

Why

Engine timers and time reads used async_io::Timer and Instant::now() directly. Under tokio they were invisible to tokio-console and ignored tokio::time::pause. Tokio periodic timers were also created in whatever runtime was ambient, not the task's own.

Maintainer decisions (2026-11-13, D33):

  • Engine timers and clocks go through the runtime: tokio under tokio, bound to the task's handle.
  • async_io stays only for three cases:
    • the meta retry with rt: None;
    • the standalone runner built without a timer source;
    • runtime::timeout, which is kept and not deprecated.
  • The runner gets an optional .timer_source(..); without one it falls back to async_io.

What

  • C43 feat(runtime): add the runtime clock and bind periodic timers
    • Adds TimerBuilder::now, defaulting to std, forwarded through both impl enums including Other.
    • Tokio now and create_delayed_timer now run under the stored handle.
    • Tokio gains the time feature; the tokio dev-dependency gains test-util.
    • Tests are isolated by feature. Before this, the non-tokio configurations did not compile their tests. Three examples and the coordination bench now require tokio.
  • C44 feat(task): read engine time from the runtime
    • Every engine time read uses the runtime clock: loop events, both throughput samplers, checkpoints (completion and teardown), the singleton sampler, and the runner's construction time, connect duration and read activity.
    • Adds a public, non-breaking TaskRunnerBuilder::timer_source(ThreadedRuntimeImpl), set at both production sites (runner_spawner.rs, singleton_task.rs).
  • C45 refactor(runner): drive timeouts through runtime deadlines
    • Adds a crate-internal re-armable Deadline with four variants: Tokio, Smol, Custom (for Other), and the async-io Fallback.
    • Behaviour: a fired latch; a reset discards the old sleep; an elapsed target never asks for a zero sleep; the absolute target is kept even when the first poll is late.
    • The runner's connect, idle and drain timeouts use it.
  • C46 refactor(task): drive loop deadlines and singleton sampling through the runtime
    • DeadlineSource uses Deadline.
    • Singleton sampling uses the runtime's periodic timer and still waits a full period before the first sample.
  • C47 test(runtime): cover timer migration across engine paths
    • Whole-path tests: runners from both production sites time out on the task's clock; a loop built with no ambient context, or under another runtime, follows its own paused runtime.
    • Send assertions; spec updates.

Unchanged: policy ordering, runner phase priorities, timeout start points, and the immediate first loop ticks. strategy/ is untouched.

Observable differences (§14.34)

  • The singleton's missed samples follow the runtime's periodic timer instead of async-io's catch-up interval. On tokio the behaviour is Delay.
  • A duration too large for Instant becomes a deadline about 30 years away instead of one that never fires.

Audit

  • Production async_io is down to the three exceptions:
    • runtime/deadline.rs: the Fallback variant;
    • runtime/mod.rs: runtime::timeout;
    • task/meta.rs: rt: None.
  • Production Instant::now() appears only inside runtime/, and there is no .elapsed().

Not done

  • No virtual-clock integration cases for the redesign policies. Those policies take an injected now and their tests are unchanged.
  • No manual tokio-console session.
  • bolt_load_probe_send is not extended; direct Send assertions replace it.

Verification

  • fmt and clippy -D warnings pass on every commit.

  • cargo test --workspace --all-features passes on every commit (0 failed, 1 ignored):

    Commit Passed
    C43 1066
    C44 1070
    C45 1085
    C46 1089
    C47 1095
  • Feature matrix at the tip (clippy and tests):

    Configuration Passed
    tokio only 848
    smol only 744
    both 856
    neither 713
  • cargo bench --no-run passes, and clippy passes under --cfg tokio_unstable.

  • Mutation: forcing the async-io fallback fails 9 of the new path tests.

  • Reviewed by codex: a plan verification and two review rounds (single backend). Final verdict "ready".

The engine compares the instants it reads with timers of its task's
runtime, but read them from the standard clock while tokio's timers run
on tokio's: a paused or advanced tokio clock moved the timers and not
the instants. `TimerBuilder::now` is the clock of a runtime's timers,
the standard one by default; both runtime wrappers forward it, and the
tokio runtimes read tokio's clock through their own handle.

The tokio runtimes also created their periodic timers in the caller's
tokio context: built outside one they panicked, and built under another
runtime they ran on that one, hidden from the task's runtime in
tokio-console and from its clock. They are now created on the runtime's
own handle, as its sleeps already were. The first tick stays immediate
and missed ticks are still delayed.

The `tokio` feature now asks for tokio's `time` explicitly, which these
timers need; the tests get `test-util` to pause and advance a runtime's
clock, and a clock that moves only when told for custom runtimes.

The library's tests now build with either runtime feature or none:
engine tests that only spawn run on the test's tokio runtime through a
helper, the whole-task suites, which need the tokio local runtime, and
the examples and the coordination bench require the `tokio` feature.
The engine read the standard clock wherever it took the time: the
loop's event times and its failure policy, both speed samplers, the
checkpoint spacing, the singleton's sampler, and the runners' build
time, connect duration and read activity. They now read the clock of
the task's runtime, the one its timers run on, so a paused or advanced
runtime moves them all together.

The speed samplers take the time of each sample, and the throughput
tick passes the loop's event time to both the download and the write
sampler. A checkpoint ends at the time the loop handles its outcome
rather than when its future completed; at teardown the time is read
after each awaited step.

The new public setter `TaskRunnerBuilder::timer_source` gives a runner
the runtime whose clock it reads; both builders in the task, the range
runners' and the singleton's, pass the task's runtime, which the
singleton now keeps.
A runner built without one reads the standard clock as before. Its
timeouts are still the `async-io` timers here.
A runner's connect, idle and drain timeouts were `async-io` timers on
the standard clock, whatever runtime ran it: invisible to tokio-console
and deaf to a paused tokio clock, while the runner's timestamps now
follow the task's runtime.

A crate-internal `Deadline` is a one-shot timer at an absolute instant,
polled and re-armed in place: a tokio sleep registered with the
runtime's own handle, a smol timer, or, for a custom runtime, its own
sleep for the time left, started at the first poll so a late first poll
does not move the target. A target already reached fires without a
sleep, since a custom runtime's default sleep is built on a periodic
timer. Once fired it stays ready until reset.

A runner with a timer source runs its timeouts on that runtime; one
built without keeps `async-io`'s timers on the standard clock, the
documented fallback of a runner used on its own. The timeouts start
where they did: connect at the first poll, idle at `Started` or when
ready and at every non-empty item or resumed read, drain at its first
poll. A timeout too long for an `Instant` saturates far in the future,
as `async-io` never fired it.

The test runtimes whose default sleep could not serve a runner's
timeout now sleep on tokio, or on the standard clock where scripted.
…he runtime

The concurrent loop's deadline, the no-progress deadline or the end of
a backoff or congestion when that comes first, was an `async-io` timer
on the standard clock, and the singleton sampled its speed on an
`async-io` interval. Both now run on the task's runtime: the deadline
is a runtime deadline, the sampling a periodic timer of the runtime.

The deadline keeps its rules: it is moved earlier only by `wake_by`,
re-armed lazily after progress, and armed only for instants after the
time it is handled at, so a retry that finds no free slot does not
spin. It fires once until the loop re-arms it or ends.

The singleton's first sample still comes a full period after the
sampling starts, and again after `Started`: the periodic timer's first
tick, which is immediate, is taken without sampling.

The loop test harness whose deadline was due because a nanosecond had
passed since it was built now runs on a manual clock and moves it by
that nanosecond.

With this, `async-io` timers remain only in the metadata retry before a
task has a runtime, in runners built without a timer source, and in
`runtime::timeout`.
The earlier commits test each piece; these tests follow the time of a
task's runtime through the paths that put the pieces together.

Both places that build runners pass the task's runtime: a runner the
range spawner starts, and the singleton's, time out when a manual
clock of the task's runtime reaches the connect timeout, not before.
The loop's events take their time from the runtime: a frame moves the
no-progress deadline from the runtime's now, and the throughput is the
bytes over the runtime's elapsed time. A loop built with no tokio
context, and one built under another runtime, runs its deadline and
its ticks on its own paused runtime and fails without progress at the
timeout on that clock, with no real wait. The loop's and the runners'
futures, and the deadlines and timer sources, stay `Send`.

The spec records the maintainer's decisions on the timers (D33) and
the task (T33), and design §14.34 the implementation's choices.
@greenhat616
greenhat616 added this pull request to stack #215 November 17, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant