Skip to content

feat(config): make startup bounds configurable - #412

Open
EnRaiha wants to merge 5 commits into
NodeDB-Lab:mainfrom
EnRaiha:drill/issue354
Open

EnRaiha wants to merge 5 commits into
NodeDB-Lab:mainfrom
EnRaiha:drill/issue354

Conversation

@EnRaiha

@EnRaiha EnRaiha commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

A store written by another build failed every boot with a message that called its WAL segments corrupt. The reader already knew the format version and threw it away, which is the root cause. This branch surfaces that version, so startup names what it found and what it requires, and it makes the two startup bounds configurable with a test pinning each one.

What changed

Area Change Paths
WAL reader Return WalError::UnsupportedVersion instead of flattening every header validation failure into StopReason::Corruption nodedb-wal/src/reader.rs
Startup gate Name the version, the segment, and whether the store needs a migration or a newer binary. A zeroed version takes the empty-tail path, so a torn write does not refuse a store that used to boot nodedb/src/wal/manager/replay.rs
Startup bounds Read raft_ready_timeout_ms and data_group_recovery_timeout_ms from [tuning.startup] as newtypes, and refuse them on the [server] path with the key named nodedb-types/src/config/tuning/startup.rs, nodedb/src/config/server/config.rs, nodedb/src/bootstrap/cluster_ready.rs, nodedb/src/bootstrap/data_group_recovery.rs, nodedb/src/control/security/auth_lease/status.rs
Log line The wrapper above the error no longer calls every validation failure corruption nodedb/src/bootstrap/wal_init.rs
Boot order Validate the WAL before the writer opens it. The writer's open recovers the newest segment, so without this the composed message was unreachable on the store that motivated the work nodedb/src/bootstrap/wal_init.rs, nodedb/src/wal/manager/replay.rs
Tests A version gap in a plaintext store and behind a preamble, a zeroed version, an empty tail, a record-less segment behind the tail, unframed bytes, and the bound a caller passes nodedb/src/wal/manager/replay.rs, nodedb/src/bootstrap/cluster_ready.rs, nodedb/tests/inproc/

Why

A 20,000-row store booted in 83.4 s, and the phase that dominates the boot runs after both startup gates, so neither bound covers it. Both bounds were hard-coded and unreachable from the server config.

Separately, the production store could not boot at all, and the message sent its reader looking for damage that was not there. The header carried the answer.

Checks

  • Formatting: cargo fmt --all -- --check passes.
  • Focused tests: exit 0, the replay module and the error-class parity module.
  • Mutation: with the reader's version error removed, exit 100, and exactly the version-message tests fail.
  • Full library suite: 8,785 pass, 0 fail.
  • End to end: a real version-1 production copy is refused with version compatibility: WAL segment '…' holds records in WAL format version 1; this build requires 3. The store needs a migration, not a repair., the word "corrupted" appears zero times in its log, and a real version-3 store boots to serving.
cargo nextest run -p nodedb --lib -E 'test(/wal::manager::replay/) or test(/class_parity/)'

Evidence, logs, the mutation diff and the end-to-end script: https://github.com/EnRaiha/nodedb/blob/pr412-c620167c/docs/review/README.md

Scope

Two units share this pull request, one commit each: the startup bounds under [tuning.startup], and the WAL format diagnosis. They are independent, and a reviewer who wants only one can take the commit that carries it. The third commit is two clippy fixes that CI's -D warnings profile requires.

Review 2: PASS, 0 blockers, bound to this head. The evidence branch, its logs and its tags are linked above.

A rebase merge keeps the unit commits separate. A squash merge would erase them, and the two commits that cover the reader test can be folded into one on request.

Closes #354

Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@EnRaiha
EnRaiha marked this pull request as draft October 6, 2026 14:19
@EnRaiha EnRaiha changed the title feat(config): make startup bounds configurable feat(config): make startup bounds configurable Oct 7, 2026
@EnRaiha
EnRaiha marked this pull request as ready for review October 7, 2026 09:29
The metadata stall bound and the data-group recovery bound were constants in the bootstrap
code, so an operator with a long WAL tail could not raise them. Both are newtypes now, read
from [tuning.startup].

The server config path already rejected such a key through `deny_unknown_fields`, but that
message lists every valid field instead of the path that is read. A guard names the
replacement path, so a misplaced key says where to put it.

The one-node harness passes its own tighter bound rather than inheriting the production
default, which keeps the harness fast without changing what production does. The single-node
harness keeps the shipped defaults on purpose, and its comment says so.
A store written by another build failed every boot with a message calling its segments
corrupt. The root cause was in the reader: WalError::UnsupportedVersion already existed and
the header returned it, but the reader flattened it, and every other validation failure,
into StopReason::Corruption. The gate then saw an empty tail and reported damage that was not
there.

The reader returns that error now, and the startup gate turns it into Error::VersionCompat
carrying the segment path, the version found, the version required, and advice that depends on
the direction of the gap. A zeroed version stays a corruption stop, because the writer paths
share this reader and a torn tail must not stop a store from opening.

The gate also runs before the writer opens. That open recovers the newest segment, so without
the ordering the composed message was unreachable on the store that motivated the work.
The planner's null check becomes is_none_or, and the reply path's negated conjunction is
written as one condition. Both are the lints the CI profile enforces.
A review found the reader-level behaviour had no reader-level test: the gate and the writer
open were covered, but nothing in reader.rs exercised the version match itself. This adds it in
both directions, an unknown version is a typed error and a zeroed version is a corruption stop.

The getting-started paragraph also described what the metadata bound does without saying that
the data-group bound is a hard deadline, which is the difference an operator needs when a boot
is slow.
A review found the zeroed-version case proved nothing: an empty record list is also what a
silent skip or a clean end returns, so a fail-open regression would pass it. The reason is
public, so the test asserts it is Corruption.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(config): make the startup stall and data-group recovery bounds configurable

2 participants