Per-slot LMDB fsync stalls block production under write-heavy load

[Dodecahedr0x]

Problem

PersistedStore::flush unconditionally calls index.flush() even when an asynchronous flush was requested. The LMDB environment runs with NO_SYNC, so index.flush() is a full fsync (F_FULLFSYNC on macOS) whose latency grows with the dirty set. Under a write-heavy burst the per-slot fsync takes longer than the slot interval: the validator stops producing blocks, its blockhash stops rotating, and every retried transaction is then rejected as AlreadyProcessed. From the outside the validator looks frozen while sample shows the block production thread parked in fcntl.

Reproduction

Prerequisites: rustup, the Solana CLI, Anchor CLI 1.x, Node 20+ with pnpm, and mb-stack (npm i -g @magicblock-labs/ephemeral-validator@0.14.10).

1. Clone the test workload and the validator (validator fix-stack tip) side by side — the workload's test harness builds and launches the sibling validator with cargo run automatically:

git clone -b dode/test-undelegation https://github.com/magicblock-labs/sealed-bid-metadao
git clone -b dode/committor-backlog-ordering https://github.com/magicblock-labs/magicblock-validator

2. Point the validator at the engine revision missing this fix (head of dode/prepare-empty-writable, the base of fix: sync on checksum #105):

sed -i '' 's/ec741da47eca2f43f5797dde2530640ecb12aa39/596da51c491d2220dd065f6f07a638c52294a699/g' magicblock-validator/Cargo.toml
# Linux: sed -i (without '')

3. Run the oversized-result lifecycle test (41,000 bid tickets delegated, recorded, and undelegated in one burst):

cd sealed-bid-metadao
pnpm install
MOCHA_GREP="grows the published result" pnpm test:local

4. Observe: during the drain the ephemeral validator's slot stops advancing for long stretches (getSlot against http://127.0.0.1:17799 freezes). On failure the harness preserves magicblock-stack-<timestamp>.log in the sealed-bid repo root; it fills with repeated Failed to accept scheduled commits / AlreadyProcessed. Sampling the ER process during a freeze (sample $(lsof -nP -tiTCP:17799 -sTCP:LISTEN) 5) shows the flush path blocked in a full filesystem sync.

Restoring rev ec741da47eca2f43f5797dde2530640ecb12aa39 (this fix applied, #105) keeps block production running through the burst.

Expected

An async flush only flushes the storage mmap. The LMDB index fsync runs only on sync flushes at checksum boundaries, where durability is actually established; a torn async flush is already caught by store validation on startup.

Context

accountsdb/src/store/mod.rs, PersistedStore::flush, dev @ cd02197 (0.3.1).