perf(persistence): park-free AOF writer poll under everysec/no

[pilotspacex-byte]

Summary

Closes the last per-cell Moon-vs-Redis durability deficit: everysec P1 SET was 0.80–0.86× Redis. Root cause found via strace -c on the shard thread during an 8s non-pipelined SET run (Linux, monoio, shards=1):

shard-thread syscalls (8s)	nodur	esec (before)	esec (after)
io_uring_enter	47,017	45,639	94,136
futex	2	149,718 (63% of syscall time)	96

The AOF writer parks in flume recv_timeout, so every producer try_send pays a futex WAKE on the shard thread (~1 per 2 SETs). Redis's AOF append is a plain memcpy into aof_buf — hence the gap.

Change

Under EverySec/No, the two std-thread writer loops (monoio TopLevel + PerShard) receive via a new poll_recv() — try_recv + adaptive sleep (wait/16, clamped 500µs–50ms) — which never registers the writer as a flume waiter, so producer sends stay pure userspace atomics. Always keeps the parked recv (its callers await the per-batch fsync ack; receive latency there is client-visible RTT).

• EverySec bound unchanged: at the 50ms IdleWait floor the deadline is re-checked within ~3ms slack.
• Idle cost: ≤20 writer wakes/s at the escalated 1s wait (writer thread only) vs 1/s parked.
• Under load try_recv hits immediately — no sleep, slightly better batching per drain.

GCE A/B (c3-standard-8, Redis 7.0.15, 3 alternated reps, -c 8 -n 300000 -r 100000 -d 64, Moon --shards 2)

config	moon-base (main 4f01e68b)	moon (this PR)	redis	vs base	vs redis
esec SET p1	116,619	134,158	135,888	+15%	0.99× (parity)
nodur SET p1 (control)	135,795	135,319	132,185	1.00×	1.02×
GET p1 (both)	~133k	~135k	~132k	—	—

Test plan

• 4 new poll_recv unit tests (immediate message, mid-wait pickup, timeout, disconnect)
• crash_matrix_per_shard_aof --ignored 3/3 green (SIGKILL durability contract intact)
• fmt; clippy ×2; cargo test --lib ×2 (monoio 3868 / tokio 3143)
• strace before/after: 149,718 → 96 shard-thread futex calls

Summary by CodeRabbit

• Performance Improvements
  • Reduced write-thread wakeups during AOF persistence by using a polling-based wait strategy for common sync modes (every-second and no-sync), lowering thread contention.
  • Preserved durability timing behavior for standard every-second syncing, while keeping the existing always-sync behavior.
• Bug Fixes
  • Improved responsiveness of background AOF channel waiting, avoiding excessive blocking and unnecessary wake events.
• Tests
  • Added coverage for the new polling-based receive behavior, including immediate receive, mid-wait delivery, timeouts, and disconnect handling.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6e0294fd-8d64-4877-bf93-fb9af93f4143

📥 Commits

Reviewing files that changed from the base of the PR and between 6d3138d and ad68b73.

📒 Files selected for processing (2)

• CHANGELOG.md
• src/persistence/aof/writer_task.rs

✅ Files skipped from review due to trivial changes (1)

• CHANGELOG.md

🚧 Files skipped from review as they are similar to previous changes (1)

• src/persistence/aof/writer_task.rs

---

📝 Walkthrough

Walkthrough

Adds a park-free polling receive path for monoio AOF writer loops under EverySec/No fsync policies, while keeping parked receive behavior for FsyncPolicy::Always. Also updates the changelog and adds unit tests for the polling receive helper.

Changes

AOF Writer Park-Free Polling

Layer / File(s)	Summary
poll_recv and recv_next implementation src/persistence/aof/writer_task.rs	Adds poll_recv with adaptive sleep/try_recv polling and deadline enforcement, plus recv_next choosing between parked recv_timeout and poll_recv based on FsyncPolicy.
Writer loop integration and tests src/persistence/aof/writer_task.rs	Updates TopLevel and per-shard monoio writer loops to use recv_next for the first-message receive; adds monoio unit tests for immediate receive, mid-wait receive, timeout, and disconnect cases.
Changelog documentation CHANGELOG.md	Adds an Unreleased entry describing the polling-based receive behavior, its effect on futex wakes, and the unchanged durability semantics.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WriterLoop
  participant recv_next
  participant poll_recv
  participant Channel

  WriterLoop->>recv_next: recv_next(rx, wait, park)
  alt park is true (FsyncPolicy::Always)
    recv_next->>Channel: rx.recv_timeout(wait)
    Channel-->>recv_next: message or Timeout
  else park is false (EverySec/No)
    recv_next->>poll_recv: poll_recv(rx, wait)
    loop until deadline
      poll_recv->>Channel: rx.try_recv()
      Channel-->>poll_recv: Empty or message
      poll_recv->>poll_recv: sleep(step interval)
    end
    poll_recv-->>recv_next: message or Timeout/Disconnected
  end
  recv_next-->>WriterLoop: RecvTimeoutResult

Loading

Possibly related PRs

• pilotspace/moon#178: Both PRs modify the AOF writer batch-drain loops in writer_task.rs affecting fsync and ack handling.
• pilotspace/moon#211: Both PRs alter AOF writer channel behavior for FsyncPolicy::EverySec/No at the producer/consumer level.
• pilotspace/moon#233: Both PRs modify the monoio AOF writer channel-receive and wait logic for EverySec timing behavior.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: park-free polling for the AOF writer under everysec/no.
Description check	✅ Passed	The description covers the summary, performance data, and testing, but it omits the template’s explicit checklist and notes sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/aof-esec-parkfree-writer

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}