Skip to content

Reduce forwarded owner lookup and qualify write capacity - #24

Merged
forhappy merged 15 commits into
mainfrom
codex/owner-hint-write-capacity
Sep 30, 2026
Merged

forhappy merged 15 commits into
mainfrom
codex/owner-hint-write-capacity

Conversation

@forhappy

@forhappy forhappy commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add a bounded, authority-validated owner hint to the optional peer HTTP adapter: 4,096 Cells, at most 5 seconds, and a lease safety margin. Retry only after a proven not-started refusal; never replay an ambiguous mutation.
  • Add forwarding tests for takeover, retirement, tombstones, expiry, cancellation, authorization, and retry behavior, plus paired latency evidence.
  • Skip local catalog and control reads when the actor has no dispatchable Cell; keep exact verification for a local hit and overlap independent catalog/control reads.
  • Record response-winning durability proof separately from later publication, actor queue/worker timing, LTX/provider phases, and network follower append timing.
  • Add fixed 12-Cell object-proof and follower-proof capacity workloads with hot, uniform, and skewed traffic, strict readback/evidence validation, and three-repeat CI jobs.
  • Add a signed TCP follower fixture and an owner-loss test that seals and reads the exact unpublished network tail. Document the measured capacity results in the dated report. Optimization plans remain outside the repository.

Measured evidence

  • Three paired adapter runs reduced warm forwarded concurrency-16 p95 from 38.966–92.130 ms to 9.543–25.913 ms, with zero sender metadata reads while the hint was live. These runs predate the entry-node change.
  • A counted-store runtime test now confirms zero local metadata body reads for remote Describe and Query, and three reads for a local query that still verifies catalog and authority. End-to-end latency for the combined client path has not yet been measured.
  • CI runs 36655966439 and 36659182959 each passed three object-proof repeats with readback integrity and zero end-window root lag. At overload, the hot owner published about 58–65 roots/s while actor queue p95 reached 136–160 ms and SQL worker p95 stayed near 2–3 ms. Serial object publication is the measured hot Cell limiter for those object-proof runs; exact throughput varies by runner.
  • CI run 36663653146 passed both three-repeat lanes. Every acknowledged write passed readback and every final root covered its receipts. Follower-proof response latency and fully served rates varied substantially across shared runners, so a repeatable throughput or latency gain is not claimed.

Verification

Focused adapter, application, parser, follower recovery, and LTX tests; workspace format/Clippy; architecture, module layout, documentation, and SQL/peer checks passed on the implementation revisions. The runtime suite, counted-store routing test, workspace all-feature check and Clippy, API docs, boundaries, module layout, document gates, and SQL/peer contract checks pass at the latest revision. All seven CI checks passed at the previous revision; the new CI run must pass before marking this PR ready.

Follow-up work outside this PR

Publication optimization needs a controlled A/A baseline, root-preparation subphase attribution, and a numeric improvement gate. Follower transport optimization needs attribution of member resolution, TCP, authority lookup, and follower fsync. These are future changes; this PR provides the measurement and recovery evidence to guide them.

Local forwarded-client verification

  • The counted-store runtime test proves a remote Describe plus Query makes zero entry-node metadata body reads; a local query still makes three exact reads. New regression checks refuse a stale peer handle after drain and suppress an in-flight query result after node-lease fencing.
  • A local paired debug-build benchmark used 128 sequential queries and 256 queries in 16 batches, comparing the old serial catalog/control resolver with runtime_with_peer. With a synthetic 2 ms delay per object read, the full signed loopback query p95 fell from 30–35 ms to 22–26 ms at concurrency 16; the routing-only p95 fell from 20–21 ms to about 5.5 ms. Entry-node metadata reads fell from three per call to zero.
  • With an in-memory zero-delay store at concurrency 16, routing-only p95 rose from about 1.1 ms to 2.8 ms even though lane completion improved from about 115 ms to 50 ms. This exposes an actor-mailbox tail cost when object reads are exceptionally cheap. The synthetic benchmark does not establish production p95 or the 20-node exit gate. Temporary benchmark source and raw values remain outside the repository under the checkout's crabbuild-target/.../evidence directory.
  • Local cargo test -p cellule-runtime --features test-support --locked -- --test-threads=1 passed, including 379 unit and 161 runtime integration tests; the 18 protocol and 18 peer HTTP tests passed. A parallel ownership run had one intermittent shutdown-result assertion failure after lease fencing; that exact test passed twice alone and the serialized ownership suite passed. The new read-output fence test passed. Formatting, focused Clippy, boundaries, and module layout passed.

@forhappy
forhappy marked this pull request as ready for review September 30, 2026 05:21
@forhappy
forhappy merged commit 9e17746 into main Sep 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant