perf: reduce durable transaction round trips

[forhappy]

Summary

This PR reduces round trips in BeyondDB's signed DynamoDB path, adds bounded background index projection, and advances Cellule follower-log recovery wiring. The latest commit also retries transient startup recovery after a fenced takeover. It keeps ExtendDB responsible for HTTP, SigV4, IAM, and DynamoDB semantics and Cellule responsible for durable Cell execution.

Follower-backed commit proof is not enabled in the serving binary. Writes still wait for object-store publication. The all-API goal of exceeding pinned ExtendDB SQLite remains open.

Cellule origin/main refresh and full release rerun

Cellule is pinned to 9e17746a633ca1046bd93866866074226091f81a in the manifest and lockfile. The release binary built; formatting, strict all-target Clippy, and all 17 library tests passed. A signed boto3 PutItem of a 1 KiB item survived an unclean serving-process restart on the new release fixture.

The latest complete 24-case report and raw JSON compare the new release with pinned ExtendDB SQLite. Both finished all cases with zero foreground request errors. At eight clients, BeyondDB/SQLite measured 235/507 GetItem, 78/55 PutItem, 4.18/473 TransactGetItems, and 1.50/131 TransactWriteItems requests/s. BeyondDB also logged deferred capacity sweeps and participant resolution. Host load changed from 47.5 to 30.3 during BeyondDB and ended at 37.3 after SQLite on 12 logical CPUs; these rates do not establish a controlled speed ratio. The all-API performance objective is not met.

An earlier attempt with the same pin stopped at eight-client TransactGetItems after a throttling cancellation under heavier load. A local larger signed SDK restart attempt failed with HTTP 503 during GSI setup when host load approached 100. GitHub Actions then passed all seven server_binary tests on the new Cellule pin, including unclean restart and 70-shard process-loss coverage. The peer_network target passed 46 of 48 tests; two failures remain under investigation: sdk_creation_recovery_tolerates_concurrent_delete (placement capacity) and signed_sdk_request_routes_across_two_owners_and_survives_restart (large transaction read received HTTP 503). Test-only peer-route diagnostics are now on this branch for the next CI run.

Earlier full release comparison

The pre-Cellule-update full 24-case report and raw JSON compare a release BeyondDB binary from the pre-retry working tree with pinned ExtendDB SQLite 7eaa89b. Both used signed boto3, 1 KiB items, five-second cases, one and eight clients, and zero foreground SDK errors. At eight clients:

API	BeyondDB req/s	SQLite req/s
GetItem	616.17	518.20
PutItem	74.99	456.42
TransactGetItems	3.79	437.19
TransactWriteItems	3.22	262.70

The host had one-minute load of roughly 22–27 on 12 logical CPUs. These sequential short runs are diagnostic and do not establish a stable speed ratio or production capacity. BeyondDB logged one deferred transaction recovery pass. The index batching change targets background GSI maintenance; the benchmark workload did not create or query a GSI. The earlier recovery-wiring rerun is also included. Historical peak rates in this PR were not reproduced by current release fixtures.

Verification

• cargo fmt --all -- --check, strict locked all-target Clippy, and all 17 library tests passed on the pushed source tree.
• On the new Cellule pin, GitHub Actions passed all seven server_binary tests, including signed SDK 70-shard process-loss and unclean restart coverage. elastic_cells passed 46 tests. peer_network passed 46 of 48; the two named failures above remain open.
• The isolated unavailable-index/healthy-index maintenance regression passed after bounded projection batching. A later repeat on the startup-retry tree stalled during integration-target compilation on the heavily loaded host; the batching code was unchanged.
• The two failing peer_network tests and full green GitHub Actions qualification are remaining PR gates.

Remaining work

Durable writes and cross-Cell transactions are still much slower than local SQLite. Enabling follower acknowledgments requires a proven multi-node crash-after-acknowledged-SDK-write path and safe successor replay. Do not infer fleet-scale capacity from these local fixtures. More detailed benchmarks, bottleneck analysis, and qualification limits are in performance documentation and follower durability design.