fix(bulk): bill the committed-slot read, bound BulkAppend estimates, validate chunk header (#822 follow-up)

[QuantumExplorer]

Follow-up to #825 (issue #822), which was merged before its last review fix landed. This carries that fix plus the CodeRabbit findings on #825.

Review on #825 (codex)

The committed-slot read is now billed on every path. The read that sizes a slot rewrite was performed inside the dense tree and discarded by BulkAppendTree::append_no_state_root, so V4 CT/Bulk appends did one unbilled seek per buffered append. The read now lives in BulkAppendTree::slot_write_accounting, which knows from the count it was opened with whether a slot holds a committed value — epoch-1 slots and compacting appends are not read at all (no phantom seek), a real rewrite costs exactly one seek + the committed bytes — and the dense tree takes SlotWriteAccounting::Overwrite { previous_value_len }. The Result-returning appends carry the read alongside the prepaid share in one AppendResult/AppendNoStateRootResult::storage_accounting_cost: OperationCost (caller bills, like hash_count; append_deferred_roots includes it in its cost and mirrors it); CommitmentTree::append_raw/append_many_raw and the GroveDB bulk op add it. The CT estimator gained the seek and the entry bytes; the #813 bound tests stay green. GroveDB tests assert the real RocksDB (+1 seek, +committed_len loaded) deltas for CT, Bulk and PDS on buffered appends from epoch 2 on, and (0, 0) in epoch 1, at compaction, and for an epoch boundary inside one batch.

BulkAppend estimates at compaction are bounds again. Worst-case arm: replaced_bytes and the slot-read storage_loaded_bytes saturate at u32::MAX (chunk entry lengths are u32; neither the op nor the worst-case layer info bounds earlier entries), and added_bytes is a real V4 bound (slot + share + blob framing at the 2^16 epoch + MMR internal nodes) instead of the old flat 64 KB. Average-case arm: when the tree's own layer is declared TreeType::BulkAppendTree(chunk_power) (the estimator now matches BulkAppend ops against that declaration, like CT/PDS) it charges the compaction as epoch × value_size replaced, the framing, and an epoch-scaled hash bound (the old arm's 1 hash per append already under-estimated buffered appends); undeclared it stays the amortized one-entry average, documented as such. New test bulk_append_estimates_dominate_actual_compaction_with_variable_sizes: fifteen 10 KiB values then a 16-byte overflow (≈150 KB replaced for a 16-byte op) — the worst case dominates; sixteen same-size values — the declared average and the worst case both dominate.

CodeRabbit on #825

• chunk_blob_entry_bytes validates the fixed-format header (count × entry_size == payload) before reporting prepaid bytes; malformed blobs prepay nothing. Test added.
• The legacy-accounting tests cover GROVE_V2 alongside V1/V3 (CT frontier cost and bulk slot/blob puts).
• "Propagate committed-slot read cost" and "restore an upper bound for variable-size BulkAppend compaction" — addressed above.
• "Track the frontier length after each successful save" — deliberately not: persisted_frontier_len is the size loaded at open, and save must keep comparing against it because a StorageBatch keeps one put per key — a session that saves twice is charged for the last put only, which has to describe the transition from the committed frontier (documented on the field). GroveDB opens a fresh handle per operation, so no handle ever saves against a stale figure; a tree built with new() has no committed frontier, so its first save creating the key as new storage is correct.

Verified: clippy -D warnings, all-targets check, verify-feature build, fmt, full workspace tests (4944 passed before the two CodeRabbit quick wins; affected suites re-run after).

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

@QuantumExplorer, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5aaa4334-5b3b-4da3-91af-30d346a8dd06

📥 Commits

Reviewing files that changed from the base of the PR and between 4eb6562 and 4f2d29b.

📒 Files selected for processing (17)

• docs/crates/costs.md
• grovedb-bulk-append-tree/src/cost/mod.rs
• grovedb-bulk-append-tree/src/cost/v0.rs
• grovedb-bulk-append-tree/src/cost/v1.rs
• grovedb-bulk-append-tree/src/tree/append.rs
• grovedb-bulk-append-tree/src/tree/mod.rs
• grovedb-bulk-append-tree/src/tree/storage_accounting_tests.rs
• grovedb-commitment-tree/src/commitment_tree/cost/mod.rs
• grovedb-commitment-tree/src/commitment_tree/mod.rs
• grovedb-dense-fixed-sized-merkle-tree/src/test_utils.rs
• grovedb-dense-fixed-sized-merkle-tree/src/tests.rs
• grovedb-dense-fixed-sized-merkle-tree/src/tree.rs
• grovedb/src/batch/estimated_costs/average_case_costs.rs
• grovedb/src/batch/estimated_costs/mod.rs
• grovedb/src/batch/estimated_costs/worst_case_costs.rs
• grovedb/src/operations/bulk_append_tree.rs
• grovedb/src/tests/append_storage_accounting_tests.rs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.37%. Comparing base (4eb6562) to head (4f2d29b).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #826   +/-   ##
========================================
  Coverage    92.36%   92.37%           
========================================
  Files          288      288           
  Lines        87470    87527   +57     
========================================
+ Hits         80794    80851   +57     
  Misses        6676     6676           

Components	Coverage Δ
grovedb-core	90.52% <100.00%> (+<0.01%)	⬆️
merk	93.27% <ø> (ø)
storage	87.08% <ø> (ø)
commitment-tree	96.29% <100.00%> (-0.02%)	⬇️
mmr	96.49% <ø> (ø)
bulk-append-tree	92.27% <100.00%> (+0.31%)	⬆️
element	97.98% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.