[Storehouse] 014 - Payloadless util create v6 checkpoint from v7

[zhangchiqing]

To be able to support mode switch especially switching from payloadless mode back to full node, we need to create a v6 checkpoint from a v7 checkpoint, so that we can simply converting the latest v7 checkpoint produced by a EN running in payloadless mode, and then restart in full node mode with the converted v6 checkpoint file. Since the wal file format didn't change, they can be reused by the full node to resume the execution.

This PR adds a util to create a v6 checkpoint from a v7 (payloadless) checkpoints and wal (write ahead logs) files.

Summary by CodeRabbit

• New Features

  • Added a command to convert payloadless V7 checkpoints into full V6 checkpoints.
  • Supports automatic discovery of source checkpoints and WAL ranges, with optional overrides.
  • Provides configurable worker counts and output locations.
  • Validates checkpoint data, verifies reconstructed hashes, and cleans up incomplete output after failures.

• Tests

  • Added coverage for successful conversion, source discovery, filename validation, retries, and missing payload scenarios.

[coderabbitai]

Important

Review skipped

The saved review history does not include the base for the last reviewed commit. This saved history cannot establish the base for an incremental review. Comment @coderabbitai full review to establish a new review baseline. No full review was started, and the last reviewed checkpoint was preserved.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: bcfca748-2f76-46fc-a144-221d8ac42d33

📥 Commits

Reviewing files that changed from the base of the PR and between c7234d3 and 4d757da.

📒 Files selected for processing (1)

• ledger/complete/wal/checkpoint_v6_convert_test.go

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Checkpoint conversion

Layer / File(s)	Summary
CLI entrypoint cmd/util/cmd/checkpoint-convert-v6/cmd.go, cmd/util/cmd/root.go	Adds the checkpoint-convert-v6 command, conversion flags, default output naming, logging, and root-command registration.
Conversion setup and source resolution ledger/complete/wal/checkpoint_v6_convert.go	Validates inputs, resolves the previous V6 checkpoint and WAL range, manages temporary output files, and verifies reconstructed root hashes.
Streaming checkpoint conversion ledger/complete/wal/checkpoint_v6_convert_stream.go	Converts subtrie and top-trie files concurrently, reconstructs V6 nodes, verifies checksums, and writes unavailable regSize values as zero.
Payload reconstruction ledger/complete/wal/checkpoint_v6_convert_stream.go	Sources payloads from previous V6 checkpoint files and WAL updates, including top-trie and unallocated leaves.
Conversion validation ledger/complete/wal/checkpoint_v6_convert_test.go	Tests round trips, source selection, WAL handling, validation errors, top-trie and unallocated leaves, missing payloads, and retry cleanup.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant checkpoint_convert_v6
  participant ConvertCheckpointV7ToV6
  participant buildPartitionPayloadPool
  participant scanWALUpdates
  participant convertSubTriesV7ToV6Concurrently
  participant verifyRootHashesMatch
  checkpoint_convert_v6->>ConvertCheckpointV7ToV6: start V7-to-V6 conversion
  ConvertCheckpointV7ToV6->>buildPartitionPayloadPool: build payload sources
  buildPartitionPayloadPool->>scanWALUpdates: read WAL update range
  scanWALUpdates-->>buildPartitionPayloadPool: return updated payloads
  ConvertCheckpointV7ToV6->>convertSubTriesV7ToV6Concurrently: convert subtrie parts
  ConvertCheckpointV7ToV6->>verifyRootHashesMatch: compare reconstructed root hashes

Loading

Merge Risk: 🟡 Moderate · up to 4d757

The conversion utility may delete a valid existing checkpoint on failure, and its validation tests may not verify the intended safeguards. These risks should be addressed before merging.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: adding a utility to create a V6 checkpoint from a payloadless V7 checkpoint. It is specific and related to the pull request.
Docstring Coverage	✅ Passed	Docstring coverage is 87.88% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 5 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch leo/payloadless-util-create-v6-checkpoint-from-v7

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[codecov-commenter]

Codecov Report

❌ Patch coverage is 51.74051% with 305 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...edger/complete/wal/checkpoint_v6_convert_stream.go	55.40%	104 Missing and 90 partials ⚠️
cmd/util/cmd/checkpoint-convert-v6/cmd.go	0.00%	57 Missing ⚠️
ledger/complete/wal/checkpoint_v6_convert.go	61.87%	28 Missing and 25 partials ⚠️
cmd/util/cmd/root.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!