[Storehouse] 010 Util convert v6 -> v7 checkpoint

[zhangchiqing]

To bootstrap a payloadless EN, a v7 root checkpoint file is needed. This PR adds a util to convert a v6 root checkpoint into a v7 root checkpoint.

It adds two ways to convert:

1. original way (less code, but use more memory), it loads the entire v6 checkpoint into memory, and then convert it into v7
2. steam way (more new code, but much less memory), it iterate each node in the v6 checkpoint file, if it's a leaf node, convert it into v7 format.

Summary by CodeRabbit

• New Features

  • Added optional streaming mode for V6-to-V7 checkpoint conversion.
  • Added validation for checkpoint metadata and malformed input.

• Bug Fixes

  • Improved cleanup of stale and partially generated checkpoint files without affecting similarly named files.
  • Improved file comparison and error reporting during checkpoint verification.
  • Conversion failures now better preserve existing input data.

• Tests

  • Added coverage for streaming conversion, checksum preservation, validation failures, corrupted input, and temporary-file cleanup.

[coderabbitai]

Important

Review skipped

The saved review history does not include the base for the last reviewed commit. This saved history cannot establish the base for an incremental review. Comment @coderabbitai full review to establish a new review baseline. No full review was started, and the last reviewed checkpoint was preserved.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 49fbf674-b581-43a3-8496-7c72257692a8

📥 Commits

Reviewing files that changed from the base of the PR and between 94228f7 and 4be13db.

📒 Files selected for processing (4)

• integration/localnet/builder/bootstrap.go
• ledger/complete/wal/checkpoint_v6_writer.go
• ledger/complete/wal/checkpoint_v7_convert_stream.go
• ledger/complete/wal/checkpoint_v7_convert_test.go

Included review availability: Your plan provides up to 4 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

The V6-to-V7 checkpoint converter now supports node-by-node streaming conversion. It centralizes validation, checksum verification, temporary-file cleanup, partial-output cleanup, and completion logging. Callers and tests pass the new mode argument.

Checkpoint conversion

Layer / File(s)	Summary
Conversion API and caller updates ledger/complete/wal/checkpoint_v7_convert.go, cmd/util/cmd/checkpoint-convert-v7/cmd.go, cmd/execution_builder.go, integration/localnet/builder/bootstrap.go, ledger/complete/wal/checkpoint_v7_convert_test.go, ledger/factory/factory_test.go	The conversion API accepts a stream argument. The command exposes --stream. Production and test callers pass the new argument.
Temporary-file cleanup and comparison validation ledger/complete/wal/checkpoint_v6_writer.go, ledger/complete/wal/checkpoint_v6_writer_test.go, ledger/complete/wal/checkpoint_v6_test.go	Cleanup targets exact output files and narrower temporary-file patterns. File comparison handles distinct files, partial reads, length differences, and read errors.
Streaming conversion workflow ledger/complete/wal/checkpoint_v7_convert_stream.go	The streaming path converts subtrie files concurrently, converts the top-trie file, verifies CRC checksums, converts leaves, preserves ordering, and writes V7 output.
Streaming conversion verification ledger/complete/wal/checkpoint_v7_convert_stream_test.go	Tests compare streaming and in-memory output, verify root hashes and worker counts, cover validation and empty tries, preserve existing output on rejected reruns, detect corrupted input, and protect overlapping V6 input files.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant ConvertCheckpointV6ToV7
  participant StreamingConverter
  participant CheckpointFiles
  CLI->>ConvertCheckpointV6ToV7: pass --stream
  ConvertCheckpointV6ToV7->>StreamingConverter: convert subtrie and top-trie files
  StreamingConverter->>CheckpointFiles: write V7 parts and checksums
  ConvertCheckpointV6ToV7->>CheckpointFiles: write V7 header

Loading

Suggested reviewers: janezpodhostnik

Merge Risk: 🟡 Moderate · up to 27fea

Payloadless bootstrap can incur a substantial memory and time spike for mainnet-scale checkpoints. Switch it to streaming conversion before merge.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding a utility to convert V6 checkpoints to V7 checkpoints. It is concise and relevant, although the wording is somewhat abbreviated.
Docstring Coverage	✅ Passed	Docstring coverage is 89.13% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 46 functions across 11 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch leo/payloadless-util-convert-checkpoint

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[codecov-commenter]

Codecov Report

❌ Patch coverage is 57.49129% with 122 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...edger/complete/wal/checkpoint_v7_convert_stream.go	55.66%	49 Missing and 45 partials ⚠️
ledger/complete/wal/checkpoint_v7_convert.go	76.19%	7 Missing and 3 partials ⚠️
cmd/util/cmd/checkpoint-convert-v7/cmd.go	0.00%	8 Missing ⚠️
ledger/complete/wal/checkpoint_v6_writer.go	65.21%	4 Missing and 4 partials ⚠️
cmd/execution_builder.go	0.00%	1 Missing ⚠️
integration/localnet/builder/bootstrap.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!