feat(convertor): add --direct-upload flag for discoball two-phase upload - #26
Open
yoon-park-rl wants to merge 14 commits into
Open
feat(convertor): add --direct-upload flag for discoball two-phase upload#26yoon-park-rl wants to merge 14 commits into
yoon-park-rl wants to merge 14 commits into
Conversation
Adds a new --output-dir flag that writes converted overlaybd artifacts to a local directory instead of pushing to a registry. This enables the blueprint-uploader tool to push blobs directly to S3 via discoball's prepare/confirm API, bypassing the convertor's registry proxy path. Output directory layout: <dir>/manifest.json - OCI manifest JSON <dir>/config.json - OCI image config JSON <dir>/config.digest - "sha256:<hex>" of config.json <dir>/blobs/<sha256-hex> - one file per layer blob Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoon-park-rl
marked this pull request as draft
July 29, 2026 18:19
…rm API Adds --direct-upload and --registry-url flags to convertor. When set, convertor converts OCI layers → overlaybd locally (via FileBasedResolver) then uploads blobs directly to S3 via discoball's two-phase prepare/confirm API, bypassing the normal registry push path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches Rust upload_one_part: 100ms, 200ms, 400ms delays before each retry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CRC-64/NVME requires init=0xFFFFFFFFFFFFFFFF and xorout=0xFFFFFFFFFFFFFFFF. The previous code used crc64.New(table) which starts with init=0 and no final XOR, producing a wrong checksum that S3 rejects with BadDigest. Fix by using crc64.Update starting from 0xFFFFFFFFFFFFFFFF and XORing the result with 0xFFFFFFFFFFFFFFFF before encoding, matching the Rust crc64fast_nvme::Digest::new() behavior.
…eTable Go's crc64.MakeTable expects the polynomial in reflected (bit-reversed) form, like crc64.ECMA = 0xC96C5795D7870F42 (the reflection of 0x42F0E1EBA9EA3693). The previous code passed 0xAD93D23594C935A9 (the normal polynomial from the NVMe spec) directly, producing a wrong lookup table. The correct reflected form is 0x95AC9329AC4BC9B5 = bit_reverse_64(0xAD93D23594C935A9).
…C9B5
Go's crc64 package builds reflected CRC tables and applies init/xorout
internally (same as the NVMe spec: init=0xFFFFFFFFFFFFFFFF, xorout=0xFFFFFFFFFFFFFFFF).
crc64.New/Checksum are correct as-is; the only bug was the polynomial.
The original 0xAD93D23594C935A9 is the normal form; the previous "fix"
0x95AC9329AC4BC9B5 is its bit-reversal but still wrong. The correct
reflected polynomial verified against the CRC RevEng check value
("123456789" → 0xAE8B14860A799888) is 0x9A6C9329AC4BC9B5.
Also reverts the incorrect init/xorout manual application from the
previous two commits, which was double-applying the complement.
Adds unit tests mirroring the Rust tests in container-registry/src/direct_upload.rs: - manifest field base64-encodes in prepare request - confirm request omits tag when nil, includes it when set - prepare response parses existing and pending blobs correctly - parseImageRef handles registry/repository/tag splitting Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoon-park-rl
changed the title
--output-dir was an intermediate step toward direct upload — it wrote converted blobs to disk so a separate blueprint-uploader tool could upload them. That tool was removed in favor of --direct-upload, which handles the full convert-and-upload pipeline in one step. Nothing calls --output-dir in the current build flow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The flag belongs to --export-tar, not --output-dir. It was accidentally dropped when --output-dir was removed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoon-park-rl
marked this pull request as ready for review
July 31, 2026 18:25
yoon-park-rl
changed the title
Previously blobs were uploaded sequentially (one blob fully done before the next starts). Since blobs are independent, switch to errgroup so all missing blobs upload in parallel. Pre-allocate confirmBlobs by index so each goroutine writes to a unique slot without a mutex. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dFromStore" This reverts commit f30069a.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a --direct-upload flag to the convertor binary. When set (with --import-tar and -r), the convertor converts OCI layers → overlaybd locally, then uploads blobs directly to S3 via discoball's two-phase prepare/confirm API instead of routing blobs through discoball as a registry proxy.
This is the blueprint build counterpart to the snapshot direct upload path already implemented in overlaybd-service (runloop java/rust/container-registry/src/direct_upload.rs). Both implement the same wire protocol against the same discoball endpoints; this PR brings the same capability to the blueprint build path via the Go convertor binary.
This is used by the blueprint-builder-overlaybd image for overlaybd blueprint builds. Cutting discoball out of the data plane reduces latency and load for large blueprint builds.
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Please check the following list: