[WIP-2] feat(convertor): add --direct-upload flag for discoball two-phase upload - #27
Draft
yoon-park-rl wants to merge 15 commits into
Draft
[WIP-2] feat(convertor): add --direct-upload flag for discoball two-phase upload#27yoon-park-rl wants to merge 15 commits into
yoon-park-rl wants to merge 15 commits into
Conversation
Adds a new --output-dir flag that writes converted overlaybd artifacts to a local directory instead of pushing to a registry. This enables the blueprint-uploader tool to push blobs directly to S3 via discoball's prepare/confirm API, bypassing the convertor's registry proxy path. Output directory layout: <dir>/manifest.json - OCI manifest JSON <dir>/config.json - OCI image config JSON <dir>/config.digest - "sha256:<hex>" of config.json <dir>/blobs/<sha256-hex> - one file per layer blob Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rm API Adds --direct-upload and --registry-url flags to convertor. When set, convertor converts OCI layers → overlaybd locally (via FileBasedResolver) then uploads blobs directly to S3 via discoball's two-phase prepare/confirm API, bypassing the normal registry push path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches Rust upload_one_part: 100ms, 200ms, 400ms delays before each retry. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CRC-64/NVME requires init=0xFFFFFFFFFFFFFFFF and xorout=0xFFFFFFFFFFFFFFFF. The previous code used crc64.New(table) which starts with init=0 and no final XOR, producing a wrong checksum that S3 rejects with BadDigest. Fix by using crc64.Update starting from 0xFFFFFFFFFFFFFFFF and XORing the result with 0xFFFFFFFFFFFFFFFF before encoding, matching the Rust crc64fast_nvme::Digest::new() behavior.
…eTable Go's crc64.MakeTable expects the polynomial in reflected (bit-reversed) form, like crc64.ECMA = 0xC96C5795D7870F42 (the reflection of 0x42F0E1EBA9EA3693). The previous code passed 0xAD93D23594C935A9 (the normal polynomial from the NVMe spec) directly, producing a wrong lookup table. The correct reflected form is 0x95AC9329AC4BC9B5 = bit_reverse_64(0xAD93D23594C935A9).
…C9B5
Go's crc64 package builds reflected CRC tables and applies init/xorout
internally (same as the NVMe spec: init=0xFFFFFFFFFFFFFFFF, xorout=0xFFFFFFFFFFFFFFFF).
crc64.New/Checksum are correct as-is; the only bug was the polynomial.
The original 0xAD93D23594C935A9 is the normal form; the previous "fix"
0x95AC9329AC4BC9B5 is its bit-reversal but still wrong. The correct
reflected polynomial verified against the CRC RevEng check value
("123456789" → 0xAE8B14860A799888) is 0x9A6C9329AC4BC9B5.
Also reverts the incorrect init/xorout manual application from the
previous two commits, which was double-applying the complement.
Adds unit tests mirroring the Rust tests in container-registry/src/direct_upload.rs: - manifest field base64-encodes in prepare request - confirm request omits tag when nil, includes it when set - prepare response parses existing and pending blobs correctly - parseImageRef handles registry/repository/tag splitting Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--output-dir was an intermediate step toward direct upload — it wrote converted blobs to disk so a separate blueprint-uploader tool could upload them. That tool was removed in favor of --direct-upload, which handles the full convert-and-upload pipeline in one step. Nothing calls --output-dir in the current build flow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The flag belongs to --export-tar, not --output-dir. It was accidentally dropped when --output-dir was removed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoon-park-rl
marked this pull request as draft
August 5, 2026 17:10
…prepare Previously DirectUploadFromStore ran after Build() completed, serializing conversion (21s) and upload (23s) for a total of ~44s. This change overlaps them: each layer is uploaded to S3 immediately after it finishes converting, while the next layer is still being converted. Changes: - Add DirectUploadPipeline that calls a new per-blob prepare API (no manifest needed) and uploads each layer as it finishes, collecting confirm tokens. - Wire Pipeline into BuilderOptions and builderEngineBase; overlaybd_builder calls pipeline.UploadBlob at the end of UploadLayer. - main.go creates the pipeline before Build() and calls DirectUploadWithPipeline after, which handles remaining blobs (config, base layer) and confirms with all pre-collected tokens. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
yoon-park-rl
force-pushed
the
yoon/convertor-output-dir-2
branch
from
August 5, 2026 18:10
5732512 to
c74abb1
Compare
Replaced by DirectUploadWithPipeline which pipelines layer uploads with conversion. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… path Replace prepareUpload(manifest) with prepareSingleBlob(config) in DirectUploadWithPipeline. The full manifest prepare was opening S3 multipart sessions for all blobs including layers already uploaded by the pipeline, wasting ~5 S3 API calls. Now only the config blob gets a new session. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #
Please check the following list: