feat: add --rdma=cpu|gpu flag, bump minio-go to v7.2.0#491
Conversation
c8eaf73 to
e05ce7a
Compare
|
#490 fixes lint/vuln |
Bumps github.com/minio/minio-go/v7 from v7.1.0 to v7.2.0 which adds
S3-over-RDMA dispatch via Options.EnableRDMA and {Put,Get}Object
Options.RDMABuffer.
--rdma is a string flag accepting "cpu" or "gpu"; empty disables.
The per-op buffer alloc is factored into pkg/bench/rdmabuf.go and
the GPU path is gated behind the rdma build tag — no separate cuda
tag.
go build . # default; --rdma=gpu surfaces a clear
# "requires -tags=rdma" error via the stub.
go build -tags=rdma . # both --rdma=cpu and --rdma=gpu work
# (libcudart + libminiocpp on the link path)
For PUT the generator output is staged into the host buffer (CPU
mode) or into a CUDA device buffer via cudaMemcpy H2D (GPU mode).
For GET the server RDMA-writes into a per-op CPU or GPU sink buffer;
Stat() carries the transferred byte count for verification.
Default HTTP PUT/GET paths are unchanged.
e05ce7a to
38e4c98
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe CLI now accepts CPU or GPU RDMA modes, propagates them through benchmark configuration, and explicitly enables MinIO RDMA. PUT and GET operations use CPU or CUDA-backed buffers, with build-tagged GPU support and non-RDMA stubs. ChangesRDMA benchmark support
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant CLI
participant getCommon
participant PutStart
participant GetStart
participant MinIO
participant CUDA
CLI->>getCommon: select rdma mode
getCommon->>PutStart: propagate RDMAMode
getCommon->>GetStart: propagate RDMAMode
PutStart->>CUDA: allocate and stage upload buffer
PutStart->>MinIO: submit PUT with RDMA buffer
GetStart->>CUDA: allocate download buffer
GetStart->>MinIO: submit GET with RDMA buffer
MinIO-->>PutStart: return upload result
MinIO-->>GetStart: return download result
PutStart->>CUDA: free upload buffer
GetStart->>CUDA: free download buffer
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cli/client.go`:
- Line 225: Validate the `--rdma` value during CLI parsing before constructing
the client, accepting only `cpu` or `gpu` (and the existing empty/default
value); reject any other value with a clear parse error. Update the relevant
flag validation in `cli/flags.go` and ensure `EnableRDMA` in the client setup is
derived only from validated input rather than any non-empty string.
In `@cli/flags.go`:
- Around line 349-352: Validate the rdmaMode value during CLI startup, alongside
the existing bench.RDMAModeGPU check. Accept only the supported RDMA modes
(including the default/disabled value), and call fatalIf with a clear
invalid-value message for anything else before setup proceeds; use the existing
rdmaMode variable and bench mode constants to locate the validation.
In `@pkg/bench/benchmark.go`:
- Around line 56-59: Update the RDMAMode documentation comment to use the same
build-tag requirement as cli/flags.go and put.go, replacing the inconsistent
“-tags=rdma,cuda” wording with “-tags=rdma” and preserving the libcudart
requirement if applicable.
In `@pkg/bench/get.go`:
- Around line 287-288: Handle the error returned by `o.Stat()` instead of
discarding it: check the error before accessing `info.Size`, and return or
report the stat failure through the surrounding benchmark error path. Update the
logic around `info` and `n` so a failed stat does not produce a misleading
size-mismatch error.
- Around line 287-297: In the object download cleanup sequence, call o.Close()
before freeRDMABuf(buf). Update the surrounding logic in the download operation
handling so the object’s teardown completes while the RDMA buffer remains valid,
then release the buffer and continue sending the operation result.
In `@pkg/bench/rdmabuf_rdma.go`:
- Around line 29-38: Replace raw CUDA error integers in allocRDMAGPU and the
corresponding error handling around lines 43-59 with human-readable messages
from cudaGetErrorString(rc); retain the numeric code only if useful, and ensure
the C error-string result is safely converted for fmt.Errorf.
- Around line 40-59: Update the doc comment for stageToGPU to accurately
describe its current behavior: it allocates a new host buffer of b.size for each
call, reads the full payload from src, and performs one host-to-device
cudaMemcpy. Remove claims about per-chunk processing, buffer reuse, and reduced
allocation overhead.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 0fb7afdb-cc21-4a59-9129-b2ab7d3c0f59
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (9)
cli/client.gocli/flags.gogo.modpkg/bench/benchmark.gopkg/bench/get.gopkg/bench/put.gopkg/bench/rdmabuf.gopkg/bench/rdmabuf_rdma.gopkg/bench/rdmabuf_stub.go
- get.go: surface o.Stat() error; close object before freeing RDMA buffer - flags.go: reject invalid --rdma values at startup - benchmark.go: correct build-tag doc (-tags=rdma) - rdmabuf_rdma.go: cudaGetErrorString for readable errors; fix stageToGPU doc
Fixes Applied SuccessfullyApplied 6 CodeRabbit feedback items across 4 files (commit Files modified:
Default build, |
klauspost
left a comment
There was a problem hiding this comment.
Approved with the toolchain bump. Other one is optional.
Summary
github.com/minio/minio-go/v7from v7.1.0 to v7.2.0 (addsS3-over-RDMA dispatch via
Options.EnableRDMAand{Put,Get}ObjectOptions.RDMABuffer).--rdmastring flag on the I/O commands (visible onwarp put/warp get/ anything inheritingioFlags).--rdma=cpuallocates a per-op host buffer.--rdma=gpuallocates a CUDA device buffer (requires-tags=rdmabuild, linkslibcudart).pkg/bench/rdmabuf.go. GPUhelpers are gated by the existing
rdmabuild tag(
rdmabuf_rdma.govsrdmabuf_stub.go); no separatecudatag.Motivation
Enables warp-driven validation of MinIO AIStor's GPU-Direct RDMA
S3 path end-to-end (both host-pinned and GPU memory), against a
cluster running the matching
libp2p_rdma.sointernode creditfix that landed alongside.
How to test
Types of changes
Checklist
go build ./...passes (default build path)go build -tags=rdma .documented (requireslibcudart+libminiocpp.soon the link path)go mod tidySummary by CodeRabbit
cpuvsgpubuffer handling when enabled.--rdmaflag (envAPP_RDMA) supportingcpu,gpu, or empty (disables RDMA), and wired it into benchmark configuration.