Skip to content

feat: add --rdma=cpu|gpu flag, bump minio-go to v7.2.0#491

Merged
harshavardhana merged 5 commits into
minio:masterfrom
harshavardhana:add-rdma-flag
Jul 10, 2026
Merged

feat: add --rdma=cpu|gpu flag, bump minio-go to v7.2.0#491
harshavardhana merged 5 commits into
minio:masterfrom
harshavardhana:add-rdma-flag

Conversation

@harshavardhana

@harshavardhana harshavardhana commented May 28, 2026

Copy link
Copy Markdown
Member

Summary

  • Bump github.com/minio/minio-go/v7 from v7.1.0 to v7.2.0 (adds
    S3-over-RDMA dispatch via Options.EnableRDMA and
    {Put,Get}ObjectOptions.RDMABuffer).
  • New --rdma string flag on the I/O commands (visible on
    warp put / warp get / anything inheriting ioFlags).
    • --rdma=cpu allocates a per-op host buffer.
    • --rdma=gpu allocates a CUDA device buffer (requires
      -tags=rdma build, links libcudart).
    • Empty (default) disables RDMA — current HTTP path is unchanged.
  • Per-op buffer management factored into pkg/bench/rdmabuf.go. GPU
    helpers are gated by the existing rdma build tag
    (rdmabuf_rdma.go vs rdmabuf_stub.go); no separate cuda tag.

Motivation

Enables warp-driven validation of MinIO AIStor's GPU-Direct RDMA
S3 path end-to-end (both host-pinned and GPU memory), against a
cluster running the matching libp2p_rdma.so internode credit
fix that landed alongside.

How to test

# HTTP (unchanged default behaviour)
go build .
./warp put --host h1:9000 --access-key K --secret-key S --obj.size 4MiB

# RDMA over host memory
go build .
./warp put --host h1:9000 ... --rdma=cpu      # errors with the stub
                                              # if not built with -tags=rdma
go build -tags=rdma .
./warp put --host h1:9000 ... --rdma=cpu

# GPU-Direct RDMA (needs CUDA + libminiocpp on link path)
go build -tags=rdma .
./warp put --host h1:9000 ... --rdma=gpu
./warp get --host h1:9000 ... --rdma=gpu

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • go build ./... passes (default build path)
  • go build -tags=rdma . documented (requires libcudart +
    libminiocpp.so on the link path)
  • No change to default (HTTP) PUT/GET paths
  • minio-go bumped to v7.2.0 via go mod tidy

Summary by CodeRabbit

  • New Features
    • Added optional RDMA dispatch for benchmark PUT and GET operations, with selectable cpu vs gpu buffer handling when enabled.
    • Introduced a --rdma flag (env APP_RDMA) supporting cpu, gpu, or empty (disables RDMA), and wired it into benchmark configuration.
    • Enables RDMA on the underlying client when the option is set.
  • Bug Fixes
    • Improved RDMA upload/download handling, including validation of transferred sizes and clearer error reporting.
  • Chores
    • Updated Go toolchain and refreshed dependency versions.

@harshavardhana harshavardhana marked this pull request as draft May 29, 2026 06:13
@klauspost

Copy link
Copy Markdown
Collaborator

#490 fixes lint/vuln

Bumps github.com/minio/minio-go/v7 from v7.1.0 to v7.2.0 which adds
S3-over-RDMA dispatch via Options.EnableRDMA and {Put,Get}Object
Options.RDMABuffer.

--rdma is a string flag accepting "cpu" or "gpu"; empty disables.
The per-op buffer alloc is factored into pkg/bench/rdmabuf.go and
the GPU path is gated behind the rdma build tag — no separate cuda
tag.

  go build .              # default; --rdma=gpu surfaces a clear
                          # "requires -tags=rdma" error via the stub.
  go build -tags=rdma .   # both --rdma=cpu and --rdma=gpu work
                          # (libcudart + libminiocpp on the link path)

For PUT the generator output is staged into the host buffer (CPU
mode) or into a CUDA device buffer via cudaMemcpy H2D (GPU mode).
For GET the server RDMA-writes into a per-op CPU or GPU sink buffer;
Stat() carries the transferred byte count for verification.

Default HTTP PUT/GET paths are unchanged.
Comment thread pkg/bench/rdmabuf_rdma.go
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d1342c47-9935-43be-9b70-887f2ac453a9

📥 Commits

Reviewing files that changed from the base of the PR and between cd6259b and 158e688.

📒 Files selected for processing (1)
  • go.mod

📝 Walkthrough

Walkthrough

The CLI now accepts CPU or GPU RDMA modes, propagates them through benchmark configuration, and explicitly enables MinIO RDMA. PUT and GET operations use CPU or CUDA-backed buffers, with build-tagged GPU support and non-RDMA stubs.

Changes

RDMA benchmark support

Layer / File(s) Summary
RDMA mode contracts and CLI wiring
pkg/bench/rdmabuf.go, pkg/bench/benchmark.go, cli/flags.go, cli/client.go, go.mod
Defines RDMA modes, adds the --rdma/APP_RDMA flag, validates GPU availability, propagates RDMAMode, enables MinIO RDMA, and updates dependencies.
RDMA buffer abstraction
pkg/bench/rdmabuf.go
Adds mode-specific buffer allocation, staging, cleanup, and unsupported-GPU handling.
CUDA and non-RDMA build variants
pkg/bench/rdmabuf_rdma.go, pkg/bench/rdmabuf_stub.go
Adds CUDA allocation, host-to-device staging, and freeing for rdma builds, with rejecting stubs otherwise.
RDMA PUT and GET execution
pkg/bench/put.go, pkg/bench/get.go
Uses RDMA buffers for non-POST uploads and downloads while preserving the existing non-RDMA paths.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant getCommon
  participant PutStart
  participant GetStart
  participant MinIO
  participant CUDA
  CLI->>getCommon: select rdma mode
  getCommon->>PutStart: propagate RDMAMode
  getCommon->>GetStart: propagate RDMAMode
  PutStart->>CUDA: allocate and stage upload buffer
  PutStart->>MinIO: submit PUT with RDMA buffer
  GetStart->>CUDA: allocate download buffer
  GetStart->>MinIO: submit GET with RDMA buffer
  MinIO-->>PutStart: return upload result
  MinIO-->>GetStart: return download result
  PutStart->>CUDA: free upload buffer
  GetStart->>CUDA: free download buffer
Loading

Poem

I’m a bunny with buffers, hopping GPU-high,
CPU ears twitch as the bytes zoom by.
PUTs fill the baskets, GETs bring them back,
CUDA clears the burrow’s track.
RDMA flags lead the way—
Hoppy benchmarks run today!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 77.78% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main changes: the new --rdma flag and the MinIO Go version bump.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@harshavardhana harshavardhana marked this pull request as ready for review July 10, 2026 08:24

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/client.go`:
- Line 225: Validate the `--rdma` value during CLI parsing before constructing
the client, accepting only `cpu` or `gpu` (and the existing empty/default
value); reject any other value with a clear parse error. Update the relevant
flag validation in `cli/flags.go` and ensure `EnableRDMA` in the client setup is
derived only from validated input rather than any non-empty string.

In `@cli/flags.go`:
- Around line 349-352: Validate the rdmaMode value during CLI startup, alongside
the existing bench.RDMAModeGPU check. Accept only the supported RDMA modes
(including the default/disabled value), and call fatalIf with a clear
invalid-value message for anything else before setup proceeds; use the existing
rdmaMode variable and bench mode constants to locate the validation.

In `@pkg/bench/benchmark.go`:
- Around line 56-59: Update the RDMAMode documentation comment to use the same
build-tag requirement as cli/flags.go and put.go, replacing the inconsistent
“-tags=rdma,cuda” wording with “-tags=rdma” and preserving the libcudart
requirement if applicable.

In `@pkg/bench/get.go`:
- Around line 287-288: Handle the error returned by `o.Stat()` instead of
discarding it: check the error before accessing `info.Size`, and return or
report the stat failure through the surrounding benchmark error path. Update the
logic around `info` and `n` so a failed stat does not produce a misleading
size-mismatch error.
- Around line 287-297: In the object download cleanup sequence, call o.Close()
before freeRDMABuf(buf). Update the surrounding logic in the download operation
handling so the object’s teardown completes while the RDMA buffer remains valid,
then release the buffer and continue sending the operation result.

In `@pkg/bench/rdmabuf_rdma.go`:
- Around line 29-38: Replace raw CUDA error integers in allocRDMAGPU and the
corresponding error handling around lines 43-59 with human-readable messages
from cudaGetErrorString(rc); retain the numeric code only if useful, and ensure
the C error-string result is safely converted for fmt.Errorf.
- Around line 40-59: Update the doc comment for stageToGPU to accurately
describe its current behavior: it allocates a new host buffer of b.size for each
call, reads the full payload from src, and performs one host-to-device
cudaMemcpy. Remove claims about per-chunk processing, buffer reuse, and reduced
allocation overhead.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0fb7afdb-cc21-4a59-9129-b2ab7d3c0f59

📥 Commits

Reviewing files that changed from the base of the PR and between c985a44 and 763f5b3.

⛔ Files ignored due to path filters (1)
  • go.sum is excluded by !**/*.sum
📒 Files selected for processing (9)
  • cli/client.go
  • cli/flags.go
  • go.mod
  • pkg/bench/benchmark.go
  • pkg/bench/get.go
  • pkg/bench/put.go
  • pkg/bench/rdmabuf.go
  • pkg/bench/rdmabuf_rdma.go
  • pkg/bench/rdmabuf_stub.go

Comment thread cli/client.go
Comment thread cli/flags.go
Comment thread pkg/bench/benchmark.go
Comment thread pkg/bench/get.go Outdated
Comment thread pkg/bench/get.go Outdated
Comment thread pkg/bench/rdmabuf_rdma.go
Comment thread pkg/bench/rdmabuf_rdma.go
- get.go: surface o.Stat() error; close object before freeing RDMA buffer
- flags.go: reject invalid --rdma values at startup
- benchmark.go: correct build-tag doc (-tags=rdma)
- rdmabuf_rdma.go: cudaGetErrorString for readable errors; fix stageToGPU doc
@harshavardhana

Copy link
Copy Markdown
Member Author

Fixes Applied Successfully

Applied 6 CodeRabbit feedback items across 4 files (commit cd6259b).

Files modified:

  • pkg/bench/get.go — surface o.Stat() error instead of discarding it; call o.Close() before freeRDMABuf() so teardown can't touch released memory
  • cli/flags.go — reject invalid --rdma values (only cpu/gpu/empty accepted) at startup instead of failing per-op later
  • pkg/bench/benchmark.go — correct build-tag doc (-tags=rdma, was -tags=rdma,cuda)
  • pkg/bench/rdmabuf_rdma.go — use cudaGetErrorString for human-readable CUDA errors; fix stageToGPU doc to match actual per-call buffer allocation

Default build, go vet, golangci-lint, and pkg/bench tests all pass.

@harshavardhana harshavardhana requested a review from klauspost July 10, 2026 08:37
@harshavardhana harshavardhana requested a review from Copilot July 10, 2026 09:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@klauspost klauspost left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with the toolchain bump. Other one is optional.

Comment thread pkg/bench/rdmabuf.go
Comment thread go.mod Outdated
@harshavardhana harshavardhana merged commit ed378c5 into minio:master Jul 10, 2026
8 checks passed
@harshavardhana harshavardhana deleted the add-rdma-flag branch July 10, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants