Skip to content

rpc: support apple RDMA as an RPC transport #26421

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ggerganov merged 4 commits into from
Aug 25, 2026
Merged

rpc: support apple RDMA as an RPC transport #26421

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
28bd8e8
rpc: support apple RDMA as an RPC transport
Aug 1, 2026
1f625d4
remove set_tensor micro optimization, rpc socket pinning per CR
ryan5rdx Aug 5, 2026
b3e5281
remove transparent reconnect
ryan5rdx Aug 6, 2026
0de310d
trigger apple builds on RPC changes
ryan5rdx Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/build-apple.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ on:
types: [opened, synchronize, reopened]
paths: [
'.github/workflows/build-apple.yml',
'ggml/src/ggml-metal/**'
'ggml/src/ggml-metal/**',
'ggml/src/ggml-rpc/**'
]

concurrency:
Expand Down
28 changes: 20 additions & 8 deletions ggml/src/ggml-rpc/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,18 @@ if (WIN32)
target_link_libraries(ggml-rpc PRIVATE ws2_32)
endif()

# RDMA auto-detection (Linux only, requires libibverbs)
if (NOT WIN32 AND NOT APPLE)
find_library(IBVERBS_LIB ibverbs)
if (IBVERBS_LIB)
# RDMA auto-detection: Linux RoCE/IB via libibverbs, Apple RDMA-over-Thunderbolt via librdma
if (APPLE)
set(RDMA_LIB_NAME rdma)
set(RDMA_DESC "Apple RDMA-over-Thunderbolt, UC")
elseif (NOT WIN32)
set(RDMA_LIB_NAME ibverbs)
set(RDMA_DESC "auto-detected")
endif()

if (RDMA_LIB_NAME)
find_library(RDMA_LIB ${RDMA_LIB_NAME})
if (RDMA_LIB)
option(GGML_RPC_RDMA "ggml: enable RDMA transport for RPC" ON)
else()
option(GGML_RPC_RDMA "ggml: enable RDMA transport for RPC" OFF)
Expand All @@ -22,12 +30,16 @@ else()
endif()

if (GGML_RPC_RDMA)
if (NOT IBVERBS_LIB)
find_library(IBVERBS_LIB ibverbs REQUIRED)
if (NOT RDMA_LIB)
find_library(RDMA_LIB ${RDMA_LIB_NAME} REQUIRED)
endif()
target_compile_definitions(ggml-rpc PRIVATE GGML_RPC_RDMA)
target_link_libraries(ggml-rpc PRIVATE ${IBVERBS_LIB})
message(STATUS " RDMA transport enabled (auto-detected)")
target_link_libraries(ggml-rpc PRIVATE ${RDMA_LIB})
if (APPLE)
target_compile_definitions(ggml-rpc PRIVATE GGML_RPC_RDMA_APPLE)
target_sources(ggml-rpc PRIVATE transport-apple.cpp)
endif()
message(STATUS " RDMA transport enabled (${RDMA_DESC})")
else()
message(STATUS " RDMA transport disabled")
endif()
7 changes: 5 additions & 2 deletions ggml/src/ggml-rpc/ggml-rpc.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,10 @@ static bool send_msg(socket_ptr sock, const void * msg, size_t msg_size) {
if (!sock->send_data(&msg_size, sizeof(msg_size))) {
return false;
}
return sock->send_data(msg, msg_size);
if (!sock->send_data(msg, msg_size)) {
return false;
}
return sock->flush();
}

static bool recv_msg(socket_ptr sock, void * msg, size_t msg_size) {
Expand Down Expand Up @@ -308,7 +311,7 @@ static bool send_rpc_cmd(socket_ptr sock, enum rpc_cmd cmd, const void * input,
if (!sock->send_data(input, input_size)) {
return false;
}
return true;
return sock->flush();

@rgerganov rgerganov Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you try to measure the performance impact of having an explicit flush()? i.e. flushing on every send vs manual flushing like here

@ryan5rdx ryan5rdx Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

measured - without this manual flush the coalesing breaks somewhat (we end up sending ~two more empty TB frames per RPC command).

flushing on every send: 21.17 t/s tg 2048
manual flush: 22.41 t/s tg 2048

prefill appears to be a little slower, but within ~margin of error 290 vs ~282 t/s pp2048

}

// RPC request : | rpc_cmd (1 byte) | request_size (8 bytes) | request_data (request_size bytes) |
Expand Down
Loading
Loading