RPC: add -sm tensor - #26610
Conversation
github-actions
Bot
added
testing
|
confirmed working on metal(RDMA 2 x M3 Ultra with #26421), testing same model (ds4 MXFP4): it does however break with dspark applied because some ops it depends on appear to not be supported with TP (add across the split), so this is without any mtp. |
|
@ryan5rdx yeah I know it doesn't work with dspark, but what do you get when for just |
|
Sorry it is on top of the that |
command for reference, let me know happy to test an alternate config: and yup just confirmed - with |
|
@ryan5rdx try using two rpc servers, one on each machine and connect via
|
|
It might be worth to add I don't know if this would be observable with RDNA, but sometimes manually moving tensors instead of standard |
|
Yeah I messed up, I think #26490 should be okay to merge though |
Don't we want to fix the DSpark support first? |
|
The support is broken over RPC I think(i.e. this PR), not in general. But I can check |
Sorry - to clarify - is this RPC servers on two RDMA linked nodes, and then a llama-server instance on one of them(if so I suppose this will just connect to the localhost RPC server)? Sorry I've never run a llama-cli/server instance where it's not also doing compute ha |
|
@ryan5rdx yes, the llama-server is just a client of these two. Basically we're trying to activate the RPC<>RPC all-reduce path rather than the one you probably got (Metal<>RPC) |
Currently on master |
Tested with Qwen327B, 0.6B and ds4. no obvious error on rpc servers or llama-server, I see the tensors copy on the RPC nodes then nothing. final client logs, llama server web UI never comes up(hangs here): command: |
for the all-reduce to work - don't RPC nodes need direct(RDMA) connections to each other in addition to (at least)TCP to the client? with an A - B(just client) - C topology where A<>B and B<>C are RDMA links, but there is no A <> C link, can this work? (in MLX they achieve this with a mesh + rdma, but here RPC servers don't know about peers yet) |
rgerganov
left a comment
rgerganov
left a comment
There was a problem hiding this comment.
Please create a mermaid sequence diagram (similar to this one ) which describe how peers communicate when -sm tensor is being used, I am still trying to understand the new flows being added and that would be very helpful. In fact, I think this should be part of our dev documentation (feel free to create an .md file) so we can maintain this in the long term.
| return true; | ||
| } | ||
|
|
||
| // minor protocol version of each connected server, used to gate newer commands (comm collectives) |
There was a problem hiding this comment.
no need to do this, we don't care about backward compatibility and we prefer to keep the code simple; just bump the version to 6.0.0 and expect all peers to be running this version
|
@ryan5rdx I run this using the directly the RDMA interfaces, for example in my case it is |
in my case the RPC nodes cannot talk to each other via the Possible I'm misunderstanding here - or maybe a mac difference because it only supports p2p (TB)rdma without routing vs the spark? |
|
@ryan5rdx not sure, you can probably ask an LLM to debug? |
Ok yup confirmed - so if I use if I instead I use the LAN IPs - it still works becuase each worker can discover the other using the ip from the client, just super slow as expected (>>10s/token). However - if we want this to work on metal, with worker<>worker RDMA we need a mesh and to allow for passing the TB IP of other workers to each rpc server(because peer addresses are not derivable from the addr we get from the client in a TB p2p network). I'm not familiar with sparks but apparently it works because "all three nodes sit on one routable RDMA fabric" |
|
@rgerganov I added a mermaid diagram. Note that the Meta backend creates a lot of subgraphs so that's why graph caching is vital. |
rgerganov
left a comment
rgerganov
left a comment
There was a problem hiding this comment.
you can improve the diagram by making a clear separation between messages sent to the standard RPC port and messages sent to the new "comm" port
| return nullptr; | ||
| } | ||
| ggml_backend_rpc_context * rpc_ctx = (ggml_backend_rpc_context *) backends[i]->context; | ||
| // one rank per endpoint: a server processes its socket sequentially, so a second |
There was a problem hiding this comment.
"one rank per endpoint" -- why having this limitation? i can have an endpoint with two devices which communicate very fast (because they are on the same physical host)
There was a problem hiding this comment.
In that case you can just create two rpc servers, one for each device?
There was a problem hiding this comment.
On local MoE models with -cmoe weights are copied to GPU during PP for faster compute. Utilizing separate processes per device would break this optimization.
The example is using local, but same optimization should be possible on rpc with multiple backends.
Utilizing same process can also be useful in general for AllReduce - 2 servers 2 GPU each could merge local results lowering the amount of network hops needed.
| } | ||
|
|
||
| static void * ggml_backend_rpc_comm_init(ggml_backend_t * backends, size_t n_backends) { | ||
| if (n_backends != 2 || std::getenv("GGML_RPC_NO_COMM") != nullptr) { |
There was a problem hiding this comment.
is this going to work with more than 2 backends or it will require major redesign?
There was a problem hiding this comment.
it's going to fall-back to meta backend's all-reduce, which is slow but works
|
@am17an have you looked at implementing https://github.com/mk1-project/quickreduce/ QuickReduce seems pretty cool and it could probably be of huge benefit for RDMA performance. |
|
is this completedi am looking to test it lol please merge the sm tensore support for rpc ohh its seems like the sm tensore flag via rpc is only for spark not for normal gpus like my case rtx 3060 and 3080 via rpc seems like i have to stick with the sm layer flag |
|
I don't understand the need for If that is the case introducing it in two separate PRs would make it much easier to comprehend and understand the reasoning why using separate RPC server instead of running computation on the client directly can be beneficial. |
|
Yes, custom all-reduce is required for a reasonable speed. In fact, it is the most important operation in any tensor parallel implementation to be performant and not that it just works over multiple devices. You can use layer parallelism for that Later I plan to add NCCL support for the custom all-reduce which allows llama.cpp to reach near vLLM speeds over 2x sparks. |
hi is this possiple for general gpus ? |
|
I agree about all-reduce being important. For all-reduce on 3+ nodes direct connection between nodes (custom channel) will be important, but I'm not sure 3 nodes (client + 2 RPC) are needed in this case. |
|
The only reason it is done this way is because of how the code is designed. The Meta backend can identify two RPC devices and enable the custom all-reduce. You can run the client on the same node as the server (that's what I'm doing) |
|
Any particular reason to run it this way and force RPC<->RPC all-reduce, or it is just how you've got it set up/it's easier for debugging? I've run some test locally, spotted two issues:
llama-bench results# removed RDMA probed/activated logs for clarity
➜ llama.cpp git:(pr-26610) build/bin/llama-bench --rpc 192.168.4.1:50052,192.168.4.2:50052 --list-devices
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 8176 MiB):
Device 0: AMD Radeon RX 6600, gfx1032 (0x1032), VMM: no, Wave Size: 32, VRAM: 8176 MiB
Available devices:
ROCm0: AMD Radeon RX 6600 (8176 MiB, 7654 MiB free)
RPC0: 192.168.4.1:50052 (8176 MiB, 7656 MiB free)
RPC1: 192.168.4.2:50052 (63661 MiB, 63661 MiB free)
➜ llama.cpp git:(pr-26610) CUDA_VISIBLE_DEVICES=-1 build/bin/llama-bench -m /models/qwen3.6-35B-A3B/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -lm dio --rpc 192.168.4.1:50052,192.168.4.2:50052 -sm tensor -ts 1/2
ggml_cuda_init: failed to initialize ROCm: no ROCm-capable device is detected
| model | size | params | backend | ngl | sm | ts | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | ------------ | ---------: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium | 21.27 GiB | 35.51 B | ROCm,RPC | -1 | tensor | 1.00/2.00 | dio | pp512 | 67.35 ± 1.53 |
| qwen35moe 35B.A3B Q4_K - Medium | 21.27 GiB | 35.51 B | ROCm,RPC | -1 | tensor | 1.00/2.00 | dio | tg128 | 5.47 ± 0.06 |
build: 3a10a7709 (10435)
➜ llama.cpp git:(pr-26610) build/bin/llama-bench -m /models/qwen3.6-35B-A3B/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -lm dio --rpc 192.168.4.2:50052 -sm tensor -ts 1/2
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 8176 MiB):
Device 0: AMD Radeon RX 6600, gfx1032 (0x1032), VMM: no, Wave Size: 32, VRAM: 8176 MiB
| model | size | params | backend | ngl | sm | ts | lm | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | ------------ | ---------: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium | 21.27 GiB | 35.51 B | ROCm,RPC | -1 | tensor | 1.00/2.00 | dio | pp512 | 66.74 ± 1.07 |
| qwen35moe 35B.A3B Q4_K - Medium | 21.27 GiB | 35.51 B | ROCm,RPC | -1 | tensor | 1.00/2.00 | dio | tg128 | 8.18 ± 0.18 |
build: 3a10a7709 (10435)
➜ llama.cpp git:(pr-26610) |
|
@Kononnable direct server to server comm is the point of this PR. If you have a client in the middle the all-reduce goes through host memory which will be slower on RDMA capable connections. w.r.t your benchmark it is not equivalent to the other one, one of the ranks is local so only half the partials travel over the network. In any case you can continue using that if that's faster. What is the speed of your RDMA connection? EDIT: I also test the client<->RPC on my 2x sparks and it is about 60% slower. Secondly, the accept bug is real if two peers cannot reach each other. |
|
As for my benchmark - I tired to get the difference between client all-reduce [1 client + 1 rpc] vs new rpc all-reduce [1 client (not doing any work) + 2 rpc]. In both cases only two nodes are doing the work and are needed in all-reduce. My RDMA setup is very far from being optimal - 2x old ConnectX-3 cards(10GbE) connected through the chipsets (limiting some of the RDMA benefits), but latency seems to be fine (~1.63 usec). Anyway, if on a proper hardware setup (2xSpark) the new approach is faster then it's all good. I still don't understand why would the difference be so big, but it has probably something to do with my hardware setup or size of the data in all-reduce operation. ib_write_lat, ib_write_bw➜ llama.cpp git:(pr-26610) ib_write_lat 192.168.4.2
Disabling dynamic polling
---------------------------------------------------------------------------------------
RDMA_Write Latency Test
Dual-port : OFF Device : rocep4s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: OFF Lock-free : OFF
ibv_wr* API : OFF Using Enhanced Reorder : OFF
TX depth : 1
Mtu : 2048[B]
Link type : Ethernet
GID index : 2
Max inline data : 220[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
Failed to query device capabilities, ret=95
local address: LID 0000 QPN 0x0789 PSN 0xa96b6a RKey 0x9801014b VAddr 0x00557044b6d000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:01
remote address: LID 0000 QPN 0x04b3 PSN 0xfbde6c RKey 0x98010127 VAddr 0x0055d88c326000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:02
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
2 1000 1.59 2.84 1.62 1.63 0.00 1.76 2.84
---------------------------------------------------------------------------------------
➜ llama.cpp git:(pr-26610) ib_write_bw 192.168.4.2
---------------------------------------------------------------------------------------
RDMA_Write BW Test
Dual-port : OFF Device : rocep4s0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON Lock-free : OFF
ibv_wr* API : OFF Using Enhanced Reorder : OFF
TX depth : 128
CQ Moderation : 1
CQE Poll Batch : Dynamic
Mtu : 2048[B]
Link type : Ethernet
GID index : 2
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x078a PSN 0xd6b873 RKey 0xa001014b VAddr 0x007f65192ad000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:01
remote address: LID 0000 QPN 0x04b4 PSN 0xeba634 RKey 0xa0010127 VAddr 0x007f2a69f19000
GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:04:02
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[MiB/sec] BW average[MiB/sec] MsgRate[Mpps]
65536 5000 1138.40 1138.40 0.018214
--------------------------------------------------------------------------------------- |
|
@am17an Please rebase on latest |
|
it's rebased on latest master. The async events seem to have added a lot of latency in sync due to worker threads wake up. When using |
Assisted-by: OpenAI Codex
|
Here are my MTP bench results on 2x sparks after #28387, #28390 This is without using NCCL and any extra dsv4 specific optimizations (which in my tests bring the wall time < 40s) |
8 participants
Overview
Add RPC
-sm tensor. This is on 2x Sparks connected via RDMA.For RPC following changes are required:
all_reduceset_tensor_2d,get_tensor_2dLooking for feedback @ggerganov @rgerganov
Additional information
sequenceDiagram participant C as Client (RPC backend) box Server A (rank 0) participant A as A: rpc port participant Ac as A: comm port end box Server B (rank 1) participant Bc as B: comm port participant B as B: rpc port end Note over C,B: initialization C->>A: COMM_INIT (rank 0) C->>B: COMM_INIT (rank 1) Note over Ac: listen on comm port Bc-->>Ac: connect + caps negotiation (transport upgrade, e.g. RDMA) A->>C: response (ok) B->>C: response (ok) Note over C,B: for each subgraph (split at reduction boundaries) C->>A: GRAPH_COMPUTE (uid) [GRAPH_RECOMPUTE on reuse] C->>B: GRAPH_COMPUTE (uid) Note over A: compute subgraph async Note over B: compute subgraph async C->>A: COMM_ALLREDUCE (partial tensor) [fire and forget, no response] C->>B: COMM_ALLREDUCE (partial tensor) Note over A: sync pending graph<br/>cast F32→BF16 if ne ≥ 32768<br/>copy partial to send buffer Note over B: sync pending graph<br/>cast F32→BF16 if ne ≥ 32768<br/>copy partial to send buffer Ac-->>Bc: partial A (rank 0 sends first) Bc-->>Ac: partial B (rank 1 receives first) Note over A: upload partial B<br/>dst = dst + partial B (async ADD) Note over B: upload partial A<br/>dst = dst + partial A (async ADD) Note over C,B: read back the output C->>A: GET_TENSOR (output) Note over A: sync all backends A->>C: dataRequirements
Stack created with GitHub Stacks CLI • Give Feedback 💬