rpc: fix apple rdma error spew on teardown

[ryan5rdx]

Overview

fixes the apple RDMA teardown so that we don't spew the (cosmetic) errors about memory failing to unmap as raised here

Additional information

tested on 2x RDMA macs - no more memory unmap error spew

Also validated though some restart loops to smoke test any obvious leaks:
10 restarts of both worker + client on a single machine after a test promot - both exit/come up cleanly(with RDMA), RSS memory is flat

10 restarts of just the client (leaving the worker up) - same as above, no error, no RSS increase on the worker.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes - used dsv4f to diagnose/draft this change, validated on test setup of 2 RDMA enabled macs.

[ryan5rdx]

cc @rgerganov @ggerganov

[ggerganov]

This fixes the error on my system with 2 MacBooks.

Another problem that I observe is the following:

• On MacBook 1, start an RPC server:

# macbook 1
GGML_RPC_DEBUG=1 ./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

• On MacBook 2, start another RPC server:

# macbook 2
GGML_RPC_DEBUG=1 ./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

• On either mac, start a server using the 2 RPC devices:

# start a server on either macbook
./bin/llama-server -hf [model] --rpc [macbook1]:50052,[macbook2]:50052 --dev RPC0,RPC1

When I look at the logs, only one of the RPC servers shows that RDMA is enabled:

# macbook 1 (RDMA is on)
  transport      : TCP (RDMA auto-negotiate enabled)
Accepted client connection
[hello] version: 6.0.0
RDMA(Apple/UC) probed: dev=rdma_en6 port=1 gid=1 qpn=2320 lid=2 mtu=4096 ring=16 x 128 KiB
RDMA(Apple/UC) activated: qpn=2320->2320 mtu=4096 rx_depth=16
Client connection closed
[~impl] closing socket 8
Accepted client connection

...

# macbook 2 (RDMA is off)
  transport      : TCP (RDMA auto-negotiate enabled)
Accepted client connection
[hello] version: 6.0.0
recv returned 0 (peer closed?)
Client connection closed
[~impl] closing socket 8
Accepted client connection

@ryan5rdx Do you observe the same thing on your system?

[ryan5rdx]

This fixes the error on my system with 2 MacBooks.

Another problem that I observe is the following:

* On MacBook 1, start an RPC server:

# macbook 1
GGML_RPC_DEBUG=1 ./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

* On MacBook 2, start another RPC server:

# macbook 2
GGML_RPC_DEBUG=1 ./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

* On either mac, start a server using the 2 RPC devices:

# start a server on either macbook
./bin/llama-server -hf [model] --rpc [macbook1]:50052,[macbook2]:50052 --dev RPC0,RPC1

When I look at the logs, only one of the RPC servers shows that RDMA is enabled:

# macbook 1 (RDMA is on)
  transport      : TCP (RDMA auto-negotiate enabled)
Accepted client connection
[hello] version: 6.0.0
RDMA(Apple/UC) probed: dev=rdma_en6 port=1 gid=1 qpn=2320 lid=2 mtu=4096 ring=16 x 128 KiB
RDMA(Apple/UC) activated: qpn=2320->2320 mtu=4096 rx_depth=16
Client connection closed
[~impl] closing socket 8
Accepted client connection

...

# macbook 2 (RDMA is off)
  transport      : TCP (RDMA auto-negotiate enabled)
Accepted client connection
[hello] version: 6.0.0
recv returned 0 (peer closed?)
Client connection closed
[~impl] closing socket 8
Accepted client connection

@ryan5rdx Do you observe the same thing on your system?

yeah this is what I see - this is the issue that I raised here in #26610 .

As is - because apple RDMA is point to point, and there's no loopback path (you can't connect to "localhost" via rdma) the client -> local RPC server link will always be TCP.

Once #26610 lands - depending on your setup:

1 Client + 1 RPC server with client doing compute:
the client <> rpc link uses RDMA, as expected (this is generally what I've been using)

1 Client + 2 RPC servers with the client not doing compute, and connecting to one client via local host
the client <> local rpc link uses TCP, but if you pass the correct IPs, the worker <> worker link that #26610 adds, does use RDMA. (And the client <> remote worker link will also use RDMA).

I'm still thinking about how to improve this since it's unintuitive as is.

On linux the fabric IPs are routeable (you can just use the same IP the client passes to discover other workers over RDMA) but on mac they're not, so we will likely have to pass a list of IPs to each worker so that they can correctly discover the other workers over RDMA.

but this will come later because currently -sm tensor currently only is supporting 2 nodes anyway.

[ggerganov]

@ryan5rdx Ok, let me know if you have any ideas. I now have an M5 Max cluster of 2 MacBooks with RDMA connection. Without -sm tensor and 1 Client + 1 RPC server it works correctly. With -sm tensor (#26610), I tried all possible IP combinations with 1 Client + 2 RPC servers and the RPC logic either:

• deadlocks upon trying to establish RDMA route
• fails to create pairwaise communicator