feat: GPU interconnect multinode support#400
Open
Zblocker64 wants to merge 8 commits into
Open
Conversation
Wire RDMA / RoCE HCAs end-to-end so multi-node tensor-parallel GPU workloads (NCCL over IB) can land on Akash providers. Provider pieces (AKT-407..AKT-412): - Inventory operator discovers RDMA capacity, NCCL HCA prefix, and fabric (infiniband|roce) from `/sys/class/infiniband` and the kubelet `rdma/rdma_shared_device_*` extended resource. - Inventory client adds placement helpers: `ResourceRequiresRDMA`, `PlacementRequiredFabric`, `PlacementRequiresRDMA`. - Reservation `Adjust` extracts the placement-level fabric pin once and calls a new `tryAdjustRDMA` per (node, replica): 1:1 HCA:GPU invariant, fabric-pin match, capacity check, stamps `SchedulerParams.Resources.RDMA`. - Manifest CRD gains `SchedulerResourceRDMA` and `Service.RDMAGroup`, with matching schema additions so the API server does not silently prune the new fields. - Workload builder injects the RDMA extended resource (request==limit for kubelet device plugins) and NCCL env vars (`NCCL_IB_DISABLE=0`, `NCCL_IB_HCA=<prefix>`) — both deferring to SDL overrides. - Workload builder applies per-`rdma_group` pod anti-affinity (hostname topology) so peers in one group land on distinct nodes. Consumes the chain-sdk RDMA tags published from feat/rdma-multinode: `pkg.akt.dev/go v0.2.13-rc1`, `pkg.akt.dev/go/sdl v0.2.2-rc1`. Helm chart bundling, runbook, reference SDLs, and the preflight script (P-8..P-11) are out of scope for this PR.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds end-to-end GPU interconnect support for InfiniBand and RoCE fabrics. Changes span CRD schema extensions, a new sysfs-based HCA discovery module exposed as a REST endpoint, node inventory accounting for interconnect resources, scheduler placement helpers with fabric-pin and group anti-colocation enforcement, and Kubernetes workload rendering with NCCL env injection and pod anti-affinity. ChangesGPU Interconnect (InfiniBand/RoCE) Support
Sequence DiagramsequenceDiagram
rect rgba(70, 130, 180, 0.5)
Note over Scheduler: Scheduling path
participant Adjust
participant tryAdjustInterconnect
participant NodeCapabilities
Adjust->>Adjust: PlacementRequiredFabric + build requiresInterconnect[]
Adjust->>tryAdjustInterconnect: per-node, requiredFabric + requiresInterconnect[i]
tryAdjustInterconnect->>NodeCapabilities: check InterconnectFabric + InterconnectResourceName + capacity
NodeCapabilities-->>tryAdjustInterconnect: accept/reject
tryAdjustInterconnect-->>Adjust: stamp SchedulerParams.Resources.Interconnect
Adjust->>Adjust: record groupClaims[interconnectGroup] = nodeIndex
end
rect rgba(34, 139, 34, 0.5)
Note over NodeDiscovery: Node inventory path
participant NodeDiscovery
participant DiscoveryPod
participant psutil
NodeDiscovery->>DiscoveryPod: dpReqIB via REST proxy
DiscoveryPod->>psutil: GET /infiniband (host-sys mount)
psutil-->>DiscoveryPod: IBDiscovery{NCCLHCAPrefixes, Fabric}
DiscoveryPod-->>NodeDiscovery: IBDiscovery
NodeDiscovery->>NodeDiscovery: initNodeInfo seeds capabilities
NodeDiscovery->>NodeDiscovery: updateNodeInfo resolves InterconnectResourceName
end
rect rgba(180, 100, 40, 0.5)
Note over workloadBuilder: Workload rendering path
participant workloadBuilder
participant podTemplate
workloadBuilder->>podTemplate: add RDMA device-plugin resource requests/limits
workloadBuilder->>podTemplate: set NCCL_IB_DISABLE=0 + NCCL_IB_HCA
workloadBuilder->>podTemplate: set NCCL_IB_GID_INDEX=3 if fabric=roce
workloadBuilder->>podTemplate: stamp AkashInterconnectGroupLabelName + PodAntiAffinity
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The hardware-discovery DaemonSet hung in ContainerCreating on every non-RDMA node because HostPathDirectoryOrCreate against /sys/class/infiniband asked the kubelet to mkdir under sysfs, which is read-only and rejects the create. CI crd-e2e caught this on the kind control-plane node. Mount the always-present /sys/class parent (HostPathDirectory) and let the /infiniband handler walk /host/sys/class/infiniband on demand. The handler already tolerates ENOENT and returns a zero IBDiscovery so non-RDMA hosts continue to advertise no fabric/capacity.
Four discrete fixes uncovered while driving an actual 405B multi-node RDMA deployment end-to-end on a 2x A100/IB cluster: - inventory.go: tryAdjustRDMA runs before tryAdjustGPU. tryAdjustGPU replaces res.GPU.Attributes with a synthesized vendor-only entry, clobbering the per-service `rdma: true` opt-in. ResourceRequiresRDMA was then returning false on every node, so the reservation succeeded as a non-RDMA bid: pods landed without the RDMA HCA resource and without NCCL_IB_* env injection. Reorder fixes both. - metrics.go (ParseGPUAttributes): skip the `rdma` and `rdma_group` GPU attribute keys. The chain SDK accepts them as per-service hints, but they are 1-token keys and the strict "must start with vendor/" parser was erroring on the first one, which propagated as "insufficient capacity" with no useful log. - node-discovery.go: mount /sys (not /sys/class) into the hardware-discovery DaemonSet. /sys/class/infiniband/<dev> is a sysfs symlink into /sys/devices/...; mounting only the parent let the directory listing succeed but every read of ports/link_layer chased out of the mount and the fabric came back empty. Sysfs is read-only so the wider mount is safe. - node-discovery.go: PCI-bus fallback in parseGPUInfo. Headless A100/ H100 nodes (NVIDIA driver lives in the GPU Operator daemonset, not on the host kernel) expose only the BMC VGA under /sys/class/drm, so ghw's GPU enumeration finds 0 compute cards. New helper gpuInfoFromPCIDevice filters class 03/02 (3D controller) and matches against the same vendor/model registry, so the inventory picks up all 8 compute cards on each node. Adds dpReqPCI + queryPCI; reuses the existing /pci endpoint psutil already exposed. End-to-end verified: bid accepts, lease creates, manifest sends, head and worker land on distinct nodes via per-rdma_group pod anti-affinity, each container gets `rdma/rdma_shared_device_ib: 8` requests+limits and `NCCL_IB_DISABLE=0` + `NCCL_IB_HCA=mlx5` injected.
chalabi2
requested changes
Jun 3, 2026
Three discrete fixes addressing chalabi2's review:
- node-discovery.go (gpuInfoFromPCIDevice): drop the PCI class/subclass
filter. Datacenter cards (A100/H100) enumerate as 03/02 ("3D
controller"), but consumer cards (RTX 4090/3090, A6000) enumerate as
03/00 ("VGA compatible controller") — the same class as the BMC VGA.
The vendor+product registry already distinguishes them: Matrox BMC
(vendor 102b) is not in the akash GPU vendor registry, while NVIDIA
and AMD compute + consumer cards both are. The class filter was a
regression for consumer-card detection.
- inventory.go (Adjust + tryAdjust + tryAdjustRDMA): hoist the
ResourceRequiresRDMA check into Adjust over the pristine
origResources, before any tryAdjustGPU pass has mutated the attribute
slice. Thread the per-resource `requiresRDMA bool` through tryAdjust
into tryAdjustRDMA. Previously the prior commit's reorder of
RDMA-before-GPU caught only replica #1 of each resource: for count > 1
the second-and-later adjusted Dup was taken from the
already-clobbered slice, so tryAdjustRDMA went no-op, SchedulerParams
stamping diverged from replica #1, and Adjust rejected with
ErrGroupResourceMismatch. Multi-replica RDMA services now bid
correctly. Regression-pinned by
TestTryAdjustRDMA_RequiredHonoredWithoutAttributes.
- workload.go (PodAntiAffinity comment): flag the bid-side gap — the
chain SDK does not serialize `rdma_group` into the on-chain Order, so
tryAdjust cannot enforce per-group node separation at fit time. Two
known misalignment cases (multi-replica single-service exceeding
distinct-node count; multi-service group greedy-fit onto one node)
result in pods stuck Pending. Tenants must size groups within the
provider's distinct-node capacity until the bid path is made
group-aware. Tracked in AKT-443.
The workload builder applies hard PodAntiAffinity on akash.network/rdma-group=<value> so RDMA peers land on distinct nodes. Before this change the bid engine had no signal at fit time: it accepted orders the kube scheduler then couldn't satisfy, leaving pods stuck Pending forever (count: N replicas of one service in one group on fewer than N nodes; multi-service group greedily co-located on one large node). The chain-SDK rc3 (go/v0.2.13-rc3 + go/sdl/v0.2.2-rc3) now serializes rdma_group as an on-chain GPU attribute alongside rdma=true. This commit teaches the provider's reservation Adjust to read it and enforce per-group node separation symmetric with the workload builder's anti-affinity. - rdma.go: new ResourceRDMAGroup helper + AttributeGPURDMAGroupKey constant. Symmetric with ResourceRequiresRDMA. - inventory.go: per-resource rdmaGroup []string read from pristine origResources (same defense-in-depth as requiresRDMA — tryAdjustGPU clobbers Attributes mid-bid). Per-bid groupClaims map[string]map[int]bool threaded through tryAdjust. Early reject when a node is already claimed by the same group; register on successful fit only after every other gate has passed so partial-fit attempts never poison the map. - workload.go: replaced the "KNOWN GAP" comment block with a note that bid + scheduler now enforce the same constraint via two paths. - go.mod: bump pkg.akt.dev/go and pkg.akt.dev/go/sdl to rc3. - rdma_group_adjust_test.go: 6 end-to-end Adjust tests pinning the success and reject paths (multi-service-one-each on 2 vs 1 node, single-service-count-4 on 4 vs 3 nodes, distinct-groups independent, empty-group regression). Existing TestTryAdjustRDMA suite + builder + apis tests stay green.
chalabi2
previously approved these changes
Jun 9, 2026
Top-to-bottom rename to consume chain-sdk rc4 (go/v0.2.13-rc4 +
go/sdl/v0.2.2-rc4). Tenants now declare gpu.attributes.interconnect:
true (fabric-agnostic); the provider picks IB vs RoCE internally.
go.mod:
- Bump pkg.akt.dev/go -> v0.2.13-rc4, sdl -> v0.2.2-rc4
- Add replace directives forcing rc4 (pkg.akt.dev/node/v2 v2.1.0
otherwise transitively pulls v0.2.14 stable which predates the rename)
Provider inventory client (formerly rdma.go -> interconnect.go):
- AttributeRDMAPlacement -> AttributeInterconnectPlacement = "capabilities/gpu-interconnect"
- AttributeGPURDMAKey -> AttributeGPUInterconnectKey = "interconnect"
- AttributeGPURDMAGroupKey -> AttributeGPUInterconnectGroupKey = "interconnect_group"
- AttributeRDMAFabricInfiniBand/RoCE -> AttributeInterconnectFabricInfiniBand/RoCE
- ResourceRequiresRDMA -> ResourceRequiresInterconnect
- ResourceRDMAGroup -> ResourceInterconnectGroup
- PlacementRequiresRDMA -> PlacementRequiresInterconnect
Reservation Adjust:
- tryAdjustRDMA -> tryAdjustInterconnect
- All requiresRDMA/rdmaGroup locals + groupClaims keyed on interconnect_group
Workload builder:
- AkashRDMAGroupLabelName -> AkashInterconnectGroupLabelName ("akash.network/interconnect-group")
- sparamsRDMA -> sparamsInterconnect
CRD manifest types + schema:
- SchedulerResourceRDMA -> SchedulerResourceInterconnect (json tag "interconnect,omitempty")
- SchedulerResources.RDMA -> Interconnect
- ManifestService.RDMAGroup -> InterconnectGroup (json tag "interconnect_group,omitempty")
- crd.yaml: rdma -> interconnect, rdma_group -> interconnect_group
- Deepcopy regenerated manually
Inventory operator (consumes chain-sdk inventory v1 renames):
- node.Resources.RDMA -> node.Resources.GPUInterconnect
- node.Capabilities.RDMAResourceName -> InterconnectResourceName
- node.Capabilities.RDMAFabric -> InterconnectFabric
- All matchRDMAResourceName + per-pod allocation tracking renamed
Tests:
- 5 file renames: rdma_*.go -> interconnect_*.go
- All identifiers + attribute string literals renamed
- 6 group-aware Adjust tests + 7 tryAdjustInterconnect tests + 5 workload
builder tests all pass
Provider attribute on-chain advertising:
- capabilities/rdma -> capabilities/gpu-interconnect
- capabilities/rdma/fabric/* -> capabilities/gpu-interconnect/fabric/*
(informational; SDL no longer pins fabric)
Unavoidable external references retained:
- rdma/rdma_shared_device_* (Mellanox/NVIDIA device-plugin resource name)
- /sys/class/infiniband (kernel sysfs path for IB + RoCE HCAs)
- NCCL_IB_DISABLE/NCCL_IB_HCA (NCCL upstream env vars)
- "infiniband"/"roce" fabric value strings (kernel link_layer values)
Zblocker64
changed the title
Zblocker64
changed the title
…ource match Bumps pkg.akt.dev/go + sdl to rc5 and adapts the provider to the nested `interconnect:` SDL surface from chain-sdk's AKT-492. Folds in AKT-493 (configurable extended-resource patterns) and AKT-494 (NCCL_IB_GID_INDEX auto-injection on RoCE) because all three touch the same code paths. AKT-492 — SDL surface refactor - Retire `AttributeGPUInterconnectKey = "interconnect"`; the chain SDK no longer emits the standalone marker. - Rename the group attribute key: `interconnect_group` → `interconnect/group` (slash-path matches `capabilities/gpu-interconnect/...` convention). - Rewrite `ResourceRequiresInterconnect` to gate on `interconnect/group` presence (any value, including the reserved literal `auto`). - `NCCLHCAPrefix string` → `NCCLHCAPrefixes []string` end-to-end: proto-derived NodeCapabilities, SchedulerResourceInterconnect CRD, inventory client plumbing, workload builder env injection. - `infiniband.NCCLIBHCAPrefix string` → `NCCLIBHCAPrefixes []string`; rewrite the sysfs scanner to publish every distinct HCA family rather than a least-common-prefix string. Replace the `_`-split heuristic with trailing-digit stripping so `bnxt_re0` correctly resolves to `bnxt_re` instead of getting truncated to `bnxt`. - Workload builder joins NCCLHCAPrefixes with `,` for NCCL_IB_HCA. AKT-493 — configurable extended-resource patterns - Add `interconnect.resource_patterns []string` to the inventory operator's provider.yaml. Empty list preserves rc4 behavior (`rdma/rdma_shared_device_*` default). Patterns containing glob metacharacters use filepath.Match; bare strings match as a literal prefix (rc4 contract). - Thread the configured patterns through `nodeDiscovery.initNodeInfo` + `updateNodeInfo` so a node that publishes `broadcom.com/rdma` or `intel.com/iwarp_shared_*` is recognized. - Malformed glob → fail-closed (no match) so a typo in provider.yaml surfaces as the node showing no interconnect, not as picking the wrong resource. AKT-494 — NCCL_IB_GID_INDEX=3 auto-inject for RoCE - When the chosen node's fabric is `roce`, the workload builder now injects `NCCL_IB_GID_INDEX=3` (RoCEv2 + VLAN GID, the production default). IB fabrics stay untouched (NCCL's default selection is correct there). - addIfNotPresent honors any tenant `NCCL_IB_GID_INDEX` set via SDL `env:` — same override semantics as the other auto-injected NCCL knobs. CRD: `SchedulerResourceInterconnect.nccl_hca_prefix: string` → `nccl_hca_prefixes: array`. The provider-pconfig-reset routine (delete home-akash-provider-0 PVC after every image bump) already covers upgrade on the test cluster. Linear: AKT-492, AKT-493, AKT-494
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
cluster/kube/operators/clients/inventory/interconnect_group_adjust_test.go (1)
77-77: ⚡ Quick winUse the shared interconnect-group key constant in the fixture.
This hardcoded key can drift from production behavior. Reuse
AttributeGPUInterconnectGroupKeyhere.Suggested change
- attrs = append(attrs, attrtypes.Attribute{Key: "interconnect/group", Value: group}) + attrs = append(attrs, attrtypes.Attribute{Key: AttributeGPUInterconnectGroupKey, Value: group})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cluster/kube/operators/clients/inventory/interconnect_group_adjust_test.go` at line 77, The hardcoded string "interconnect/group" in the attrs append statement should be replaced with the shared constant AttributeGPUInterconnectGroupKey to prevent the test fixture from drifting out of sync with production behavior. Replace the Key field value in the Attribute struct initialization from the hardcoded string to the constant reference.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cluster/kube/operators/clients/inventory/interconnect_group_adjust_test.go`:
- Around line 174-182: The test
TestAdjust_InterconnectGroup_DistinctGroupsIndependent is not effectively
validating cross-group co-location independence because the current resource
requests (8 GPUs each) cannot fit on a single node anyway (each node has 8
GPUs), making the test pass regardless of whether group independence logic is
broken. Modify the test to create a scenario that requires co-location: change
interconnectGroupCluster from 2 nodes to 1 node, and reduce each
interconnectGroupResource request from 8 GPUs to 4 GPUs. This ensures the test
will fail if group independence is accidentally broken and services from
different groups are prevented from co-locating on the single available node.
In `@operator/inventory/node-discovery.go`:
- Around line 878-893: The updateNodeInfo function retains previously set
InterconnectResourceName and GPUInterconnect values when no matching resource is
found, causing stale device-plugin resources to persist in node inventory. Reset
the interconnect capability fields (InterconnectResourceName and related
GPUInterconnect capacity/allocatable fields) to their initial empty state at the
beginning of updateNodeInfo before attempting to match and set new values,
ensuring that removed or renamed device-plugin resources are properly cleared
from the node inventory.
- Around line 845-857: The matchInterconnectResourceName function currently
iterates through resources in the outer loop and patterns in the inner loop,
which causes pattern priority to be ignored since map iteration order is
undefined. Reverse the iteration order so that patterns are iterated first (in
the outer loop using the effective slice which preserves order) and resources
are iterated second (using range over quantities), then return the first
resource that matches the current pattern. This ensures the highest-priority
pattern is checked first and correctly enforces configured pattern precedence
regardless of resource iteration order.
---
Nitpick comments:
In `@cluster/kube/operators/clients/inventory/interconnect_group_adjust_test.go`:
- Line 77: The hardcoded string "interconnect/group" in the attrs append
statement should be replaced with the shared constant
AttributeGPUInterconnectGroupKey to prevent the test fixture from drifting out
of sync with production behavior. Replace the Key field value in the Attribute
struct initialization from the hardcoded string to the constant reference.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 76f7d9cd-e331-4c61-a539-5c06be60578d
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (19)
cluster/kube/builder/builder.gocluster/kube/builder/interconnect_test.gocluster/kube/builder/workload.gocluster/kube/operators/clients/inventory/interconnect.gocluster/kube/operators/clients/inventory/interconnect_adjust_test.gocluster/kube/operators/clients/inventory/interconnect_group_adjust_test.gocluster/kube/operators/clients/inventory/interconnect_test.gocluster/kube/operators/clients/inventory/inventory.gocluster/types/v1beta3/clients/inventory/metrics.gogo.modoperator/inventory/config.gooperator/inventory/infiniband.gooperator/inventory/infiniband_test.gooperator/inventory/interconnect_resource_match_test.gooperator/inventory/node-discovery.gooperator/psutil.gopkg/apis/akash.network/crd.yamlpkg/apis/akash.network/v2beta2/manifest.gopkg/apis/akash.network/v2beta2/zz_generated.deepcopy.go
✅ Files skipped from review due to trivial changes (1)
- pkg/apis/akash.network/v2beta2/zz_generated.deepcopy.go
🚧 Files skipped from review as they are similar to previous changes (2)
- operator/psutil.go
- operator/inventory/infiniband_test.go
Zblocker64
added a commit
that referenced
this pull request
Jun 16, 2026
Resolves the go.mod / go.sum conflict from main's 62f162a (`fix(provider): report reclamation window in status`, bumping pkg.akt.dev/go to v0.2.16). The replace directive pinning rc5 is kept; main's require version coexists since the replace overrides at build time. Also addresses CodeRabbit review on PR #400 (2026-06-15): 1. matchInterconnectResourceName: reverse loop order so patterns are the outer loop. Map iteration over kubelet's ResourceList is non-deterministic, so the rc5 "first hit wins" guarantee silently depended on Go's hash seed. Pattern order now decides which resource wins — pinned by a new precedence test. 2. updateNodeInfo: clear node.Capabilities.InterconnectResourceName + GPUInterconnect.{Capacity,Allocatable} up-front so a node that loses its device-plugin resource (operator restart, plugin uninstall, pattern config change) properly drops to non-interconnect rather than retaining the stale name + zeroed capacity indefinitely. Pinned by TestUpdateNodeInfo_ClearsStaleInterconnectWhenResourceGone. 3. TestAdjust_InterconnectGroup_DistinctGroupsIndependent: 8 GPUs per service × 2 services on a 2-node × 8 GPU cluster is the only fitment pattern regardless of group independence — the test passed even if groups were collapsed. Shrink to 1 node × 4 GPUs per service so the *broken* code (groups collapsed) would reject the second service. Added interconnectGroupResourceN helper for variable-unit sizing. 4. Use AttributeGPUInterconnectGroupKey constant in the test attrs fixture instead of the hardcoded "interconnect/group" string — keeps test in sync if the on-chain key ever moves.
Zblocker64
added a commit
that referenced
this pull request
Jun 19, 2026
Resolves the go.mod / go.sum conflict from main's 62f162a (`fix(provider): report reclamation window in status`, bumping pkg.akt.dev/go to v0.2.16). The replace directive pinning rc5 is kept; main's require version coexists since the replace overrides at build time. Also addresses CodeRabbit review on PR #400 (2026-06-15): 1. matchInterconnectResourceName: reverse loop order so patterns are the outer loop. Map iteration over kubelet's ResourceList is non-deterministic, so the rc5 "first hit wins" guarantee silently depended on Go's hash seed. Pattern order now decides which resource wins — pinned by a new precedence test. 2. updateNodeInfo: clear node.Capabilities.InterconnectResourceName + GPUInterconnect.{Capacity,Allocatable} up-front so a node that loses its device-plugin resource (operator restart, plugin uninstall, pattern config change) properly drops to non-interconnect rather than retaining the stale name + zeroed capacity indefinitely. Pinned by TestUpdateNodeInfo_ClearsStaleInterconnectWhenResourceGone. 3. TestAdjust_InterconnectGroup_DistinctGroupsIndependent: 8 GPUs per service × 2 services on a 2-node × 8 GPU cluster is the only fitment pattern regardless of group independence — the test passed even if groups were collapsed. Shrink to 1 node × 4 GPUs per service so the *broken* code (groups collapsed) would reject the second service. Added interconnectGroupResourceN helper for variable-unit sizing. 4. Use AttributeGPUInterconnectGroupKey constant in the test attrs fixture instead of the hardcoded "interconnect/group" string — keeps test in sync if the on-chain key ever moves.
Zblocker64
force-pushed
the
feat/rdma-multinode
branch
from
2c463f4 to
97b307b
Compare
Resolves the go.mod / go.sum conflict from main's 62f162a (`fix(provider): report reclamation window in status`, bumping pkg.akt.dev/go to v0.2.16). The replace directive pinning rc5 is kept; main's require version coexists since the replace overrides at build time. Also addresses CodeRabbit review on PR #400 (2026-06-15): 1. matchInterconnectResourceName: reverse loop order so patterns are the outer loop. Map iteration over kubelet's ResourceList is non-deterministic, so the rc5 "first hit wins" guarantee silently depended on Go's hash seed. Pattern order now decides which resource wins — pinned by a new precedence test. 2. updateNodeInfo: clear node.Capabilities.InterconnectResourceName + GPUInterconnect.{Capacity,Allocatable} up-front so a node that loses its device-plugin resource (operator restart, plugin uninstall, pattern config change) properly drops to non-interconnect rather than retaining the stale name + zeroed capacity indefinitely. Pinned by TestUpdateNodeInfo_ClearsStaleInterconnectWhenResourceGone. 3. TestAdjust_InterconnectGroup_DistinctGroupsIndependent: 8 GPUs per service × 2 services on a 2-node × 8 GPU cluster is the only fitment pattern regardless of group independence — the test passed even if groups were collapsed. Shrink to 1 node × 4 GPUs per service so the *broken* code (groups collapsed) would reject the second service. Added interconnectGroupResourceN helper for variable-unit sizing. 4. Use AttributeGPUInterconnectGroupKey constant in the test attrs fixture instead of the hardcoded "interconnect/group" string — keeps test in sync if the on-chain key ever moves.
Zblocker64
force-pushed
the
feat/rdma-multinode
branch
from
97b307b to
e7ec745
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provider side of the GPU interconnect feature. Multi-node tensor-parallel GPU workloads (NCCL over IB or RoCE) can now land on Akash providers. Consumes chain-sdk rc4 (
pkg.akt.dev/go v0.2.13-rc4,pkg.akt.dev/go/sdl v0.2.2-rc4).What's in here
/sys/class/infiniband+ the kubeletrdma/rdma_shared_device_*extended resource. Also a PCI-bus fallback that catches GPUs on headless GPU-Operator nodes (where ghw's/sys/class/drmwalk only sees the BMC).interconnect_groupnode-claim tracking so the bid step refuses orders the kube scheduler couldn't satisfy.SchedulerResourceInterconnect+Service.InterconnectGroup, with matching CRD schema additions to survive API-server pruning.request==limit), NCCL env (NCCL_IB_DISABLE=0,NCCL_IB_HCA=<prefix>), and per-interconnect_grouppod anti-affinity (hostname topology). All SDL-overridable.Linear
AKT-407..412 (P-1..6), AKT-436..439 (review fixes), AKT-443 (group-aware Adjust).
Test plan
go test ./cluster/... ./pkg/... ./operator/...rdma/rdma_shared_device_ib: 8+ NCCL env injected.