fix(inventory): exclude tainted nodes from the inventory#406
Conversation
Node discovery only excluded nodes that were not-ready, cordoned, or in the operator exclude list. Nodes carrying a NoSchedule/NoExecute taint (e.g. control-plane, CriticalAddonsOnly) were still reported as available capacity, so the provider bid on workloads it could never schedule and the deployment was closed minutes later. Treat any NoSchedule/NoExecute taint as a reason to keep the node out of the inventory. PreferNoSchedule is a soft preference and is left alone. refs: akash-network/support#253
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughThe ChangesTaint-based node exclusion
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
| adjConfig.FilterOutStorageClasses(presentSc) | ||
|
|
||
| isExcluded := !isNodeReady(knode.Status.Conditions) || knode.Spec.Unschedulable || adjConfig.Exclude.IsNodeExcluded(knode.Name) | ||
| isExcluded := !isNodeReady(knode.Status.Conditions) || knode.Spec.Unschedulable || nodeHasBlockingTaint(knode.Spec.Taints) || adjConfig.Exclude.IsNodeExcluded(knode.Name) |
There was a problem hiding this comment.
lets keep content of nodeHasBlockingTaint within isNodeReady
There was a problem hiding this comment.
Done. folded the taint check into isNodeReady, which now takes the node and treats a NoSchedule/NoExecute taint as not-ready. Dropped the separate helper.
Per review: keep the taint check inside isNodeReady rather than a separate helper. isNodeReady now takes the node and treats a NoSchedule/NoExecute taint as not-ready, so the exclusion logic stays a single predicate.
What
Node discovery (
operator/inventory/node-discovery.go) only excluded nodes that were not-ready, cordoned (Unschedulable), or in the operator exclude list. Nodes carrying aNoSchedule/NoExecutetaint were still reported as available capacity.This adds
nodeHasBlockingTaintand treats anyNoSchedule/NoExecutetaint as a reason to keep the node out of the inventory.PreferNoScheduleis a soft preference and is left alone.Why
Closes the behaviour reported in akash-network/support#253: providers with tainted nodes (e.g.
node-role.kubernetes.io/control-plane,CriticalAddonsOnly) kept bidding using that capacity once the rest of the cluster was full. Akash pods carry no custom tolerations, so the pod can never schedule onto a tainted node - the deployment failsFailedSchedulingand the provider closes it minutes later. Reported in the wild on multiple providers and on a support call.Testing
Added
TestGenerateLabels_Taints: a ready, untainted node is marked akash-managed;NoScheduleandNoExecutetainted nodes are excluded;PreferNoScheduleis not. Fails without the fix.refs: akash-network/support#253