Repository navigation
Pool: reconciler, controller, membership, fleet simulator and placement planner - #74
Open
data-angel wants to merge 11 commits into
Open
data-angel wants to merge 11 commits into
data-angel wants to merge 11 commits into
Conversation
Treat every cluster node as one pool and propose which model lanes run where, with which residency, and how each alias routes, plus the diff from what runs now. Phase 1 is read-only: nothing is started, stopped or rerouted, and per-node admission stays the safety layer. - Node inventory (/gateway/node?inventory=1, cached): structured compute archs, memory domains, model-root disk, cached recipe weights, commands and Docker images, and NVIDIA Sync fabric links. - Model-centric catalog from managed runtimes, federated targets, unmanaged lanes declared with a model `pool` block, external models and (opt-in) bundled recipes, with measured footprint and live-metric or benchmark performance. - Pure planner: memory per domain with headroom, TP only across directly linked nodes, arch/platform/accelerator matching, pins and cordons as hard constraints; lexicographic objective (coverage, external fallback, SLOs, demand-weighted latency plus load-time churn); greedy placement plus single-workload local search. Models sharing a runtime share one placement. - Per-alias demand: /gateway/metrics adds `requested` buckets. - Recipes may declare requirements.computeArchs, requirements.tensorParallel and settings.maxSequences (schema, validator, example). - Admission and pool planning share one memory-budget helper. Fixture tests reproduce the three-Spark fleet: all nodes give the sparkqwen-pair-rc16 profile; cordoning the pair gives standalone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pool-reconciler: orders a plan into make-before-break steps (prefetch, start, route, drain+stop), swaps through interim members, residency downgrades before stops; dependency-aware executor skips dependents of a failed step so a lane that never starts never strands its aliases. - pool-controller: off/advise/auto loop with urgency (coverage restored), stable-plan confirmation, cooldowns, an hourly load budget, slower hysteresis for residency-only changes, and failure backoff. - pool-sim: discrete-event fleet model (requests, admission + LRU eviction, keep-warm restore, memory guard aborts, node hangs/failures, operator actions) driving the real planner/reconciler/controller; scenarios for pair taken, pair crash, a Mac joining, a low memory estimate and noisy demand, compared against today's behaviour (off). - planner: budgets capped at the runtime memory guard ceiling (0.9 by default) so plans never rely on loads the guard would abort; variants can exclude nodes after failed starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nodes can join a head with a one-time enrollment token and keep a lease with heartbeats from `serve`, so the pool planner knows who is in the pool. Membership is inert until a token is issued; static cluster.nodes config and `cluster add-node` are unchanged. - src/pool-membership.mjs: pure membership core (tokens, enroll, heartbeat, cordon/drain/leave, ready/suspect/lost/left leases, planner node shape). Tokens and node secrets are stored only as SHA-256 hashes. - src/pool-membership-store.mjs: pool-membership.json next to the config, loaded on first use, written atomically (0600) only on changes. - Head routes: POST /gateway/pool/enroll (token only, rate limited) and /gateway/pool/heartbeat (node secret only) authenticate in their handlers; GET /gateway/pool/members is admin-read; tokens and cordon/uncordon/drain are admin writes. - src/pool-node.mjs: `lloom pool join` and the serve heartbeat loop with backoff; the secret lives in a private file or an env var. - CLI: lloom pool token create | members | cordon | uncordon | drain | join. - Planner: members merge with configured nodes (config wins on endpoint and auth; membership adds liveness, cordon and inventory). Enrolled-only nodes are planned but flagged as needing `cluster add-node` to route. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fleet simulator showed admission could approve a load the runtime memory guard was certain to abort: admission kept only reserveMemoryGb (8 GB) free, while the guard trips at 90% host use (12.8 GB free on a 128 GB Spark). After an abort the runtime is latched until a manual start, so the lane stayed down. - runtime-policy: the host check keeps max(reserve, guard margin) free and cluster node budgets are capped at the guard ceiling. - runtime-memory-safety: memorySafetyCeiling() shared by the guard, admission and pool planning. - runtime-manager: a guard abort records usedGb and consumedGb (the load's lower-bound footprint) and status reports memorySafetyLatched. - pool-catalog: raises a variant's memory to the abort lower bound and excludes a latched lane until it is started by hand. - reconciler: per-step timeouts so a stuck step cannot wedge the controller; simulator found and now covers a drain that never settled. - simulator: models the latch and fixed admission (legacyAdmission keeps the old behaviour for comparison) and logs controller hold reasons. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The simulator can run nodes through pool membership (enroll, heartbeat, lease loss) so node loss is detected the way a head detects it; new lease-loss scenario (crash plus a harmless 60 s partition). - The scenario found the controller blocked for 30 minutes on a manual restart of a lane whose node had just died. Stability now tracks the moves, not just the plan; an urgent plan for a different target cancels a change in progress (executeReconcile takes an AbortSignal); the same target never preempts itself. - Reconciler re-issues an idempotent start for lanes still loading so route flips wait for them. - Register membership tests in check:js and test:unit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lloom pool simulate <scenario> [--mode off|advise|auto|compare]: runs a
scenario through the planner, reconciler and controller; exits non-zero
when expectations fail.
- docs/pool-planning.md: reconciling, the controller, simulation and the
scenario results; runtime-policy.md: the guard margin in admission.
- Membership tests sliced the token body with split('_'), but base64url
bodies can contain '_', so a short fragment sometimes matched the state
JSON; slice the fixed prefix instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Security review findings: - Medium: any enrollment token could re-enroll any node id, replacing a live node's secret, endpoint and reported capacity (and flipping a configured peer unavailable in plans). Tokens can now be bound to a node id (lloom pool token create --node-id); claiming a current member or a configured cluster.nodes id needs a token bound to it (409 otherwise). lloom pool leave / POST /gateway/pool/nodes/:id/leave frees an id. - Heartbeats are refused before the body is read unless the bearer has the node-secret shape, and are rate limited per client. - Rate limiters evict least recently used clients instead of clearing the whole map when full. - Node-reported text with control characters is dropped (no escape sequences in lloom pool members output). - join reports the saved secret path and the pool.head to set when the config update fails after enrolling, and warns about plain HTTP heads off loopback and tailnets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugs (each with a test): - A model that moved between nodes left its old lane running: an unplanned lane is now retired once every planned lane serving its models is up. - Lanes already stopping were treated as freed memory; they now hold it and get an idempotent stop that starts on that node wait for. - The planner could plan past the guard when only memorySafety was set; the catalog now passes memorySafetyCeiling() itself. - A cluster node with unknown capacity got a negative admission budget, blocking no-request plans; the clamp is floored at zero, and the guard margin rounds up. - Memory-safety abort evidence never cleared; a healthy start clears it and a measured footprint supersedes it. - An alias moved to an interim member during a swap was never moved back; the flip back is queued and ordered after the interim flip. Hardening: ticks never overlap; throwing observers cannot wedge the executor or controller; concurrency is at least 1; a lane already loading is not charged to the load budget again; one move larger than the budget may go in an empty hour; only route changes preempt; vacating a cordoned or draining node is urgent. The simulator reports draining lanes as stopping and the controller's last plan, and memory-estimate-low now asserts the abort, the latch and video planned as uncovered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anner Brings in origin/pool-head-sim (which includes pool-planner-phase1): pool reconciler/controller, membership enrollment tokens and node leases, deterministic fleet simulator, and the dry-run placement planner. package.json and CHANGELOG resolved against current main, with pool sources/tests added to check:js and test:unit.
This was referenced Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrates the pool management feature set (branches
pool-head-simandpool-planner-phase1) into currentmain:src/pool-reconciler.mjs,src/pool-controller.mjs): converge pool state, with controller preemption and admission kept below the memory guard (guard aborts are learned from).src/pool-membership.mjs,src/pool-membership-store.mjs,src/pool-node.mjs): enrollment tokens, node leases with node-id takeover protection, and a planner bridge.src/pool-sim.mjs,src/pool-sim-cli.mjs): deterministic scenario-driven simulator (lloom pool simulate) with membership leases; scenarios intest/fixtures/pool/scenarios/.src/pool-planner.mjs,lloom pool plan): plans pool placement against the pool catalog/inventory without mutating state.docs/pool-planning.md, updates todocs/clusters.md,docs/architecture.md,docs/runtime-policy.md,README.md,CHANGELOG.md) and 9 new test files (109 pool tests, all passing).Integration notes
package.jsonandCHANGELOG.mdwere resolved against currentmain(the branches forked ~30 commits back); pool sources/tests were added tocheck:jsandtest:unit.npm run check:jspasses; fulltest:unitpasses (one unrelated flakyresearch-workerstiming test failed once and passed on 4 subsequent runs, including on cleanmain).Sanitization
Reviewed the full diff for credentials, tokens, local paths, and private hostnames: none present. All content is product code, docs, and test fixtures.