fix(chart): interim control-plane trim so the smallest training envelope fits a 2-core / 8 GiB node again (1.9.112)

[LukasWodka]

Summary

Interim trim of the control plane's resource requests so the smallest training envelope fits the smallest supported node again. Chart 1.9.111 → 1.9.112.

Why. Since 1.9.109 the kubelet reservation is honest (#1017) and the jobs-manager's admission subtracts the chart's real requests. On a 2-core / 8 GiB node: allocatable 1900m / 5538 Mi, minus the control plane's requests (800m / 3008 Mi without the collector) and k3s's coredns + metrics-server (200m / 140 Mi), leaves 900m / 2390 Mi, and the smallest envelope needs 1000m / 3072 Mi. Every task on staging was refused NEVER_STEADY_STATE today (e2e fleet run 34473120598, 15/16 red at Train).

The requests were never sized from usage: 1050m / 3276 Mi requested against 36m / ~1022 Mi used. Both halves of the accounting are now honest; the third number was not.

What changed

component	before	after	basis
mysql	250m / 1 Gi	150m / 704 Mi	peak measured 0.40–0.55× of 1 Gi; +25 % headroom; requests==limits kept
jobs-manager	250m / 1 Gi	250m / 768 Mi	168 Mi idle (0.16×); restart peak not yet measured, so a modest step; cpu untouched
pods-monitor	100m / 512 Mi	100m / 320 Mi	requests==limits kept (oom_score_adj)
requests-proxy	100m / 256 Mi	50m / 128 Mi	a few requests/min; limits unchanged (1000m / 512 Mi)
egress-proxy	50m / 64 Mi	50m / 160 Mi	up: measured 2.45× over its request

Render: 3136 MiB / 900 m → 2336 MiB / 750 m (all four CI profiles). That node now leaves 1050m / 3190 Mi; the envelope fits with 50m / 118 Mi to spare.

Everything derived from the render moves with it, none of it restated:

• control-plane-footprint.sh ceiling ratcheted to 2336 / 750.
• Installer embed regenerated (gen-footprint-embed.sh, bash + PowerShell), --check green.
• Template fallback defaults aligned with values (no second copy of the old numbers).
• envelope-schedulability.sh + .bats + the PowerShell twin: the 8 GiB reproduction is pinned as fitting unreduced (the regression guard for a chart change pushing the footprint back up); the reduce/refuse paths run on nodes derived from the footprint (SMALL_MI, TINY_MI); the cpu-only overshoot replays the pre-trim 900 m because no real node can produce one any more; the guard's positive control reports "does not apply" and the bats test accepts that shape.
• Mutation needle in envelope-schedulability-mutations.sh retargeted to the new embed literal.
• values.yaml comment carries the derivation and what is still owed.

What this is NOT

The final trim. The final requests come from the load run (ambient, one cycle, five-queued burst, node restart under 8 GiB) that is running now; jobs-manager's request in particular is justified by the restart peak and moves only once that leg is measured.

Follow-ups (other repos)

• client-runtime: regenerate envelope_contract.json's overhead from the new render (contract v4) and its vectors.
• client + cli: re-vendor that contract (fixtures, installer embed of the overhead constants, drift refs).

Test plan

• helm unittest client: 728/728.
• scripts/tests/control-plane-footprint.sh: OK, 2336 MiB / 750 m at the ceiling.
• scripts/gen-footprint-embed.sh --check: embed equals render.
• scripts/tests/envelope-schedulability.sh: 24 ok / 0 failed; -mutations.sh: 11/11 caught (unmutated tree green).
• scripts/tests/qos-reachability.sh: 22/22.
• bats: envelope-schedulability, install-client-helm, chart-version-guard — green locally.
• Pester (PowerShell twin) — CI only, no pwsh locally.
• After promotion to staging: the e2e fleet trains on all 16 tasks again on the 2-core k3d node.

Checklist

• Targets develop
• Chart version bumped, manifest regenerated

---

Note

Medium Risk
Changes scheduler reservations and OOM scoring tradeoffs (jobs-manager/pods-monitor requests below limits) across core data-plane pods; mitigated by limits, tests, and footprint drift checks but not a final measured trim.

Overview
Chart 1.9.112 lowers control-plane resource requests (and bumps egress-proxy memory request) so the rendered footprint fits under the installer’s 3 GiB reserve and the smallest training envelope can schedule on a 2-core / 8 GiB node again.

Helm defaults move from measured-over-reserved numbers toward usage-based interim values: mysql 150m/704Mi (requests still equal limits), jobs-manager 150m/640Mi request with 1 Gi limit kept, pods-monitor 384Mi request with 512 Mi limit kept (deliberately undoing #1144 request==limit parity on scheduler accounting only), requests-proxy 50m/128Mi, egress-proxy 160Mi request (up from 64Mi). Template fallbacks and values.yaml comments document backend#2461 derivation.

Installer and guards regenerate the embedded control-plane footprint (~650m / ~2272 MiB render) in bash and PowerShell, ratchet control-plane-footprint.sh ceilings, and retarget envelope-schedulability tests: the 8 GiB case is now pinned as fitting unreduced; reduce/refuse/cpu-only paths use footprint-derived node sizes or replay the pre-trim CPU footprint where a real node no longer reproduces overshoot.

^{Reviewed by Cursor Bugbot for commit ac90616. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[LukasWodka]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[LukasWodka]

bugbot run

[saadqbal]

Reproduced everything locally on 8f60126 — render 2336 MiB / 750 m on all four profiles, make drift 48/48, helm unittest 728/728, bats 274, and the Pester box you left unchecked is fine (*envelope schedulability* 20/20 on pwsh 7 / Pester 5.7.1). Bugbot's PowerShell overflow finding is a false positive, by the way: 3136 * 1MB widens to Double (exact at that magnitude), nothing throws, and the undersized case runs green.

Two things before I approve.

1. The trim doesn't reach the node it's aimed at, once the collector is counted. The footprint you embed is 750 m / 2336 MiB and that includes the otel-collector DaemonSet — helm template can't do a lookup, so tracebloc.telemetryCollectorState falls through to enabled and the 100m/128Mi lands in the render the ratchet and gen-footprint-embed.sh both read. Your justification arithmetic uses 650 m / 2208 MiB, the collector-less figure. On the node the PR names, your own _fit_training_envelope says:

--- node '1900m 5538Mi' (footprint 750m/2336Mi)
    resolver: cpu=1,memory=2Gi  ->  verdict=refused
      control plane (chart) 2336 MiB / 750 m + system pods 140 MiB / 200 m = 2476 MiB / 950 m
      cpu: 1000 + 950 = 1950 m > 1900 m (50 m OVER)
      what fits is 3062 MiB / 950 m -- not even a 1-core / 1-GiB run

Memory tells the same story for the admission's 3072 MiB envelope: 5538 − 2336 − 140 = 3062, ten MiB short. So on a collector-enabled 2-core edge this changes the failure mode but not the outcome. Either the collector belongs outside the footprint constant (and outside the guard's render), or the trim needs roughly another 100 m — and either way the values.yaml derivation should say which of the two numbers is the contract, because right now the file and the embed disagree by exactly the amount that decides the fit. If staging's edge genuinely has no collector, say so in the comment; the last unchecked box in the test plan is the one that will tell us.

2. pods-monitor is the one row with no measurement behind it, and it's the largest relative cut in the PR (512 → 320 Mi, −37.5%). requests==limits means that's a hard cgroup ceiling, and the scenario it has to survive is the cluster-recovery mass restart #1144 pinned it for — oom_score_adj protects against node pressure, not against its own limit, so above 320 Mi it's exit-137 again. Is there a number for it, or should that one wait for the load run like jobs-manager's?

Also four comments this falsified — worth the same pass, since the PR is otherwise careful about exactly this:

• scripts/lib/install-client-helm.sh:727-731 — still "3136 MiB / 900 m, from the render", still "over-asked at EVERY machine size".
• scripts/tests/envelope-schedulability.sh:30 — the WHAT IT ASSERTS list still says case 3 is REDUCED.
• scripts/tests/control-plane-footprint.bats:78 — "the live render the number is 3136 MiB either way"; you fixed the twin sentence in the .sh and not this one.
• scripts/tests/install-k8s.Tests.ps1:9783 — "the platform out-requests the reserve today" (the gate under it is right, just the prose).

Nice work on deriving SMALL_MI / TINY_MI from the footprint rather than writing new machine sizes down — that part ages well.

[LukasWodka]

bugbot run

[LukasWodka]

@saadqbal both points taken, fixed in fd3b874.

1. The collector is counted, and the trim now reaches the node. You are right that the render (and therefore the guard, the embed and my derivation) carries the otel collector's 100m / 128Mi, and I have kept it that way rather than carving it out: on a real edge the collector is rendered once jobs-manager has written the token Secret, so the worst case the render shows is the one the node must survive. To get there the trim goes roughly another 100m as you estimated: jobs-manager cpu 250m → 150m (the whole plane idles at 36m in the load-run summary). Render is now 650m / 2272Mi (was 750m / 2336Mi); on the 1900m / 5538Mi node that leaves 1050m / 3126Mi against the 1000m / 3072Mi envelope. Thin margins (50m / 54Mi), stated as such in the values.yaml derivation, which now also says explicitly that the collector is inside the constant and why.

2. pods-monitor. Agreed it was the one unmeasured row and that requests == limits turned the cut into a hard ceiling. I've split the two: jobs-manager request 1Gi → 640Mi and pods-monitor request 512Mi → 384Mi, with both limits left exactly where #1144 put them (1Gi / 512Mi). The request is what the scheduler and admission subtract; the limit is the restart-peak ceiling, and that peak is the load-run leg nobody has produced yet. Cost is a higher oom_score_adj under node pressure, which I judged cheaper than a ceiling nothing measured. Template fallbacks, the helm-unittest pins and the #1144 comment in the template say the same thing.

Stale comments — all four fixed: installer fit preamble (install-client-helm.sh), the WHAT-IT-ASSERTS list in envelope-schedulability.sh (case 3 now reads FITS, 3b is the derived REDUCED), the footprint bats fixture note (no literal any more), and the Pester positive-control note (explains it re-arms on its own if a chart out-requests the reserve again).

Regenerated from the render: ceilings 2272/650, installer + PowerShell embeds (2382364672 B / 650 m), mutation needle, manifest. Local: helm unittest 728/728, footprint OK, embed --check OK, schedulability 24/24, mutations 11/11, qos-reachability 22/22, bats 312/312. I still cannot run Pester here — could you run it once more on this head?

The red Prereqs — almalinux:8 on b0aebbf was a dnf mirror stall (dnf install sudo failed 3 bounded attempts, then the 12m step bound killed the container); the job is green on the last seven develop runs, and this push gives it a fresh run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit fd3b874. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[saadqbal]

Both blocking items are closed at ac906165, and the collector one you took the harder of the two options on. Clearing my change-request.

The collector is now the contract, and it's written down. control-plane-footprint.sh's header says it outright — the render counts the telemetry collector because helm template cannot look up the token Secret, "that is the worst case a real edge reaches once the Secret exists, and the one the ceiling must hold." That's the sentence that was missing: the file and the embed can no longer disagree about which number is the contract, because the collector-inclusive one is now named as it.

And you closed the gap rather than redefining it. jobs-manager cpu 250m → 150m is the ~100 m I said it needed, so the ceiling is 2272 MiB / 650 m collector-inclusive. On the node the PR names, both axes now clear:

cpu:    650 + 200 + 1000 = 1850 <= 1900   (50 m spare)
memory: 2272 + 140 + 3072 = 5484 <= 5538   (54 MiB spare)

Against 750 m / 2336 MiB that was 50 m over and 10 MiB short, so this is the difference between changing the failure mode and fixing it.

pods-monitor is answered better than the number I asked for. Requests 384 Mi with the limit back at 512 Mi means requests ≠ limits, so the hard cgroup ceiling is #1144's again and the mass-restart scenario keeps its headroom — the request is a scheduling hint now, not a ceiling. That removes the exit-137 risk without needing a measurement for 384, which is the right shape of answer. Same pattern on jobs-manager (requests 640 Mi, limit restored to 1Gi) and it reads deliberately, not incidentally.

All four falsified comments are fixed, and two of them in ways worth noting because a grep would call them unchanged:

• install-client-helm.sh:730 now scopes the stale figure to "When this was written the chart requested 3136 MiB / 900 m" and records the trim in the following sentence, so the "over-asked at EVERY machine size" claim is past tense where it's true.
• envelope-schedulability.sh case 3 keeps the word REDUCED, but it now describes 3b — a node sized from the live footprint to over-ask — while the 8 GiB reproduction "now FITS unreduced". Keeping the claim phrase so the mutation check retains its anchor is the right call; changing the wording would have silently unhooked a mutant.

Green everywhere: 55 checks passing, nothing pending, no unresolved threads. Nice work under a production-blocking clock.