Skip to content

[Docs] Document the capacity high-water mark's hysteresis band - #21

Merged
slin1237 merged 2 commits into
mainfrom
codex/nightly-docs-0ccd2f9b7fd65110
Oct 2, 2026
Merged

slin1237 merged 2 commits into
mainfrom
codex/nightly-docs-0ccd2f9b7fd65110

Conversation

@github-actions

@github-actions github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What this PR does

When does the high-water mark that budgets are checked against come down after the cluster loses accelerators, and how do I tune that?

Why we need it

Source change: ome-projects/ome@37ce902

pkg/quota/capacity/highwater.go (Track, TrackAll, installed) in the current checkout: the mark is a running peak of installed capacity (allocatable plus capacity parked on cordoned or NotReady nodes); growth is believed immediately, and a drop is believed only when installed capacity falls more than --capacity-hysteresis-percent below the mark (cmd/ome-quota-manager/main.go:173; chart value quotaManager.capacity.hysteresisPercent, default 10 in charts/ome-quota-manager/values.yaml:114-124; 0 disables damping, >=100 never lowers the mark). A (resource, flavor) pair that stops reporting keeps its mark until it reaches zero. The current page src/lib/content/guides/operate-ome/accelerator-quota.md line 151 explains the mark only as 'also counts cordoned and not-Ready nodes, so it stays put while you drain nodes', which misses the damped-running-peak behavior — after a real decommission within the band the mark never comes down — and the Helm-value table (lines 286-291) omits hysteresisPercent. One concern: one stale explanation of the mark plus its single tuning knob, introduced by commit 18 which added highwater.go and the flag.

Scope: quota / capacity-high-water-mark-hysteresis. Other concerns are deferred.

How to test

  • Passed the documentation path and size guard (under 1000 added plus deleted lines; no file-count limit).
  • Passed an independent accuracy and single-concern review.
  • Passed git diff --check and website content/link tests, type checks, lint and production build.

Checklist

  • Every commit is signed off (git commit -s)
  • pnpm lint && pnpm check && pnpm test && pnpm build passes (run by the publisher on an isolated copy)

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 4f1cbc3d-1423-442d-87d6-f9b965a0878d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Documentation maintenance: ready. Unsuccessful content rounds: 0/3. Operational attempts: 0/3.

Head: a7878e1c533b1eb5329def3508ad74cf30fbe9b0; reviewed main: 40ff7deafa9a366fda6cc0cb9598dbef9b7fe515.

All edits serve the single concern (the high-water mark's damped-running-peak behavior and its hysteresisPercent knob) and every claim verifies against the OME checkout at bc1f94db: Track/TrackAll semantics in pkg/quota/capacity/highwater.go (growth believed at once; a drop believed only past the band, falling all the way to the observation; band<=0 disables damping, >=100 never lowers), the --capacity-hysteresis-percent flag (cmd/ome-quota-manager/main.go:173) with chart default 10 (charts/ome-quota-manager/values.yaml:124, passed in templates/deployment.yaml:98), the strict-comparison worked example (24 vs threshold 28.8 lowers the mark; 30 vs 28.8 keeps 32), CapacityExceeded against the mark with subtree freezing (capacitycheck.go, tree.Frozen, materialize.go), and mark persistence in root.Status.Capacity across restarts. The previously rejected clearing procedure is replaced by a paragraph that now matches executable behavior: capacity.Sum never observes a pair with zero installed capacity (acceleratorsOf skips zero allocatable), TrackAll's carry-forward ignores the band so damping off cannot lower a retained mark, reconcileCapacity's empty-pool samples preserve recorded marks, and the controller test "a pool that disappears keeps its mark" (capacity_test.go:434) exercises exactly this scenario; the advice to delete budgets naming a retired pool is sound since such budgets keep passing the capacity check against the stale mark. Output examples match attributeCapacity's cross product, the #lower-the-high-water-mark anchor matches the slugger, and the diff adds only Markdown to the one page with no nav/redirects/code edits. No review threads exist in the context, so none are reported addressed.

Workflow evidence

Human review threads and CODEOWNER approval remain under repository policy.

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@slin1237
slin1237 merged commit 23c0cbc into main Oct 2, 2026
2 checks passed
@slin1237
slin1237 deleted the codex/nightly-docs-0ccd2f9b7fd65110 branch October 2, 2026 18:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant