docs(memory): sizing model, coordinated budget, and how to validate it - #2079
Merged
Conversation
The memory guide gave sizing rules as multiples of dataset size and a minimum RAM figure, with no account of the fixed cost those multiples sit on top of. That reads as "memory scales with data", which makes a proportionally downscaled staging environment look like a valid model of production. It is not one, and the docs said nothing that would have warned anyone off it. Reframes the guide around `Total = Baseline + Working Set`. The baseline is a function of DATASET COUNT, not data volume: each Cayenne table holds a segment cache (1/128 of memory, clamped to 256 MB-1 GB), a CDC table adds a PK keyset cache (1/32, clamped 256 MiB-8 GiB) and a coalesce buffer. The clamps are the point — the per-dataset cost stops shrinking once the environment is small, so it is a rounding error at production volumes and the dominant term below them. A worked table shows the same ten-dataset Spicepod at ~76% baseline in 4 GiB and ~9% in 32 GiB; the existing multiples are relabelled as working-set figures. Four things from the runtime that the docs did not carry, all of which change what an operator should actually do: `How the Runtime Partitions Memory` — the 70/20/10 partition (pools / in-memory CDC tier / off-pool headroom), the compaction carve out of the query pool, and the 50% floor. Makes the baseline/working-set split concrete rather than conceptual, and gives the reservation clawback a place to be explained. `Tuning the Memory Limit Safely` — the limit is bounded on BOTH sides when CDC is active, and both directions are counterintuitive. Lowering it need not reduce resident memory: the tier is sized from what is left after the pools and floats up to a quarter of memory to reclaim it, so the memory moves rather than releasing. Raising it squeezes the tier below its floor into a refuse-all gate and ingestion falls back to disk. Hence the guidance to leave it unset and size the container. `Bounding Peak Memory with Concurrency` — `max_concurrent_queries` defaults to 4x the CPU entitlement's cores and `target_partitions` to the cores; neither consults the memory limit. With the entitlement at twice the pod's CPU request, a `requests.cpu: 4` pod admits 32 concurrent queries however little memory it has. This is the better first lever for peak memory: it lowers the peak, where lowering `memory_limit` only shrinks each query's share of it. `Check the Startup Budget Warnings First` — the partition is computed before any data loads and warns when it cannot fit, at `warn` level. An unfittable configuration is therefore detectable in the first seconds of a run rather than after days of load, which is the cheapest sizing check available and belongs ahead of load testing. Four warnings tabulated with meaning and action. (The comment on the first records a ~122 GiB host OOM-killed at SF-1000 whose only trace was that line.) Also `Reading a Memory Graph` (a flat high plateau is caches at their ceilings and allocator retention, not a leak — what matters is whether and where it plateaus), `Validating a Memory Configuration` (the four properties that make a load test predictive: data volume, peak concurrency, duration to steady state, predicate diversity; shadow/canary as the alternative), and `Client-Side Resiliency` (retry in-cluster before falling back; instance loss is independent of memory tuning). Accuracy fixes found on the way: - `cayenne_segment_cache_mb` was documented as a flat `256` default. It is auto-derived — 256 MB is the floor and it reaches 1 GB above a 32 GiB host. - The cache-sizing example counted accelerator caches once per deployment. They are per dataset; the example now uses four and says what a fifth costs. - Two statements attributed the 70% base to "Cayenne acceleration is active". It is specifically the in-memory CDC tier being reachable — Cayenne bulk-only keeps the 90% base less the full reservation. Both now match the source, and agree with performance-tuning.md, which already had it right. Supporting pages pick up the levers where the problem is actually hit: performance-tuning gets the which-knob-for-which-intent table, deployment/kubernetes gets a Memory sizing section beside the CPU one, troubleshooting gets an OOM-despite-a-configured-limit entry. Every constant is checked against the shipped source: the 90/70/50 percentages, the 20% compaction carve, the tier's 1/5 ceiling and 1/32 floor and 1/4 float, the two cache clamps, the 128 MiB coalesce buffer, `QUERIES_PER_CORE = 4`, and `target_partitions = cores`.
Contributor
✅ Pull with Spice PassedPassing checks:
|
Contributor
|
🚀 deployed to https://cd1df99b.spiceai-org-website.pages.dev |
bjchambers
reviewed
Aug 11, 2026
bjchambers
reviewed
Aug 11, 2026
sgrebnov
reviewed
Aug 11, 2026
krinart
approved these changes
Aug 11, 2026
…y cost, task history Review feedback from @bjchambers and @sgrebnov. The baseline was described as "present at idle", which is wrong in a way that matters: most of it is bounded buffers — the SQLite metastore's page cache, gRPC/HTTP serialization buffers, the accelerator caches — that reach their ceilings only as traffic drives them there. It is the CONVERGED footprint, not the startup footprint, and a freshly started runtime under-reports it. Corrected here, on the Kubernetes page, and in the cache-sizing example. The working set was described as scaling with dataset size alone. It also scales with the data each query touches — `SELECT 1` against `SELECT *` against a multi-way join with a sort — and that per-query cost is then multiplied by concurrency. Both terms therefore understate themselves early, which is now stated where the model is introduced. Task history added to the baseline table (@sgrebnov). It is an in-memory accelerated table, so its footprint is task rate x `retention_period` (8h default) rather than a fixed allocation, and `captured_plan` / `captured_output` inflate every record. Recommends shortening retention over disabling it — it backs `runtime.task_history`, the primary tool for diagnosing exactly what this page is about. The cache-sizing example gains a warning that it is a planning floor, not a prediction: it sums only what can be named in advance and so under-estimates actual usage. It omits traffic-driven buffer growth, task history, allocator retention, and transient peaks, and the query memory limit in it is a ceiling rather than a reservation. Not to be used as the container limit. `max_concurrent_queries` now states the trade-off in both places it appears: it reduces peak MEMORY, and raises peak QUERY TIME, because excess queries wait for admission and that wait counts toward end-to-end duration. Says to measure total duration, not execution time. Circuit breakers added to client resiliency: retries handle isolated failures and make sustained ones worse, so after N consecutive failures stop traffic for a cooldown and probe before restoring — noting this usually belongs at the load balancer or mesh, where the decision is shared across callers rather than relearned by each. The "a plateau is not a leak" claim now appeals to the allocators' own documentation rather than asserting it: jemalloc's page retention and decay schedule, glibc's `M_TRIM_THRESHOLD`, and the cgroup v2 docs for why RSS is nonetheless what `memory.max` charges. All three added to External References.
…tory is time-bounded Two precision fixes on the review changes. The Baseline bullet still said the baseline is a function of dataset count alone, which the refinement directly below it now contradicts — task history scales with task rate and the buffers with traffic served. It now names both drivers. The task history row read "Unbounded by size", which is literally true (there is no byte cap) but invites the reading that it grows without limit. It is bounded by time; the row now says so.
Contributor
|
🚀 deployed to https://56174974.spiceai-org-website.pages.dev |
Contributor
|
🚀 deployed to https://7148aeb8.spiceai-org-website.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The memory guide gave sizing rules as multiples of dataset size plus a minimum RAM figure, with no account of the fixed cost those multiples sit on top of. That reads as memory scales with data, which makes a proportionally downscaled staging environment look like a valid model of production. It is not one, and nothing in the docs warned anyone off it.
Prompted by a support conversation about a deployment being OOM-killed in a small container; the guidance here is written generally, and every number is checked against the shipped runtime.
The reframing
Total = Baseline + Working Set. The baseline is a function of dataset count, not data volume — each Cayenne table holds a segment cache (1/128 of memory, clamped to 256 MB–1 GB), and a CDC table adds a PK keyset cache (1/32, clamped 256 MiB–8 GiB) and a coalesce buffer.The clamps are the point: the per-dataset cost stops shrinking once the environment is small, so it is a rounding error at production volumes and the dominant term below them. A worked table shows the same ten-dataset Spicepod at ~76% baseline in 4 GiB and ~9% in 32 GiB. The existing multiples are relabelled as working-set figures rather than totals.
Four things from the runtime the docs did not carry
Each of these changes what an operator should actually do:
How the Runtime Partitions Memory— the 70/20/10 partition (pools / in-memory CDC tier / off-pool headroom), the compaction carve out of the query pool, and the 50% floor.Tuning the Memory Limit Safely— the limit is bounded on both sides when CDC is active, and both directions are counterintuitive. Lowering it need not reduce resident memory: the tier is sized from what is left after the pools and floats up to a quarter of memory to reclaim it, so the memory moves rather than releasing. Raising it squeezes the tier below its floor into a refuse-all gate and ingestion falls back to disk. Hence: leave it unset, size the container.Bounding Peak Memory with Concurrency—max_concurrent_queriesdefaults to 4× the CPU entitlement's cores andtarget_partitionsto the cores; neither consults the memory limit. With the entitlement at twice the pod's CPU request, arequests.cpu: 4pod admits 32 concurrent queries however little memory it has. This is the better first lever for peak memory — it lowers the peak, where loweringmemory_limitonly shrinks each query's share of it.Check the Startup Budget Warnings First— the partition is computed before any data loads and warns when it cannot fit, atwarnlevel. An unfittable configuration is detectable in the first seconds of a run rather than after days of load, so it belongs ahead of load testing. Four warnings tabulated with meaning and action. (The source comment on the first records a ~122 GiB host OOM-killed at SF-1000 whose only trace was that line.)Also added:
Reading a Memory Graph(a flat high plateau is caches at their ceilings and allocator retention, not a leak),Validating a Memory Configuration(the four properties that make a load test predictive — data volume, peak concurrency, duration to steady state, predicate diversity — plus shadow/canary), andClient-Side Resiliency(retry in-cluster before falling back; instance loss is independent of memory tuning).Accuracy fixes found on the way
cayenne_segment_cache_mbwas documented as a flat256default. It is auto-derived — 256 MB is the floor, and it reaches 1 GB above a 32 GiB host.performance-tuning.md, which already had it right.Supporting pages
The levers are surfaced where the problem is actually hit:
performance-tuninggets a which-knob-for-which-intent table,deployment/kubernetesgets a Memory sizing section beside the CPU one,troubleshootinggets an OOM-despite-a-configured-limit entry.Verification
cd website && npm run buildpasses — the config throws on broken links and anchors, so the build gates every cross-reference added here.Constants checked against the shipped source: the 90/70/50 percentages, the 20% compaction carve, the tier's 1/5 ceiling, 1/32 floor and 1/4 float, the two cache clamps, the 128 MiB coalesce buffer,
QUERIES_PER_CORE = 4, andtarget_partitions = cores.Reviewer notes
Two things worth a maintainer's eye:
Trunk
website/docs/only, so this lands as Next and flows into the next release; no versioned snapshots touched.