[ML] pytorch_inference OOM-killed (~34 GiB RSS) during ELSER wikipedia-1g reindex in nightly QA (since build 975)

Summary

The nightly custom-ml-cpp QA suite step family/appex-qa-debian-12 [pytorch_tests] has failed every night since build 975 (24 Sep 2026) on a single scenario: test_scenario_inference.py::TestScenarioInference::test_scenario_nightly[wikipedia-1g]. The last green run was build 974 (23 Sep 2026).

Root cause is a host out-of-memory kill of the ELSER pytorch_inference process, not a controlled stop. On an identical 64 GB / swap-less Debian-12 box running the identical workload, build 974 completed the ~2.7 h ELSER reindex with zero OOM kills, whereas the failing builds consistently drive the ELSER process to ~34 GiB RSS and it gets reaped by the kernel mid-reindex. The in-flight/queued inference requests then fail and the test (which asserts 0 failures) goes red.

This looks like a memory-pressure regression introduced between 23 → 24 Sep. Attribution (ml-cpp/libtorch footprint vs. ES adaptive-allocation sizing) is still open — see next steps.

Evidence

Kernel dmesg, build 981 (logs.tar.gz):

[Tue Sep 29 06:56:00 2026] Out of memory: Killed process 7607 (pytorch_inferen)
    total-vm:37466804kB, anon-rss:36124772kB ... oom_score_adj:667
oom-kill:constraint=CONSTRAINT_NONE ... global_oom ... task=pytorch_inferen,pid=7607

(anon-rss 36,124,772 kB ≈ 34.5 GiB; box = 67,105,128 kB ≈ 64 GiB, Total swap = 0kB. Three co-resident pytorch_inference procs: ~34.5 GiB + ~7.1 GiB + ~0.55 GiB.)

ES side correlates exactly (build 981, node server log):

[controller/5506] Child process with PID 7607 was terminated by signal 9 (SIGKILL). This is likely due to the OOM killer.
[.elser_model_2_linux-x86_64] inference process crashed ... pytorch_inference/7607 process stopped unexpectedly
[.elser_model_2_linux-x86_64] clearing [12] requests pending results
[inference process] notifying [30] queued requests that have not been processed before shutdown

→ 12 pending + 30 queued = the 42 doc failures the test reports.

Reproduced with the same signature on build 980: pytorch_inferen pid 7664 OOM-killed at anon-rss 34,630,404 kB (≈ 33 GiB), global OOM.

Build history (QA pipeline appex-qa-stateful-custom-ml-cpp-build-testing):

Build	Date	pytorch_tests	ELSER OOM
970–974	19–23 Sep	passed	none
975–978, 980, 981	24–29 Sep	failed	pytorch_inference ~33–34.5 GiB

Not related

Independent of the controller-protocol.version packaging fix ([ML] Publish controller-protocol.version in Linux and DRA packaging #3224 / Prune controller-protocol.version from -deps bundle #3225).
The truncated Fatal error: '[W929 ... Quantizer.cpp:111] Warning: torch.quantize_per_tensor ... deprecated' line in the ES log is a libtorch deprecation warning captured from the process's stderr at startup — a red herring; the actual cause is SIGKILL from the OOM killer.
Side note (not causal): in these runs the controller routes pytorch_inference via the legacy path because the --disableSandbox operator kill switch is set (legacy_reason":"kill_switch", sandbox2_established:false), so Sandbox2 is currently disabled in this QA lane.

Next steps / investigation

Confirm attribution by bisecting the ml-cpp / libtorch change merged around 23 → 24 Sep (same QA harness across the pass→fail boundary, so the change is in the nightly-rebuilt artifact and/or the ES snapshot under test).
Capture the periodic pytorch_inference RSS report (ml-cpp#3160) at a log level that survives into the QA artifacts, so peak footprint is directly observable in future runs.
Check whether the ES snapshot's ML adaptive-allocations / memory sizing changed in that window (could raise the allocation count and hence RSS).
If footprint growth is expected/unavoidable, revisit the QA box sizing (add swap / more RAM) or cap concurrent deployments for the nightly scenario — but only after (1) rules out a genuine footprint regression.

Artifacts

Failing: ml-cpp-pr-builds #3259 → QA #981, also [7.x][ML] Use largest ordered subset of categorization tokens for regex #980.
Last green: QA [ML] Return total SHAP per feature as a new result type #974.

Bisection update — RSS reporting (#3160) ruled out

Correlating the ml-cpp-pytorch-builds nightly commits with the QA pass/fail boundary:

Nightly	Date	ml-cpp commit	QA result
828	23 Sep 03:00	`beee73be` (= #3160 merge)	QA #974 passed
829	24 Sep 03:00	`12c3ebfa`	QA #975 failed (first red)

So #3160 "Periodically report pytorch_inference RSS" (merged 22 Sep, beee73be) shipped in the last green build and is not the regression.

The ml-cpp source delta beee73be..12c3ebfa is only two commits:

19f76ea2 — [ML] Do not treat third-party warnings as errors #3208 Do not treat third-party warnings as errors (build-flag only; very unlikely to affect runtime memory)
12c3ebfa — [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 Linear child ownership and fault-injected lifecycle for Sandbox2 (edits CSandboxedProcessSpawner — the child spawn/reap path)

Leading suspect: #3187. It fits the evidence that three pytorch_inference processes were co-resident at OOM (34.5 + 7.1 + 0.55 GiB) — a child-ownership/reaping regression leaking/orphaning inference processes would accumulate resident memory until the 64 GB box OOMs.

Caveats before calling it confirmed:

The runtime path in these QA runs is the legacy spawner (--disableSandbox kill switch on → Launching './pytorch_inference' without Sandbox2), whereas [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 only edits lib/sandbox/. Need to check whether [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 altered shared child-ownership/reaping logic that the legacy CDetachedProcessSpawner path also depends on.
The nightly rebuilds the PyTorch Docker image, so a libtorch / base-image / dependency bump in build 829's image is an equally-plausible cause independent of the two source commits.

Next: verify whether the co-resident processes are orphans (reaping regression) vs. expected allocations, and confirm/deny a libtorch delta between the 828 and 829 images.

libtorch moved at the pass→fail boundary — now the leading suspect

The nightly image (build_pytorch_linux_build_image.sh) does not build the pinned v2.7.1; it compiles pytorch_inference against the moving viable/strict PyTorch branch, rebuilding whenever that branch HEAD moves. Pulling the PyTorch SHA baked into each nightly:

Nightly	ml-cpp commit	PyTorch `viable/strict`	QA result
828 (23 Sep)	`beee73be`	`ef7af4063de3047e2b39938d53529e84058f0535`	#974 passed
829 (24 Sep)	`12c3ebfa`	`4de991c72a8c0c3b9c1646b5865d67b215c33e85`	#975 failed

So the red build ran a different libtorch than the last green one — 116 PyTorch commits in ef7af406..4de991c7. The range includes memory/allocator-touching commits (e.g. inductor autotuning allocation aliasing and its revert, "Allocate receive buffers just in time", memory-timeline fusion guard), though most are inductor/AOTI/compile-time paths not exercised by CPU ELSER inference — no single obvious CPU-inference smoking gun, but a large surface.

Implication: the regression is now most likely a libtorch bump, not the ml-cpp source delta (#3187 / #3208). This also sits better with the earlier observation that the runtime path was the legacy spawner (Sandbox2 disabled), which #3187 doesn't touch.

Decisive next experiment: hold the ml-cpp source constant and vary only libtorch — build pytorch_inference at the same ml-cpp commit against ef7af406 vs 4de991c7 and run the wikipedia-1g ELSER reindex against each, watching peak RSS. That cleanly separates libtorch from the ml-cpp commits. If libtorch is exonerated, fall back to auditing whether #3187 changed shared child-ownership/reaping logic used by the legacy CDetachedProcessSpawner path.

Green-arm libtorch A/B triggered

To decide between the two boundary hypotheses (libtorch bump vs the ml-cpp delta), I've launched a single green-arm re-run: hold the ml-cpp source at the first red nightly commit and swap only libtorch back to the last-green SHA.

Scratch branch experiment/libtorch-ab-3226 off 12c3ebfa — which is exactly the [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 merge ("Linear child ownership and fault-injected lifecycle for Sandbox2"), i.e. the ml-cpp delta at the pass→fail boundary.
Tooling change (CI-only, nightly behaviour unchanged when unset):
- Dockerfile: new pytorch_pin arg → fetch/checkout an exact pytorch_commit instead of the moving viable/strict tip.
- build_pytorch_linux_build_image.sh: PYTORCH_COMMIT_OVERRIDE (pins SHA, bypasses skip-if-unchanged) + IMAGE_TAG_OVERRIDE (distinct tag, doesn't clobber pytorch_latest).
- build_pytorch_docker_image.yml.sh: flows the pinned tag to the downstream DOCKER_IMAGE so the QA run pulls the pinned image.
Build: ml-cpp-pytorch-builds #836, env PYTORCH_COMMIT_OVERRIDE=ef7af4063de3047e2b39938d53529e84058f0535, IMAGE_TAG_OVERRIDE=pytorch_ab_ef7af406. Builds libtorch @ green SHA, then runs the pytorch_tests QA suite (wikipedia-1g ELSER reindex).

Read-out:

Passes → ml-cpp 12c3ebfa (incl. [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187) is fine; the OOM regression is the libtorch bump ef7af406 → 4de991c7. Next: bisect the 116-commit libtorch range.
Still OOMs → libtorch exonerated; the culprit is the ml-cpp delta ([ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 / [ML] Do not treat third-party warnings as errors #3208) — pivot to the [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 shared-reaping audit.

Cleanup afterwards: delete the experiment/libtorch-ab-3226 branch and the throwaway pytorch_ab_ef7af406 image tag.

libtorch source scan (ef7af406..4de991c7, 116 commits / 512 files): no RSS-relevant source change

Full-coverage audit of the range (paged per-commit to defeat GitHub's 300-file compare cap → 512 unique files):

0 changes to CPU allocator / caching allocator / arena (c10/core/CPUAllocator*, Allocator*).
0 oneDNN / MKLDNN / ideep / MKL source changes.
0 Storage / StorageImpl / resize-realloc changes.
0 ParallelNative / ParallelOpenMP / thread-pool-sizing / OpenMP-default changes (only test files).
0 c10/util intrusive_ptr / SmallVector-growth changes.
c10/core/TensorImpl.* edits (3 commits) are FakeTensor-only — extend the lazily-allocated ExtraMeta/FakeTensorMode; sizeof(TensorImpl) unchanged, fires only under tracing, not eager inference.
CPU-math edits (#197419, #198207) are gated behind AT_VEC_CUSTOM_MATH/FBCODE_CAFFE2 — dead code in an OSS libtorch x86_64 build. s390x vector work (#173072) doesn't apply to x86_64.
Only genuine eager-kernel change is #195787 (index_put_→index_fill_, lower grain size) — alters task granularity, not steady-state RSS.

Implication: a libtorch source memory regression is unlikely. If libtorch is nonetheless the cause, it's more likely a build/config/submodule difference (submodule bump, MKL/oneDNN version, OpenMP thread count, allocator env). The green-arm experiment #836 still discriminates this: checking out ef7af406 reverts pytorch source and its pinned submodules, while the Dockerfile holds MKL (2024.0), compiler and thread settings constant. So:

Outlier memory usage code adds negative of unsigned number #836 passes → the delta is a libtorch build/submodule/config difference (not a source commit; bisect submodules/build).
Outlier memory usage code adds negative of unsigned number #836 OOMs → libtorch fully exonerated → ml-cpp [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187/[ML] Do not treat third-party warnings as errors #3208 (awaiting the [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 reaping audit).

Complementary ml-cpp green-arm triggered — 2×2 factorial now in flight

Added the orthogonal control to isolate the other variable (holding libtorch ~fixed, reverting ml-cpp):

Branch experiment/mlcpp-beee73be-3226 off beee73be — the [ML] Periodically report pytorch_inference RSS #3160 (RSS reporting) merge, i.e. the last-green nightly's ml-cpp commit (pre-[ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187).
Reuses the nightly pytorch_latest image (latest viable/strict) — no libtorch rebuild, no clobber (skip-if-unchanged), then runs pytorch_tests (wikipedia-1g).
Build: ml-cpp-pytorch-builds #837.

Full matrix:

	libtorch green `ef7af406`	libtorch red/latest
ml-cpp green `beee73be` (pre-#3187)	nightly 828 → PASS	#837 (running)
ml-cpp red `12c3ebfa` (#3187)	#836 (running)	nightly 829 → FAIL

Joint read-out:

[ML]Documented developing, building and running the ml-cpp project #837 PASS + Outlier memory usage code adds negative of unsigned number #836 PASS → interaction (needs both changes).
[ML]Documented developing, building and running the ml-cpp project #837 FAIL + Outlier memory usage code adds negative of unsigned number #836 PASS → libtorch sole driver.
[ML]Documented developing, building and running the ml-cpp project #837 PASS + Outlier memory usage code adds negative of unsigned number #836 FAIL → ml-cpp [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 sole driver.
both FAIL → two independent regressions.

Caveat: #837 uses latest viable/strict (newer than the red 4de991c7), so a #837 PASS is slightly ambiguous (ml-cpp-required vs an upstream libtorch fix landed since 4de991c7); a #837 FAIL is unambiguous. Can re-run #837 pinned to 4de991c7 if we need it airtight.

Corroborating context: the libtorch source scan found no RSS-relevant source change in ef7af406..4de991c7, which lowers the prior on the pure-libtorch-source explanation and slightly favours the #3187 / interaction branches.

Upstream search + allocator grounding: mimalloc regression ruled out; glibc-arena is the cheap lever

Searched PyTorch issues/forums for a matching upstream report. No "viable/strict OOM/leak" issue exists (consistent with a manual Google search), but there is a strongly on-theme, recent culprit class for our exact symptom (freed memory not returned → host OOM, tens of GiB RSS):

mimalloc arena retention — #178726 (2.9→2.10, "5× RAM", host OOM), from mimalloc-by-default #164741; mitigations MIMALLOC_ARENA_RESERVE=0 / mi_option_purge_delay=0, fix PR #178851, new torch.cpu.empty_cache() PR #192078.
oneDNN-upgrade CPU regressions #169964.
Long-standing MKL/oneDNN multi-thread inference leak #64412, #121485 — proven fix jemalloc / MALLOC_ARENA_MAX.

Why the mimalloc story does NOT apply to us:

Both boundary builds are PyTorch 2.15.0a0 — identical allocator config; mimalloc default predates both.
No allocator/build change exists in ef7af406..4de991c7 (source scan).
USE_MIMALLOC is gated WIN32 OR (CPU_AARCH64 AND NOT APPLE) in PyTorch CMakeLists.txt; our failing host is x86_64 Linux → glibc, not mimalloc.

Actionable takeaway. Our allocator is glibc, whose classic failure mode is per-thread arenas + non-return of freed pages → large RSS when thread count / alloc pattern shifts. This is cheap to test and plausibly links to #3187 (child/process/thread lifecycle): if #3187 changed pytorch_inference's thread/fork behaviour, glibc arena RSS could balloon into OOM with no leak and no libtorch change.

Planned probe (cheaper than a libtorch bisect), gated on #836/#837: re-run the failing config with MALLOC_ARENA_MAX=1 (and/or MALLOC_TRIM_THRESHOLD_=0), or LD_PRELOAD=jemalloc. If the OOM vanishes → glibc-arena/fragmentation, likely tickled by a thread/lifecycle change (→ candidate fix: set arena cap at pytorch_inference launch / mallopt(M_ARENA_MAX)). Open design point: the env var must reach the controller-spawned pytorch_inference via the ES node environment in the QA harness — need to confirm the QAF harness propagates it.

Is Elasticsearch a factor? Third variable confirmed, but no smoking gun in the delta

The pytorch_tests QA builds ES from source at a specific commit, and that commit moved between the green and red nightly QA runs (both nominally main / 9.6.0-SNAPSHOT):

QA run	date	ES commit	result
appex-qa 974	Sep 23	`1e7276a414be9bb111a184b6a714d406ae15ee8d`	passed
appex-qa 975	Sep 24	`40cc061e4a48ddc10a1296624179452277d4f09e`	failed

So ES is a genuine third moving variable at the boundary (alongside ml-cpp #3187/#3208 and libtorch) — and it is not controlled by experiments #836/#837, which both float on today's snapshot.

Diff of 1e7276a4..40cc061e = 68 commits. Scanning all 68 subjects + changed files for anything on the ELSER / pytorch_inference path (native-process management, trained-model deployment memory, inference queue/concurrency, ML autoscaling) found nothing relevant:

[Native] commits (#159442, #159649) are Lucene int8 kNN vector scoring — not the ML native process.
inference/EndpointMetadata + capabilities CSV are inference-API metadata, not memory/concurrency.
painless/AllocationEstimators (painless heap accounting), WriteLoadConstraintDecider (shard allocation) — unrelated to ML native memory.

Caveat: the numeric ELSER deployment settings (allocations / threads-per-allocation / queue depth / model memory) are not in the Buildkite log — they live in the ES server logs / QAF scenario config — so an ES-driven workload change can't be 100% excluded, only that the ES code delta shows no such change.

Conclusion: ES is uncontrolled but shows no plausible OOM driver in the Sep 23→24 delta. To neutralise it, pin ES across the arms (re-run pinned to red ES 40cc061e) if #836/#837 come back ambiguous.

#3187 reaping audit: does NOT touch the legacy CDetachedProcessSpawner path

Question: did #3187 refactor shared child-reaping / ownership logic also used by the legacy CDetachedProcessSpawner (the path in use when Sandbox2 is disabled)?

Answer: No.

Every file [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187 changed is under lib/sandbox/ — CSandboxedProcessSpawner_Linux.cc, its header, CMake, and unit tests. It touches zero files in lib/core/.
The legacy CDetachedProcessSpawner has its own self-contained reaper: ::waitpid(-1, &status, WNOHANG) at lib/core/CDetachedProcessSpawner.cc:170 — unchanged by [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187.
lib/core contains no reference to sandbox/ or CSandboxedProcessSpawner (grep is empty). Per the layered dependency rule (core is lower than sandbox), core cannot depend on sandbox, so [ML] Linear child ownership and fault-injected lifecycle for Sandbox2 #3187's changes are physically unreachable from the legacy path.

Implication: the failing QA runtime used the legacy spawner (Sandbox2 disabled via --disableSandbox), so #3187's code does not execute there — which effectively exonerates #3187 as the OOM cause, conditional on the legacy-path fact. Live candidates now: #3208 (build flags), the libtorch build/config difference, or an ES/interaction effect — to be discriminated by #836 (ml-cpp #3187 + green libtorch) and #837 (pre-#3187 + latest libtorch).

⚠️ Experiments #836/#837 are VOID — never reached the OOM workload

Both downstream QA runs (appex-qa #982, #983) failed early, during the local ES distribution build, at:

> Task :x-pack:plugin:ml:verifyControllerProtocolVersion FAILED
Local Elasticsearch distribution build failed

Zero wikipedia/reindex/pytorch_inference/OOM lines in either log — they died at deployment creation, long before the workload.

Root cause: the experiment branches are old ml-cpp commits (beee73be Sep 22, 12c3ebfa Sep 23) built against today's ES main. Today's ES enforces a controller protocol version those old artifacts don't satisfy (the verifyControllerProtocolVersion gate from #3224/#3225). Old ml-cpp + new ES is rejected outright.

Consequence — ES is a compatibility constraint, not just a variable: you cannot float ES against old ml-cpp; the two are version-coupled via verifyControllerProtocolVersion. Both arms floated on today's ES, so they were void regardless of libtorch/ml-cpp.

Revised plan — pin ES to a contemporaneous commit. The decisive libtorch test becomes: hold ml-cpp = 12c3ebfa (red) and ES = 40cc061e (the red Sep-24 ES commit that ran the red nightly 829) fixed, and vary only libtorch red→green:

Control: 12c3ebfa + libtorch 4de991c7 + ES 40cc061e → should reproduce the OOM (829).
Test: 12c3ebfa + libtorch ef7af406 + ES 40cc061e → if it passes, libtorch is the cause; if it OOMs, libtorch exonerated (→ [ML] Do not treat third-party warnings as errors #3208 build flags / ES-workload / interaction).

Open question: whether the QAF harness accepts an ES commit SHA (not just a branch) for pinning — derive_qa_stack_env.sh treats ES_BRANCH as a branch name; the actual checkout is QAF-side. Need to confirm before re-running.

[ML] Thread ES_COMMIT through QA triggers for pinned-ES A/B (#3226)

QAF supports pinning ES to a commit (ES_COMMIT) — valid re-run launched

Confirmed the harness supports it: qaf's ElasticsearchRepoTask.Settings exposes commit via the ES_COMMIT env var (typer option envvar="ES_COMMIT"); when set, update_repo() checks out that exact ES commit instead of the branch tip.

Buildkite downstream builds don't inherit env automatically, so I threaded ES_COMMIT (when set) through both trigger layers on experiment/libtorch-ab-3226 (commit 06df572bf): pytorch-image pipeline → ml-cpp-pr-builds, and run_pytorch_tests → appex-qa. Nightly behaviour unchanged when unset.

Valid libtorch A/B re-launched, ml-cpp fixed at 12c3ebfa, ES pinned to 40cc061e (the red Sep-24 commit, contemporaneous → passes verifyControllerProtocolVersion):

Build	libtorch	ES	expectation
#840 CONTROL	red `4de991c7`	`40cc061e`	reproduce OOM (≈829)
#839 TEST	green `ef7af406`	`40cc061e`	decisive

Read-out: CONTROL OOMs + TEST passes → libtorch is the cause. Both OOM → libtorch exonerated → #3208 build flags / ES-workload / glibc-arena. CONTROL passes → OOM is nondeterministic/flaky near capacity (re-assess).

Latest scheduled nightly (#838, Sep 30) did NOT OOM — QA passed

Nightly #838 → QA appex-qa 984 passed: ran the full wikipedia-1g reindex (~6h), 55 passed, 1 skipped, 0 failed, no OOM/kill.
[7.x] Emit prediction_probability field alongside prediction field in ml results. (#818) #838 is red only due to unrelated failures — a Java Multi-Node integration test and the build-timing-regression check — not the pytorch QA.

Pivot: the OOM has stopped reproducing. Confirmed OOM was Sep 24 (#829 / QA 975); by Sep 30 the same workload passes. Likely transient — tied to a specific libtorch/ES combination that both viable/strict and ES main have since moved past — or flaky near capacity. The intervening nightly failures #830–#837 were most likely the verifyControllerProtocolVersion gate (pre-#3224/#3225), not OOM.

Experiment status:

TEST arm [[ML] Re-added Boost program options to model_extractor Makefile #839] was skipped by Buildkite skip_queued_branch_builds (queued [ML] Remove unary_function/binary_function and old-style binders #840 on the same branch a second later).
[[ML] Remove unary_function/binary_function and old-style binders #840 CONTROL] (red libtorch 4de991c7 + red ES 40cc061e + ml-cpp 12c3ebfa) is now the gating test — does the Sep-24 combo still deterministically OOM?
- [ML] Remove unary_function/binary_function and old-style binders #840 OOMs → reproducible → re-run green-libtorch TEST (fresh branch) + bisect.
- [ML] Remove unary_function/binary_function and old-style binders #840 passes → OOM was nondeterministic/transient near capacity → likely a headroom/capacity issue, not a code regression.

Holding the TEST re-trigger until #840 reports.

[ML] Thread ES_COMMIT through the QA-tests trigger generator (elastic…

Investigation outcome: transient PyTorch `viable/strict` regression, not ml-cpp

Reconstructed the full nightly → QA → {ml-cpp, libtorch} map from node logs, dmesg, and the PyTorch image-build logs. The headline: the OOM is caused by PyTorch viable/strict drift, not ml-cpp, and it has already self-resolved upstream.

Evidence map

QA	Date	ml-cpp	libtorch (viable/strict)	Outcome
974	Sep 23	`beee73be`	`ef7af406` (Sep 22)	✅ pass
975	Sep 24	`12c3ebfa`	—	❌ 30s ELSER redeploy timeout (not OOM; wikipedia-1g passed)
976	Sep 25	`16c5935bb`	—	❌ 30s ELSER redeploy timeout (not OOM; wikipedia-1g passed)
977	Sep 26	`e3ae4842`	—	⚫ void — `verifyControllerProtocolVersion` gate
978	Sep 27	`e3ae4842`	—	⚫ void — `verifyControllerProtocolVersion` gate
980	Sep 28	`b0c52aa4`	`4b0647ed` (Sep 27)	🔴 OOM — pytorch_inference ~34.6 GiB, SIGKILL
981	Sep 29	`b0c52aa4`	`f2398806` (Sep 28)	🔴 OOM — pytorch_inference ~34.5 GiB, SIGKILL
984	Sep 30	`b0c52aa4`	`ddae06a3` (Sep 29)	✅ pass

Decisive observation

Holding ml-cpp constant at b0c52aa4 across three consecutive nights, only libtorch changes:

libtorch 4b0647ed → OOM
libtorch f2398806 → OOM
libtorch ddae06a3 → pass

Same ml-cpp binary → 🔴 🔴 ✅. This exonerates ml-cpp and pins the OOM on a PyTorch viable/strict memory regression that entered after ef7af406 (Sep 22), was present at 4b0647ed/f2398806 (Sep 27–28), and was fixed by ddae06a3 (Sep 29).

The OOM signature: pytorch_inference RSS grows to ~34–35 GiB over the ~1.5h wikipedia-1g ELSER reindex and is host-OOM-killed (CDetachedProcessSpawner.cc@223: ... terminated by signal 9 (SIGKILL). This is likely due to the OOM killer; corroborated by dmesg: Out of memory: Killed process ... (pytorch_inferen) anon-rss:36124772kB). Route is legacy throughout (so #3187 / Sandbox2 is not involved — it is not on the executed path).

Corrections to the original report

The title/premise "OOM since build 975" is inaccurate. 975 and 976 did not OOM — they hit an unrelated benign 30s ELSER redeploy timeout (model_deployment_timeout_exception), with wikipedia-1g passing. The genuine OOM is only 980/981 (Sep 28–29).
977/978 failed earlier at the verifyControllerProtocolVersion build gate (ES main had advanced past the pre-[ML] Publish controller-protocol.version in Linux and DRA packaging #3224/Prune controller-protocol.version from -deps bundle #3225 ml-cpp), never reaching the workload.

Recommendation

No ml-cpp change required. The regression is upstream PyTorch and already fixed in viable/strict. Suggest closing once a couple of clean green nightlies confirm stability (984/Sep 30 is already green on the same ml-cpp b0c52aa4). A full QA bisect of the libtorch window was considered but not pursued — it would cost ~7× (image build + ~6h QA) to localise a bug that is already gone.

Side benefit

The ability to pin ES to a commit SHA for these experiments has been factored into a standalone PR: #3228 (ES_COMMIT passthrough to the appex-qa trigger).

Closing: root cause is a transient upstream PyTorch regression, no ml-cpp action required

Per the investigation above, the OOM is not an ml-cpp defect. Holding ml-cpp constant at b0c52aa4 across three consecutive nightlies, the result tracks solely with the PyTorch viable/strict libtorch build:

libtorch 4b0647ed (Sep 27) → OOM (QA 980)
libtorch f2398806 (Sep 28) → OOM (QA 981)
libtorch ddae06a3 (Sep 29) → pass (QA 984)

The regression entered viable/strict after ef7af406 (Sep 22, good), was present at 4b0647ed/f2398806, and was fixed by ddae06a3. The subsequent nightly (984, Sep 30) is green on the same ml-cpp binary, confirming recovery.

Also note the earlier framing was imprecise: nightlies 975/976 were an unrelated benign 30s ELSER redeploy timeout (not OOM), and 977/978 failed at the verifyControllerProtocolVersion build gate. The genuine OOM is confined to 980/981.

Closing as not planned (external/transient, already resolved upstream). If a future nightly OOMs again we can reopen and bisect the libtorch window with the new ES_COMMIT pinning support (#3228) to keep the ES side reproducible.

Reopening — the OOM recurred; the "transient/fixed" conclusion was wrong

The Oct-1 nightly (appex-qa #987) OOM-killed pytorch_inference again, at ~40.6 GiB RSS (Out of memory: Killed process ... (pytorch_inferen) anon-rss:40650312kB), surfacing as wikipedia-1g: inference task failure count. Same ml-cpp b0c52aa4; libtorch bb63328c (viable/strict, Oct 1 — newer than the ddae06a3 that passed on Sep 30).

Corrected sequence, ml-cpp constant at b0c52aa4:

QA	Date	libtorch	Outcome	RSS
980	Sep 28	`4b0647ed`	OOM	34.6 GiB
981	Sep 29	`f2398806`	OOM	34.5 GiB
984	Sep 30	`ddae06a3`	pass	(under ceiling)
987	Oct 1	`bb63328c`	OOM	40.6 GiB

That's 3 OOMs in 4 nightlies across 4 different libtorch commits. The Sep-30 pass now looks like it merely stayed under the 64 GiB ceiling rather than a real fix. So this is not a cleanly-fixed transient libtorch regression — it's a persistent high-RSS condition (~34–40 GiB and trending up) in the b0c52aa4 era that nondeterministically tips the box.

Next step

Isolate ml-cpp vs libtorch with a single-variable A/B: hold libtorch fixed (current bb63328c) and vary ml-cpp — green-era beee73be vs OOM-era b0c52aa4, both against the same pinned, contemporaneous Elasticsearch (so the verifyControllerProtocolVersion gate passes for both) — and compare peak pytorch_inference RSS. ES pinning uses the new ES_COMMIT support (#3228).

[ML] RSS probe: make distribution download/unzip cwd-independent (ela…

[ML] RSS probe: move probe body into committed script to survive pipe…

[ML] RSS probe: variable-length requests + heavier parallel load to c…

Nightly OOM history (ml-cpp × libtorch)

Each nightly QA run pulls ml-cpp main HEAD plus that night's libtorch, built from PyTorch viable/strict. Correlating the two:

Date	QA run	Nightly	ml-cpp	libtorch (`viable/strict`)	OOM?	`pytorch_inference` peak RSS
Sep 23	974	828	`beee73be`	`ef7af406` (Sep 22)	No	completed cleanly
Sep 24	975	829	`12c3ebfa`	—	No	failed, but redeploy 30s timeout (wikipedia-1g passed)
Sep 25	976	830	`16c5935bb`	—	No	failed, but redeploy 30s timeout (wikipedia-1g passed)
Sep 26	977	831	`e3ae4842`	—	Void	`verifyControllerProtocolVersion` gate (no QA)
Sep 27	978	832	`e3ae4842`	—	Void	proto gate (no QA)
Sep 28	980	834	`b0c52aa4`	`4b0647ed` (Sep 27)	Yes	34.6 GiB (SIGKILL)
Sep 29	981	835	`b0c52aa4`	`f2398806` (Sep 28)	Yes	34.5 GiB (SIGKILL)
Sep 30	984	838	`b0c52aa4`	`ddae06a3` (Sep 29)	No	stayed under ceiling
Oct 1	987	—	`b0c52aa4`	`bb63328c`	Yes	40.65 GiB (SIGKILL)

Key observations

The OOMs begin at ml-cpp b0c52aa4 (merged ~Sep 28). Everything before it (beee73be, 12c3ebfa, 16c5935bb) either passed or failed for unrelated reasons (ELSER redeploy 30s timeout, controller-protocol gate void) — none were OOMs.
Holding ml-cpp constant at b0c52aa4, the outcome tracks libtorch: 4b0647ed → OOM, f2398806 → OOM, ddae06a3 → pass, bb63328c → OOM. So it is not purely ml-cpp or purely libtorch; there is at least an interaction, and ddae06a3 is the one OOM-era libtorch that did not blow up.
That ddae06a3 pass is the counter-example that keeps both suspects live and motivates the isolation testing below.

Caveats

The Sep 24–27 libtorch SHAs were not pinned (those nights did not OOM).
beee73be is the last known-good ml-cpp and is used as the "green ml-cpp" arm in the isolation runs.

Isolation testing in progress

A standalone RSS probe (no Elasticsearch) drives pytorch_inference directly with the public ELSER model and a synthetic variable-length BERT request stream, sampling /proc/<pid> RSS, as a throwaway step on the Linux x86_64 build agents (64 GiB). It reuses already-built distributions (no recompile), so single-variable comparisons run in minutes instead of the ~6h QA cycle. Currently comparing identical ml-cpp b0c52aa4 against bb63328c vs ee0d01e8 libtorch under a heavy variable-length load.

[ML] RSS probe: fix RssWatcher._stop shadowing Thread._stop (elastic#…

Navigation Menu

Repository navigation

[ML] pytorch_inference OOM-killed (~34 GiB RSS) during ELSER wikipedia-1g reindex in nightly QA (since build 975) #3226

Summary

Evidence

Not related

Next steps / investigation

Artifacts

Bisection update — RSS reporting (#3160) ruled out

libtorch moved at the pass→fail boundary — now the leading suspect

Green-arm libtorch A/B triggered

libtorch source scan (ef7af406..4de991c7, 116 commits / 512 files): no RSS-relevant source change

Complementary ml-cpp green-arm triggered — 2×2 factorial now in flight

Upstream search + allocator grounding: mimalloc regression ruled out; glibc-arena is the cheap lever

Is Elasticsearch a factor? Third variable confirmed, but no smoking gun in the delta

#3187 reaping audit: does NOT touch the legacy CDetachedProcessSpawner path

⚠️ Experiments #836/#837 are VOID — never reached the OOM workload

QAF supports pinning ES to a commit (ES_COMMIT) — valid re-run launched

Latest scheduled nightly (#838, Sep 30) did NOT OOM — QA passed

Investigation outcome: transient PyTorch `viable/strict` regression, not ml-cpp

Evidence map

Decisive observation

Corrections to the original report

Recommendation

Side benefit

Closing: root cause is a transient upstream PyTorch regression, no ml-cpp action required

Reopening — the OOM recurred; the "transient/fixed" conclusion was wrong

Next step

Nightly OOM history (ml-cpp × libtorch)

Key observations

Caveats

Isolation testing in progress

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[ML] pytorch_inference OOM-killed (~34 GiB RSS) during ELSER wikipedia-1g reindex in nightly QA (since build 975) #3226

Description

Summary

Evidence

Not related

Next steps / investigation

Artifacts

Activity

edsavage commented on Sep 30, 2026

Bisection update — RSS reporting (#3160) ruled out

edsavage commented on Sep 30, 2026

libtorch moved at the pass→fail boundary — now the leading suspect

edsavage commented on Sep 30, 2026

Green-arm libtorch A/B triggered

edsavage commented on Sep 30, 2026

libtorch source scan (ef7af406..4de991c7, 116 commits / 512 files): no RSS-relevant source change

edsavage commented on Sep 30, 2026

Complementary ml-cpp green-arm triggered — 2×2 factorial now in flight

edsavage commented on Sep 30, 2026

Upstream search + allocator grounding: mimalloc regression ruled out; glibc-arena is the cheap lever

edsavage commented on Sep 30, 2026

Is Elasticsearch a factor? Third variable confirmed, but no smoking gun in the delta

edsavage commented on Sep 30, 2026

#3187 reaping audit: does NOT touch the legacy CDetachedProcessSpawner path

edsavage commented on Sep 30, 2026

⚠️ Experiments #836/#837 are VOID — never reached the OOM workload

edsavage commented on Sep 30, 2026

QAF supports pinning ES to a commit (ES_COMMIT) — valid re-run launched

edsavage commented on Sep 30, 2026

Latest scheduled nightly (#838, Sep 30) did NOT OOM — QA passed

edsavage commented on Oct 1, 2026

Investigation outcome: transient PyTorch viable/strict regression, not ml-cpp

Evidence map

Decisive observation

Corrections to the original report

Recommendation

Side benefit

edsavage commented on Oct 1, 2026

Closing: root cause is a transient upstream PyTorch regression, no ml-cpp action required

edsavage commented on Oct 1, 2026

Reopening — the OOM recurred; the "transient/fixed" conclusion was wrong

Next step

edsavage commented on Oct 2, 2026

Nightly OOM history (ml-cpp × libtorch)

Key observations

Caveats

Isolation testing in progress

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Investigation outcome: transient PyTorch `viable/strict` regression, not ml-cpp