[ML] Add Sandbox2 security integration for PyTorch inference by valeriy42 · Pull Request #2873 · elastic/ml-cpp

valeriy42 · 2025-10-28T13:55:38Z

PyTorch inference runs untrusted TorchScript models supplied by Elasticsearch users, so the native process must be treated as hostile. Today pytorch_inference relies largely on in-process seccomp and graph validation, but that still leaves a large syscall and filesystem surface exposed inside the same address space as libtorch. This PR hardens the Linux production path by spawning pytorch_inference inside Google Sandbox2 from the ML controller, so isolation, syscall policy, and filesystem access are enforced before the model binary starts executing.

Architecture

MlSandbox (lib/sandbox/): owns CSandboxedProcessSpawner, CPytorchInferenceSandboxPolicy, and CSandbox2Diagnostics. SANDBOX2_AVAILABLE is scoped to this library, not MlCore.
CProcessSpawnerRouter (bin/controller/): controller-owned routing layer. Nominates ./pytorch_inference explicitly, strips --disableSandbox, and chooses sandboxed vs legacy spawn based on the ES kill switch (xpack.ml.trained_models.sandbox_enabled).
CDetachedProcessSpawner (lib/core/): generic POSIX posix_spawn spawner again — no pytorch_inference-specific dispatch or Sandbox2 knowledge.
pytorch_inference: runtime-only seccomp gate on Linux when ML_SANDBOXED=1; skips in-process BPF when running under Sandbox2.

Build support vendors Abseil and the Sandboxed API on Linux, wires them into CMake, and adds the required license files.

Kill switch

When sandboxing cannot be enforced (e.g. user namespaces unavailable) or the operator disables it, the controller falls back to the legacy posix_spawn + in-process seccomp path via --disableSandbox.

Testing

lib/sandbox/unittest: spawn, allowlist drift guard, fail-closed behaviour, and policy-violation differential tests.
Coverage modes: the sandbox suite probes what the runner's kernel permits and covers enforced (real sandboxed spawn/terminate plus the filesystem-policy differential) or fail_closed (spawn refusal plus kill-switch hint) accordingly. The allowlist drift guard is mode independent and always runs. ML_SANDBOX2_REQUIRE=enforced pins the aarch64 host runner so enforced coverage cannot silently degrade; the container run is left unpinned and supplies the fail-closed half.
.buildkite/scripts/steps/diagnose_userns.sh: reports which user-namespace stage a runner denies, on the host and across container configurations, without needing a build. Kept as a regression canary for the pin above.
test/test_sandbox2_attack_defense.py: end-to-end attack-defense harness.
Extended evil-model tests; seccomp filter kept coherent with Sandbox2 policy via CPytorchInferenceSyscallAllowlist.h.

Review sequence (post-restructure)

lib/sandbox/CSandboxedProcessSpawner_Linux.cc + CPytorchInferenceSandboxPolicy_Linux.cc — spawn and policy.
bin/controller/CProcessSpawnerRouter.cc — kill-switch routing.
lib/core/CDetachedProcessSpawner.cc — generic legacy spawn baseline.
lib/seccomp/CSystemCallFilter_Linux.cc + include/seccomp/CPytorchInferenceSyscallAllowlist.h — legacy vs Sandbox2 syscall coherence.
bin/pytorch_inference/Main.cc — runtime seccomp gate.
3rd_party/CMakeLists.txt, lib/sandbox/CMakeLists.txt, lib/core/CMakeLists.txt — build wiring.
Tests: lib/sandbox/unittest/CSandboxedProcessSpawnerTest_Linux.cc — the user-namespace probe and the enforced/fail-closed dispatch it drives; test/test_sandbox2_attack_defense.py.
CI: .buildkite/scripts/steps/run_tests.sh (where each mode is covered), diagnose_userns.sh + userns_probe.sh.
Docs: docs/changelog/2873.yaml, docs/sandbox2_production_failure_modes.md.

ES companion PRs

prodsecmachine · 2025-10-28T13:55:53Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

- Added new tests for Sandbox2 functionality, including privilege validation, filesystem isolation, and syscall filtering. - Introduced a TestCleanup class for managing temporary files during tests. - Updated the CDetachedProcessSpawner_Linux.cc to support new command line arguments for log handling and model path. - Renamed modelDir to modelPath for clarity and adjusted policy building to accommodate file access. - Improved overall test coverage for Sandbox2 features and ensured graceful degradation when Sandbox2 is not available.

- Improved error handling for cases when Sandbox2 is disabled or unavailable for pytorch_inference processes. - Enhanced logging to provide clearer feedback on spawning failures with Sandbox2. - Updated comments for clarity regarding the fallback to standard posix_spawn for non-pytorch_inference processes.

- Added a new function to apply standard ML syscall restrictions using Sandbox2's PolicyBuilder, ensuring consistent security across ML processes. - Updated the CDetachedProcessSpawner_Linux to utilize the new syscall policy for pytorch_inference, eliminating the need for seccomp filtering in this context. - Enhanced comments and documentation to outline future migration plans for other ML processes to Sandbox2. - Noted the gradual transition from seccomp filters to Sandbox2 policies in the CSystemCallFilter_Linux implementation.

- Removed SetUserAndGroup from PolicyBuilder due to updates in the sandboxed-api. - Updated AddTmpfs to include a size parameter for better resource management. - Refactored Sandbox2 instantiation to use unique_ptr for the executor, improving memory management. - Enhanced comments to clarify changes and provide context for future updates.

- Removed outdated syscall number definitions and replaced them with fallback definitions for newer syscalls, ensuring compatibility with RHEL8 headers. - Updated the handling of input pipes in the sandbox policy to allow read and write access, improving functionality. - Enhanced comments for better clarity on syscall handling and future maintenance.

…l-cpp into enhancement/sandbox2

…for Linux - Deleted Sandbox2SecurityTest.cc as it is no longer needed. - Updated CMakeLists.txt to remove references to the deleted test file. - Introduced CDetachedProcessSpawnerTest_Linux.cc, which includes tests for process spawning and integration with Sandbox2. - Enhanced CMakeLists.txt to link against Sandbox2 libraries for the new tests.

After the policy file merge, comments still pointed at the deleted _Linux.cc path. Update allowlist/sync pointers to CPytorchInferenceSandboxPolicy.cc and attribute ML_SANDBOXED=1 to CSandboxedProcessSpawner_Linux.cc.

valeriy42 · 2026-09-08T19:05:50Z

Frozen at db5b14ac8 — this PR is being replaced rather than rebased.

Last green integration evidence: Buildkite #3081.

This branch will not be rebased, force-pushed, or merged, and its history is preserved. The Sandbox2 integration is being reconstructed from current main as a sequential, independently reviewable PR series:

PR	Repo	Scope
A	ml-cpp	Dormant Sandbox2/Abseil dependency, licences, checked-in version-pinned patches, dormant `MlSandbox` target. No production routing.
B	ml-cpp	One machine-readable syscall declaration generating both the applied BPF allowlist and the Sandbox2 grants; typed seccomp install result and readiness marker.
C	ml-cpp	Typed internal launch specification, single validated `0700` per-child IPC root, minimal mounts, private tmpfs, allowlisted mechanism probe.
D	ml-cpp	Explicit child lifecycle state machine, kill-and-reap guard, single `AwaitResult` owner, injectable seams, classified `pidfd_open` outcomes.
E	ml-cpp	Typed internal routing on the retained `start` protocol, pinned enforced/fail-closed CI on both architectures, repaired attack-defense harness.
F	elasticsearch	Default-`false` operator setting, unique child IPC paths, unchanged wire serialization, `PyTorchSandboxIT`.

Why replace rather than repair: the recurring defects in this branch share one cause — there was no explicit capability graph and no resource lifecycle state machine, while tests frequently asserted an outcome without proving the mechanism that produced it. Slicing the work lets each boundary be reviewed against a stated contract instead of inferred from a large diff. Security logic is rebuilt from that contract rather than bulk-cherry-picked; only audited vendoring and build material is transplanted mechanically.

The established tab-separated start command between Elasticsearch and the controller is unchanged — no new verb, wire format, or response JSON. Sandbox2 ships default-off behind the operator setting xpack.ml.trained_models.sandbox_enabled and is activated through a staged rollout.

This PR stays open and will be closed as superseded once every replacement PR is linked here.

M2: replace the implicit "seccomp precedes initIo" attestation with an explicit structured degradedModeAttestationMarker(), logged over the existing log pipe and asserted by a dedicated test, instead of a controller/ES observer inferring readiness from the absence of a fatal log line. M4: carry forward, into the shared declaration, syscall fixes frozen PR elastic#2873 discovered the hard way - clone3-by-number (57f00ed, already present via ML_NR_clone3) and __NR_prlimit64 (03b1ee4, newly added) - each with a named regression test (testCarryForwardSyscallsPresent) plus the x86_64 legacy-fs set (ec7d3ed, already present, now also asserted). d9a856d (Sandbox2 AllowFutexOp arg-filtering) and 730933d/f8b0a534 (Buildkite run_tests.sh/build.sh packaging) are out of this file's scope; see the bd note on elastic-workspace-3b59.3 for the explicit classification and deferral to PR C/D/E rather than silent coverage here. Verified on valeriy-devbox-26-04: ml_test_seccomp now 7/7 passing (2 new cases); real pytorch_inference run confirms both "ml.seccomp.installed" and the new structured marker {"ml_sandbox2_route":"legacy","event":"seccomp_installed"} are emitted.

PR A of the Sandbox2 clean rebuild plan (see docs/projects/mlcpp-sandbox2-pr2873 in elastic/elastic-workspace, and the frozen elastic#2873 for prior art). Reconstructs the dependency/build layer from current main, replacing elastic#2873's inline file(WRITE)/string(REGEX REPLACE) source rewrites with checked-in, version-pinned patches that fail configure loudly on upstream drift. - 3rd_party/CMakeLists.txt: FetchContent sandboxed-api v20241008 on Linux, apply 4 patches via `git apply` with FATAL_ERROR on --check failure (idempotent across reconfigure via a reverse-check pre-test) - 3rd_party/patches/sandboxed-api/: the 4 patches (disable vendored gtest, stop -fno-exceptions propagating into ml-cpp targets, make Python3 optional, link zlib + static libstdc++/libgcc into the forkserver binary), each verified against a real shallow clone of the pinned tag - 3rd_party/licenses/{abseil,sandbox2}-*: license/attribution files - lib/sandbox/: dormant MlSandbox target (CMlSandboxAvailability query only - no policy/spawner/diagnostics, no controller wiring; those land in later PRs) plus a Linux-only forkserver runtime smoke test against a statically-linked, dependency-free payload binary Verified on Linux x86_64 (GCP devbox): configure reaches "Sandbox2 enabled: using sandbox2::sandbox2"; ml_test_sandbox passes under both a normal and a -DCMAKE_UNITY_BUILD=ON configure, including the forkserver smoke test actually forking/execing/reaping the payload. Not yet verified: aarch64, macOS, Windows non-Sandbox2 builds, license/Snyk scan.

…aded seccomp Replace the hand-maintained BPF jump-offset table in CSystemCallFilter_Linux.cc with a program builder that derives every jump from the allowlist vector's own size/index. The applied program is generated from CPytorchInferenceSyscallAllowlist.h, a single machine-readable declaration, instead of a parallel hardcoded list (design.md V3/MG6). CSystemCallFilter::installSystemCallFilter() returns a typed ESystemCallFilterInstallOutcome across all three platform implementations instead of void, and logs the ml.seccomp.installed readiness marker on success. pytorch_inference/Main.cc gains a decideDegradedModeAction() decision that would terminate before CIoManager::initIo() on any degraded-mode seccomp failure; that termination stays behind an internal switch defaulting to false until ml-cpp PR E's typed controller routing can guarantee a degraded-mode launch was deliberate (design.md MG8, "activation is deliberately split across two slices"). The four non-PyTorch callers now make their unchanged log-and-continue policy explicit instead of silently discarding the result. Adds CSeccompFilterBuilderTest.cc: decodes the actually-built BPF program to prove it matches the declaration and that jump offsets are derived, not hand-maintained; fault-injected coverage of decideDegradedModeAction() for every install-failure class (V13); a named regression test per design.md M4 carry-forward syscall (clone3 by number, prlimit64, x86_64 legacy filesystem syscalls) so a blank-slate reconstruction cannot silently drop a hard-won compatibility fix from frozen PR elastic#2873; and an explicit structured degradedModeAttestationMarker() (design.md M2) so a controller/ES observer can assert seccomp installation directly instead of inferring it from the absence of a fatal log line. The Sandbox2-grants half of V3 (CPytorchInferenceSandboxPolicy.cc) does not exist yet in this clean rebuild lineage (ml-cpp PR C); this change establishes the single declaration for that PR to consume. d9a856d (Sandbox2 AllowFutexOp arg-filtering) and 730933d/f8b0a534 (Buildkite run_tests.sh/build.sh packaging) are likewise out of this file's scope - see the bd note on elastic-workspace-3b59.3 for the explicit deferral to PR C/D/E. Verified on a real x86_64 devbox: ml_test_seccomp (7 test cases) and the real pytorch_inference/autodetect/categorize/normalize/ data_frame_analyzer binaries all build, link, and run correctly against the new typed API; the generated BPF program installs via a real prctl(PR_SET_SECCOMP) call.

…aded seccomp Replace the hand-maintained BPF jump-offset table in CSystemCallFilter_Linux.cc with a program builder that derives every jump from the allowlist vector's own size/index. The applied program is generated from CPytorchInferenceSyscallAllowlist.h, a single machine-readable declaration, instead of a parallel hardcoded list (design.md V3/MG6). CSystemCallFilter::installSystemCallFilter() returns a typed ESystemCallFilterInstallOutcome across all three platform implementations instead of void, and logs the ml.seccomp.installed readiness marker on success. pytorch_inference/Main.cc gains a decideDegradedModeAction() decision that would terminate before CIoManager::initIo() on any degraded-mode seccomp failure; that termination stays behind an internal switch defaulting to false until ml-cpp PR E's typed controller routing can guarantee a degraded-mode launch was deliberate (design.md MG8, "activation is deliberately split across two slices"). The four non-PyTorch callers now make their unchanged log-and-continue policy explicit instead of silently discarding the result. Adds CSeccompFilterBuilderTest.cc: decodes the actually-built BPF program to prove it matches the declaration and that jump offsets are derived, not hand-maintained; fault-injected coverage of decideDegradedModeAction() for every install-failure class (V13); a named regression test per design.md M4 carry-forward syscall (clone3 by number, prlimit64, x86_64 legacy filesystem syscalls) so a blank-slate reconstruction cannot silently drop a hard-won compatibility fix from frozen PR #2873; and an explicit structured degradedModeAttestationMarker() (design.md M2) so a controller/ES observer can assert seccomp installation directly instead of inferring it from the absence of a fatal log line. The Sandbox2-grants half of V3 (CPytorchInferenceSandboxPolicy.cc) does not exist yet in this clean rebuild lineage (ml-cpp PR C); this change establishes the single declaration for that PR to consume. d9a856d (Sandbox2 AllowFutexOp arg-filtering) and 730933d/f8b0a534 (Buildkite run_tests.sh/build.sh packaging) are likewise out of this file's scope - see the bd note on elastic-workspace-3b59.3 for the explicit deferral to PR C/D/E. Verified on a real x86_64 devbox: ml_test_seccomp (7 test cases) and the real pytorch_inference/autodetect/categorize/normalize/ data_frame_analyzer binaries all build, link, and run correctly against the new typed API; the generated BPF program installs via a real prctl(PR_SET_SECCOMP) call.

The clean rebuild's Sandbox2 policy builder grants syscalls to pytorch_inference by looping over legacyBpfAllowedSyscalls() alone - the same list used to build the legacy in-process BPF filter. That is not sufficient: Sandbox2's namespace/threading setup makes pytorch_inference exercise syscalls (scheduling, epoll, pipes, directory/file management for forecast temp storage) that the simpler legacy filter never needed a grant for, since it never sets up namespaces or Sandbox2's own thread/monitor machinery. #2873's enhancement/sandbox2 branch already solved this with a dedicated sandbox2ExplicitSyscalls() list, granted in addition to what Sandbox2's PolicyBuilder helpers (AllowRead/AllowWrite/AllowOpen/etc.) cover implicitly. That list was never carried into this rebuild's CPytorchInferenceSyscallAllowlist.h, so any syscall in it (confirmed via a real run: sched_getaffinity) got denied and killed the sandboxed process. Ports sandbox2ExplicitSyscalls() and sandbox2HelperCoveredSyscalls() from the original branch, wires the explicit list into buildPytorchInferencePolicy() alongside the existing legacy-list loop, and adds sandbox2AllowsAllLegacySyscalls() plus a regression test so a future change to legacyBpfAllowedSyscalls() without a corresponding Sandbox2 grant fails loudly instead of silently regressing.

PR A of the Sandbox2 clean rebuild plan (see docs/projects/mlcpp-sandbox2-pr2873 in elastic/elastic-workspace, and the frozen #2873 for prior art). Reconstructs the dependency/build layer from current main, replacing #2873's inline file(WRITE)/string(REGEX REPLACE) source rewrites with checked-in, version-pinned patches that fail configure loudly on upstream drift. - 3rd_party/CMakeLists.txt: FetchContent sandboxed-api v20241008 on Linux, apply 4 patches via `git apply` with FATAL_ERROR on --check failure (idempotent across reconfigure via a reverse-check pre-test) - 3rd_party/patches/sandboxed-api/: the 4 patches (disable vendored gtest, stop -fno-exceptions propagating into ml-cpp targets, make Python3 optional, link zlib + static libstdc++/libgcc into the forkserver binary), each verified against a real shallow clone of the pinned tag - 3rd_party/licenses/{abseil,sandbox2}-*: license/attribution files - lib/sandbox/: dormant MlSandbox target (CMlSandboxAvailability query only - no policy/spawner/diagnostics, no controller wiring; those land in later PRs) plus a Linux-only forkserver runtime smoke test against a statically-linked, dependency-free payload binary Verified on Linux x86_64 (GCP devbox): configure reaches "Sandbox2 enabled: using sandbox2::sandbox2"; ml_test_sandbox passes under both a normal and a -DCMAKE_UNITY_BUILD=ON configure, including the forkserver smoke test actually forking/execing/reaping the payload. Not yet verified: aarch64, macOS, Windows non-Sandbox2 builds, license/Snyk scan.

The clean rebuild's Sandbox2 policy builder grants syscalls to pytorch_inference by looping over legacyBpfAllowedSyscalls() alone - the same list used to build the legacy in-process BPF filter. That is not sufficient: Sandbox2's namespace/threading setup makes pytorch_inference exercise syscalls (scheduling, epoll, pipes, directory/file management for forecast temp storage) that the simpler legacy filter never needed a grant for, since it never sets up namespaces or Sandbox2's own thread/monitor machinery. #2873's enhancement/sandbox2 branch already solved this with a dedicated sandbox2ExplicitSyscalls() list, granted in addition to what Sandbox2's PolicyBuilder helpers (AllowRead/AllowWrite/AllowOpen/etc.) cover implicitly. That list was never carried into this rebuild's CPytorchInferenceSyscallAllowlist.h, so any syscall in it (confirmed via a real run: sched_getaffinity) got denied and killed the sandboxed process. Ports sandbox2ExplicitSyscalls() and sandbox2HelperCoveredSyscalls() from the original branch, wires the explicit list into buildPytorchInferencePolicy() alongside the existing legacy-list loop, and adds sandbox2AllowsAllLegacySyscalls() plus a regression test so a future change to legacyBpfAllowedSyscalls() without a corresponding Sandbox2 grant fails loudly instead of silently regressing.

## Summary Reconstructs the dormant dependency/build foundation for Sandbox2 from current `main`, as a clean first slice ahead of the sandbox policy, spawner, and controller-routing changes that land in follow-up PRs. No controller or `pytorch_inference` routing changes in this PR — Sandbox2 is not selectable from any production code path yet. Frozen PR #2873 attempted this feature in one large branch; this PR takes just its dependency/build layer and replaces its inline `file(WRITE)`/`string(REGEX REPLACE)` source rewrites with checked-in, version-pinned patches that fail the configure step loudly instead of silently no-op'ing on upstream drift. - `3rd_party/CMakeLists.txt`: FetchContent `sandboxed-api` `v20241008` on Linux, applying 4 checked-in patches via `git apply` (fails configure loudly on upstream drift, idempotent across reconfigure). - `3rd_party/patches/sandboxed-api/`: the 4 patches (disable vendored gtest, stop `-fno-exceptions` propagating into ml-cpp targets, make Python3 optional, link zlib + static libstdc++/libgcc into the forkserver binary). - `3rd_party/licenses/{abseil,sandbox2}-*`: license/attribution files. - `lib/sandbox/`: dormant `MlSandbox` target (`CMlSandboxAvailability` query only — no policy/spawner/diagnostics) plus a Linux-only forkserver runtime smoke test that forks/execs/reaps a dynamically-linked payload via `PolicyBuilder::AddLibrariesForBinary()`. Verified locally (Linux x86_64, both a plain configure and `-DCMAKE_UNITY_BUILD=ON`) before pushing, and green on this repo's own Linux/macOS/Windows CI, license/security scans, and Java integration suites. <sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>

The clean rebuild's Sandbox2 policy builder grants syscalls to pytorch_inference by looping over legacyBpfAllowedSyscalls() alone - the same list used to build the legacy in-process BPF filter. That is not sufficient: Sandbox2's namespace/threading setup makes pytorch_inference exercise syscalls (scheduling, epoll, pipes, directory/file management for forecast temp storage) that the simpler legacy filter never needed a grant for, since it never sets up namespaces or Sandbox2's own thread/monitor machinery. #2873's enhancement/sandbox2 branch already solved this with a dedicated sandbox2ExplicitSyscalls() list, granted in addition to what Sandbox2's PolicyBuilder helpers (AllowRead/AllowWrite/AllowOpen/etc.) cover implicitly. That list was never carried into this rebuild's CPytorchInferenceSyscallAllowlist.h, so any syscall in it (confirmed via a real run: sched_getaffinity) got denied and killed the sandboxed process. Ports sandbox2ExplicitSyscalls() and sandbox2HelperCoveredSyscalls() from the original branch, wires the explicit list into buildPytorchInferencePolicy() alongside the existing legacy-list loop, and adds sandbox2AllowsAllLegacySyscalls() plus a regression test so a future change to legacyBpfAllowedSyscalls() without a corresponding Sandbox2 grant fails loudly instead of silently regressing.

…aded seccomp Replace the hand-maintained BPF jump-offset table in CSystemCallFilter_Linux.cc with a program builder that derives every jump from the allowlist vector's own size/index. The applied program is generated from CPytorchInferenceSyscallAllowlist.h, a single machine-readable declaration, instead of a parallel hardcoded list (design.md V3/MG6). CSystemCallFilter::installSystemCallFilter() returns a typed ESystemCallFilterInstallOutcome across all three platform implementations instead of void, and logs the ml.seccomp.installed readiness marker on success. pytorch_inference/Main.cc gains a decideDegradedModeAction() decision that would terminate before CIoManager::initIo() on any degraded-mode seccomp failure; that termination stays behind an internal switch defaulting to false until ml-cpp PR E's typed controller routing can guarantee a degraded-mode launch was deliberate (design.md MG8, "activation is deliberately split across two slices"). The four non-PyTorch callers now make their unchanged log-and-continue policy explicit instead of silently discarding the result. Adds CSeccompFilterBuilderTest.cc: decodes the actually-built BPF program to prove it matches the declaration and that jump offsets are derived, not hand-maintained; fault-injected coverage of decideDegradedModeAction() for every install-failure class (V13); a named regression test per design.md M4 carry-forward syscall (clone3 by number, prlimit64, x86_64 legacy filesystem syscalls) so a blank-slate reconstruction cannot silently drop a hard-won compatibility fix from frozen PR elastic#2873; and an explicit structured degradedModeAttestationMarker() (design.md M2) so a controller/ES observer can assert seccomp installation directly instead of inferring it from the absence of a fatal log line. The Sandbox2-grants half of V3 (CPytorchInferenceSandboxPolicy.cc) does not exist yet in this clean rebuild lineage (ml-cpp PR C); this change establishes the single declaration for that PR to consume. d9a856d (Sandbox2 AllowFutexOp arg-filtering) and 730933d/f8b0a534 (Buildkite run_tests.sh/build.sh packaging) are likewise out of this file's scope - see the bd note on elastic-workspace-3b59.3 for the explicit deferral to PR C/D/E. Verified on a real x86_64 devbox: ml_test_seccomp (7 test cases) and the real pytorch_inference/autodetect/categorize/normalize/ data_frame_analyzer binaries all build, link, and run correctly against the new typed API; the generated BPF program installs via a real prctl(PR_SET_SECCOMP) call.

The clean rebuild's Sandbox2 policy builder grants syscalls to pytorch_inference by looping over legacyBpfAllowedSyscalls() alone - the same list used to build the legacy in-process BPF filter. That is not sufficient: Sandbox2's namespace/threading setup makes pytorch_inference exercise syscalls (scheduling, epoll, pipes, directory/file management for forecast temp storage) that the simpler legacy filter never needed a grant for, since it never sets up namespaces or Sandbox2's own thread/monitor machinery. #2873's enhancement/sandbox2 branch already solved this with a dedicated sandbox2ExplicitSyscalls() list, granted in addition to what Sandbox2's PolicyBuilder helpers (AllowRead/AllowWrite/AllowOpen/etc.) cover implicitly. That list was never carried into this rebuild's CPytorchInferenceSyscallAllowlist.h, so any syscall in it (confirmed via a real run: sched_getaffinity) got denied and killed the sandboxed process. Ports sandbox2ExplicitSyscalls() and sandbox2HelperCoveredSyscalls() from the original branch, wires the explicit list into buildPytorchInferencePolicy() alongside the existing legacy-list loop, and adds sandbox2AllowsAllLegacySyscalls() plus a regression test so a future change to legacyBpfAllowedSyscalls() without a corresponding Sandbox2 grant fails loudly instead of silently regressing.

…aded seccomp (#3182) Replace the hand-maintained BPF jump-offset table in `CSystemCallFilter_Linux.cc` with a program builder that derives every jump from the allowlist vector's own size/index. The applied program is generated from `CPytorchInferenceSyscallAllowlist.h`, a single machine-readable declaration, instead of a parallel hardcoded list. `CSystemCallFilter::installSystemCallFilter()` returns a typed `ESystemCallFilterInstallOutcome` across all three platform implementations instead of `void`, and logs an `ml.seccomp.installed` readiness marker on success. `pytorch_inference/Main.cc` gains a `decideDegradedModeAction()` decision that would terminate before `CIoManager::initIo()` on any degraded-mode seccomp failure; that termination stays behind an internal switch defaulting to false until the controller has a typed way to know a degraded-mode launch was a deliberate operator choice rather than the only option available. Flipping it on today would terminate every launch on a host lacking seccomp BPF, with no operator fallback to select instead. The four non-PyTorch callers now make their unchanged log-and-continue policy explicit instead of silently discarding the result. Also adds an explicit structured `degradedModeAttestationMarker()` so a controller/Elasticsearch observer can assert seccomp installation directly instead of inferring it from the absence of a fatal log line, and carries forward two pytorch_inference/libtorch compatibility fixes from #2873 into the shared declaration (`clone3` allowed by literal syscall number, and `prlimit64`) with a named regression test, so a from-scratch rewrite doesn't silently drop them. `CSeccompFilterBuilderTest.cc` decodes the actually-built BPF program to prove it matches the declaration and that jump offsets are derived, not hand-maintained, plus fault-injected coverage of `decideDegradedModeAction()` for every install-failure class. A future Sandbox2 filesystem/network policy doesn't exist yet in this branch's history; this change establishes the single declaration for that policy to consume once it's written.

The clean rebuild's Sandbox2 policy builder grants syscalls to pytorch_inference by looping over legacyBpfAllowedSyscalls() alone - the same list used to build the legacy in-process BPF filter. That is not sufficient: Sandbox2's namespace/threading setup makes pytorch_inference exercise syscalls (scheduling, epoll, pipes, directory/file management for forecast temp storage) that the simpler legacy filter never needed a grant for, since it never sets up namespaces or Sandbox2's own thread/monitor machinery. #2873's enhancement/sandbox2 branch already solved this with a dedicated sandbox2ExplicitSyscalls() list, granted in addition to what Sandbox2's PolicyBuilder helpers (AllowRead/AllowWrite/AllowOpen/etc.) cover implicitly. That list was never carried into this rebuild's CPytorchInferenceSyscallAllowlist.h, so any syscall in it (confirmed via a real run: sched_getaffinity) got denied and killed the sandboxed process. Ports sandbox2ExplicitSyscalls() and sandbox2HelperCoveredSyscalls() from the original branch, wires the explicit list into buildPytorchInferencePolicy() alongside the existing legacy-list loop, and adds sandbox2AllowsAllLegacySyscalls() plus a regression test so a future change to legacyBpfAllowedSyscalls() without a corresponding Sandbox2 grant fails loudly instead of silently regressing.

## Summary Stacks on #3187. Adds typed controller-side routing behind two symmetric controller tokens. Landlock fallback when Sandbox2 is unavailable is split to stacked #3215. - `CCommandProcessor` parses at most one of `--disableSandbox` (operator kill switch, forces legacy) or `--requireSandbox` (operator opt-in, forces Sandbox2 on capable hosts) for the exact configured `pytorch_inference` path. Duplicate occurrences of either token, a mismatched process path, or both tokens present together are all rejected before spawn. The selected token is stripped before the child ever sees it. - New `CProcessSpawnerRouter` dispatches the decided route to `CSandboxedProcessSpawner` or `CDetachedProcessSpawner` with no automatic fallback: a required Sandbox2 launch that fails returns a failed start response, it never retries through the legacy spawner. - A `start` command with neither token always takes the legacy route - permanent behaviour for any caller that sends no routing token (support/debug scripts, direct controller invocation, the test harness), not a rollout seam. Elasticsearch is expected to always send exactly one of the two tokens per launch, chosen from its own operator setting's live value. - Structured once-per-launch observability signal (`sandbox2_launch`: `deployment_id`, `model_id`, `route`, `sandbox2_established`, `mode`, `legacy_reason`, `sandbox2_compiled_in`) so a consumer can distinguish an operator kill switch from the no-token default, and "Sandbox2 supported but no routing token sent" from "built without Sandbox2 support." - `ML_SANDBOXED` is stripped from every legacy-route child's environment (POSIX and Windows) so an inherited or injected value can never fail-open the mandatory in-process seccomp filter. - Active Sandbox2 capability probe logged once at controller start (`CSandbox2Diagnostics`) so operators see whether `--requireSandbox` can succeed before a deployment fails closed. - Staged user-namespace capability probe plus CI wiring: aarch64 runs enforced-mode coverage; x86_64 runs fail-closed-only coverage, since x86_64 Buildkite k8s pods get `EPERM` on `mount("proc", ...)` and there is currently no userns-capable x86_64 CI runner. - Repaired `test_sandbox2_attack_defense.py`: reached markers, unsandboxed positive controls, per-case cleanup/reap assertions, the real `ml-child-ipc/<child-id>` IPC layout, an assertion that the controller's own `sandbox2_launch` signal reports the expected route before any security-boundary check runs, and explicit `--requireSandbox`/`--disableSandbox` tokens per case instead of a global env-var default. - Producer-side controller-protocol capability token (`3rd_party/controller-protocol.version`, now version 2) for a future cross-repo compatibility check. - Build packaging: per-library `install_libs()` guard and unconditional `$ORIGIN` RPATH on bundled ELF libraries so sibling `.so` dependencies resolve when the controller environment is cleared. ## Filesystem-policy fixes found during enforced-mode qualification End-to-end qualification on the qaf harness with `xpack.ml.trained_models.sandbox_enabled=true` (real `mode:enforced` route, not the legacy fallback) surfaced several launch/filesystem-policy gaps that previously only manifested once a real `pytorch_inference` ran under the enforced sandbox. Fixed here: - **Per-child IPC directory is created before validation.** The native controller now creates `$TMPDIR/ml-child-ipc/<child-id>` (mode 0700) before `validateChildIpcLaunchSpec()`'s live `realpath()` calls, on both spawn paths. Elasticsearch only ever constructs the IPC path strings; nothing created the directory they name. - **Per-child IPC root is mounted at the same path inside and outside the sandbox** (no `/run/elastic/ml-ipc` remap), because `pytorch_inference` receives its `--input=`/`--output=`/`--restore=`/`--logPipe=` argv as host paths under that root - a remap left them unresolvable in the sandbox mount namespace. - **Sandbox2-specific syscall allowlist** ported from the frozen reference (#2873): Sandbox2's namespace/threading setup exercises syscalls the legacy in-process BPF filter never needed, so `legacyBpfAllowedSyscalls()` alone is insufficient. - **`/proc` is mounted (PID-namespaced) inside the sandbox rootfs.** Sandbox2 mounts a fresh PID-namespaced procfs on the outer root, then `pivot_root`s into the chroot and detaches the old root, so the rootfs had no `/proc`. Without it `readlink(/proc/self/exe)` fails with `ENOENT`, breaking Intel oneMKL's runtime dispatcher (it reads `/proc/self/exe` to self-locate and `dlopen` its CPU-specific `libmkl_*.so.3` kernels) - every enforced `pytorch_inference` aborted with `Intel oneMKL FATAL ERROR: Cannot load <mkl-loader>`. The mount binds the already-namespaced procfs (never the host's): inside the sandbox `/proc` shows only the sandboxee's own PIDs. `/sys` is left unmounted. - **Regression test:** `ml_sandbox_probe` now checks `/proc/self/exe` is readable inside the real sandbox, asserted by `CPytorchInferenceSandboxPolicyMechanismTest_Linux` - turning the MKL crash into a build-time failure. Verified: ml-cpp sandbox unit tests 33/33; qaf boot check and `test_scenario_buildly` (8/8) pass under `mode:enforced` with zero MKL crashes and no legacy fallbacks.

valeriy42 added 2 commits October 28, 2025 12:15

sandbox2 and deps licenses

311ad21

build integration

7382f3b

valeriy42 added the >enhancement label Oct 28, 2025

valeriy42 changed the title ~~Add Sandbox2 security integration for PyTorch inference~~ [ML] Add Sandbox2 security integration for PyTorch inference Oct 28, 2025

valeriy42 added :ml v9.3.0 labels Oct 28, 2025

valeriy42 marked this pull request as draft October 28, 2025 14:00

valeriy42 added 13 commits October 28, 2025 14:43

formatting

b43e999

fix header

5cd22f9

sandbox2 library correctly linked

5c5a2a0

test 1 passes without sandbox2

9dadd42

benign model passes. policy very permissive.

404c6b5

evil model generator

878fadf

restrict policies.

d4adb62

formatting

fbc3c00

valeriy42 added the ci:run-qa-tests Run a subset of the QA tests label Nov 6, 2025

valeriy42 added 8 commits November 6, 2025 20:37

changelog

9d5d2ba

patch SAPI cmake file to make Python3 dependency optional

7506aab

Merge branch 'enhancement/sandbox2' of https://github.com/valeriy42/m…

6f49269

…l-cpp into enhancement/sandbox2

fix python checks on CI

ac26ca6

Add syscall allowance for prlimit64 in sandbox policy

03b1ee4

formatting

b93dfec

adjust for arm64

0594140

valeriy42 added 2 commits September 8, 2026 09:19

formatting

17ff410

valeriy42 marked this pull request as draft September 8, 2026 19:05

valeriy42 mentioned this pull request Sep 9, 2026

[ML] Land dormant Sandbox2/Abseil dependency foundation #3181

Merged

valeriy42 mentioned this pull request Sep 9, 2026

[ML] Generate syscall policies from one declaration; fail-closed degraded seccomp #3182

Merged

valeriy42 mentioned this pull request Sep 14, 2026

[ML] Typed routing behind the established controller protocol #3188

Merged

valeriy42 closed this Oct 16, 2026

Navigation Menu

Repository navigation

[ML] Add Sandbox2 security integration for PyTorch inference - #2873

[ML] Add Sandbox2 security integration for PyTorch inference#2873
valeriy42 wants to merge 93 commits into
elastic:mainfrom
valeriy42:enhancement/sandbox2

valeriy42 commented Oct 28, 2025 •

edited

Loading

Uh oh!

prodsecmachine commented Oct 28, 2025 •

edited

Loading

Uh oh!

valeriy42 commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

valeriy42 commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Architecture

Kill switch

Testing

Review sequence (post-restructure)

ES companion PRs

Uh oh!

prodsecmachine commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

valeriy42 commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

valeriy42 commented Oct 28, 2025 •

edited

Loading

prodsecmachine commented Oct 28, 2025 •

edited

Loading