fix(podman): create managed workspace volumes owned by the workload identity by matthewgrossman · Pull Request #3981 · NVIDIA/OpenShell

matthewgrossman · 2026-09-30T18:52:57Z

Summary

Fix Podman sandboxes that cannot start when their image or policy selects a non-root user. Podman now creates the managed /sandbox volume for that user, so the workload can start and write there without first running as root.

This is the second of four stacked PRs split out of #3801: #3979 (remove unreachable sandbox code) ← this PR ← #3982 (Podman OCI WORKDIR) ← #3931 (shared OCI image tests). Review after #3979.

What breaks today, and why this fix

For example, build an image with USER 2345:2346 and no /sandbox, or use a USER-less image with policy run_as_user: "2345" and run_as_group: "2346". On a rootful Podman gateway, OpenShell gives the workload a managed /sandbox volume, but Podman creates its root as root:root with mode 0700. User 2345 cannot enter it, so sandbox provisioning fails before the agent command can run (locally, ContainerExited ... code 1; the entrypoint can report Permission denied). The user cannot fix this from inside the sandbox because the process has no authority to change the directory's ownership.

Docker already prepares its managed /sandbox directory owned by the workload UID/GID before starting the capability-free process: it stages a directory in the container archive with those ownership and permission settings. Podman uses a named volume instead of that Docker archive directory. We could start Podman's workload as root, adjust /sandbox ownership/permissions, then drop to the requested user—as the old root-then-drop path did for some launches—but that adds a privileged setup step to every affected start. Instead, this PR passes the final UID/GID directly to Podman when creating the volume. Podman makes the volume root owned by that user, and the workload starts as that user from the outset with no added capabilities. The channel volume remains unchanged.

Why the startup code no longer needs `rootless`

This does not remove Podman's rootful/rootless distinction. Rootless describes how the Podman service runs on the host; it does not mean the workload must run as container root or as a particular non-root user. Host privileges, UID/GID mappings, networking, and storage still differ between the two modes, and both still need testing.

Previously, the startup code used rootless to decide whether to start as container root, fix /sandbox ownership, and then drop to the workload user. Creating the volume with the correct ownership beforehand removes the need for that startup workaround: both modes can start directly as the selected workload user. We remove only the now-unused stored driver field and builder argument in this context; Podman's rootless status is still read and logged, and existing checks and user-namespace configuration remain.

Related Issue

Replaces #3932, which GitHub automatically marked merged during stack reordering.

Fixes #3937. This bug was found while working on #3801 / #2526: the shared oci-image suite's default_workdir_uses_managed_workspace scenario exposed it on rootful Podman. Docker already gives the image user a writable /sandbox.

Changes

client.rs: create_owned_volume(..., owner: Option<(u32, u32)>) sends Options: {"o": "uid=…,gid=…"} and verifies the created or existing volume against a shared volume_options_match_owner predicate. Podman records the parsed UID/GID next to o. Add ContainerConfig.user for admission.
driver.rs: create the workspace volume owned by (identity.uid, identity.gid) and the channel volume with no options. Resource admission accepts a workspace volume whose options match the container's numeric Config.User, or have no options (created by older gateways). The channel volume still requires empty options. Remove the now-unused rootless driver field.
container.rs: remove the launch-capability-free root-then-drop branch and IsolationSpecInput.rootless. The workload always runs --bootstrap as the final identity with cap_drop: ALL and no added capabilities. launch-capability-free stays in openshell-sandbox because the VM driver still uses it.
e2e/rust/tests/podman_oci_identity.rs: the workload container user is now {OCI_UID}:{OCI_GID} instead of 0:0.
Podman README: note the volume ownership.

Testing

All checks ran against a local rootful Podman machine (macOS, libkrun).

mise run pre-commit passes
Unit tests: cargo test -p openshell-driver-podman --lib (228 passed), with new create_owned_volume_verifies_requested_owner and admission_accepts_workspace_volume_owned_by_workload_identity, and updated isolation-spec tests. cargo clippy -p openshell-driver-podman --all-targets -- -D warnings is clean.
E2E: podman_oci_identity passes.
Manual repro through e2e/with-podman-gateway.sh. Each case runs id; stat /sandbox; touch /sandbox/probe:
- USER-less ubuntu:24.04 with policy run_as_user: "2345", run_as_group: "2346": on main, provisioning fails with ContainerExited ... code 1. With this PR it runs as 2345:2346, /sandbox is 2345:2346 700, and the write succeeds.
- Image with USER 2345:2346 and no /sandbox: same results on main and with this PR.
Resource admission: with resource_admission.enabled = true temporarily set in e2e/support/podman-gateway-config.sh (not committed), oci-image's default_workdir_uses_managed_workspace passes. With the owner-match branch of the admission check disabled, it fails with sandbox lacks attachment provenance.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable) — Podman driver README

copy-pr-bot · 2026-09-30T18:53:01Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-10-01T05:27:32Z

Label test:e2e applied, but pull-request/3981 does not exist yet. A maintainer needs to comment /ok to test 5920b8ddc8ea4ce55728eed0b4b63a8af0cf961b to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

matthewgrossman · 2026-10-01T17:10:38Z

/ok to test 5920b8d

…dentity Podman now creates the managed /sandbox volume with uid/gid options for the resolved workload identity, so the workload starts directly as that identity. This fixes rootful sandboxes whose image USER or policy run_as_user could not write to a root-owned /sandbox, and removes the root-then-drop workspace chown start path. Resource admission accepts the managed workspace volume when its options match the workload container's final identity, or are empty for volumes created by older gateways. The channel volume still requires empty options. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-10-01T19:15:53Z

/ok to test 53e7b71

elezar

LGTM

Need to re-review.

elezar

No blocking correctness issues found. Three nonblocking suggestions below to clarify the helper's contract and strengthen regression coverage.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar

Approved. The ownership fix looks sound, and the independent review found no blocking correctness issues.

@matthewgrossman, please feel free to push back on any of the test or readability changes we added. They are nonblocking suggestions, and I am happy to adjust or drop them if you prefer a different approach.

matthewgrossman added this pull request to stack #3983 September 30, 2026 18:53

matthewgrossman force-pushed the podman-workdir/b-volume-ownership branch from e10d66f to 71f24e5 Compare September 30, 2026 19:54

Base automatically changed from podman-workdir/0-remove-dead-workspace-code to main September 30, 2026 23:15

matthewgrossman force-pushed the podman-workdir/b-volume-ownership branch from 71f24e5 to 5920b8d Compare September 30, 2026 23:15

matthewgrossman added the test:e2e Requires end-to-end coverage label Oct 1, 2026

elezar mentioned this pull request Oct 1, 2026

bug: sandbox from a USER root image fails with an unexplained "UID or GID zero" error #4030

Open

6 tasks

matthewgrossman marked this pull request as ready for review October 1, 2026 17:10

matthewgrossman requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 1, 2026 17:10

matthewgrossman force-pushed the podman-workdir/b-volume-ownership branch from 5920b8d to 53e7b71 Compare October 1, 2026 19:15

elezar reviewed Oct 2, 2026

View reviewed changes

Comment thread crates/openshell-driver-podman/src/driver.rs

elezar previously approved these changes Oct 2, 2026

View reviewed changes

elezar reviewed Oct 2, 2026

View reviewed changes

Comment thread e2e/rust/tests/podman_oci_identity.rs

Comment thread crates/openshell-driver-podman/src/client.rs Outdated

Comment thread crates/openshell-driver-podman/src/client.rs Outdated

elezar mentioned this pull request Oct 2, 2026

bug(podman): verify managed volume ownership before sandbox cleanup #4111

Open

6 tasks

elezar added 3 commits October 2, 2026 13:38

test(podman): cover managed volume reuse and workspace access

20aae0d

Signed-off-by: Evan Lezar <elezar@nvidia.com>

refactor(podman): clarify managed volume creation and validation

59ba0b8

Signed-off-by: Evan Lezar <elezar@nvidia.com>

test(podman): verify workspace access across user namespaces

bd04726

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar approved these changes Oct 2, 2026

View reviewed changes

Navigation Menu

fix(podman): create managed workspace volumes owned by the workload identity - #3981

fix(podman): create managed workspace volumes owned by the workload identity#3981
matthewgrossman wants to merge 4 commits into
mainfrom
podman-workdir/b-volume-ownership

matthewgrossman commented Sep 30, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

matthewgrossman commented Oct 1, 2026

Uh oh!

matthewgrossman commented Oct 1, 2026

Uh oh!

Uh oh!

elezar left a comment

Uh oh!

elezar left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elezar left a comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

matthewgrossman commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What breaks today, and why this fix

Why the startup code no longer needs rootless

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

matthewgrossman commented Oct 1, 2026

Uh oh!

matthewgrossman commented Oct 1, 2026

Uh oh!

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthewgrossman commented Sep 30, 2026 •

edited

Loading

Why the startup code no longer needs `rootless`