Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 25 additions & 11 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,12 @@ Podman provisions a paired workload and supervisor container using its native
libpod API. The workload uses `network=none`; the external supervisor alone joins
the configured network. A per-sandbox named volume carries their mutually
authenticated gRPC Unix socket, with supervisor credentials kept in its separate
filesystem. Both containers run as the resolved non-root identity with all
capabilities dropped. They share only a user namespace for volume ownership,
not PID, mount, or network namespaces. Podman owns paired lifecycle and health;
the common protocol owns process, identity, TCP, DNS, and forwarding semantics.
filesystem. The supervisor and final sandbox runtime run as the resolved non-root
identity with all capabilities dropped. Only the managed `/sandbox` fallback uses
a trusted root bootstrap to prepare its driver-owned workspace before dropping
irreversibly to that identity. The containers do not share PID, mount, or network
namespaces. Podman owns paired lifecycle and health; the common protocol owns
process, identity, TCP, DNS, and forwarding semantics.

## Driver Contract

Expand Down Expand Up @@ -305,7 +307,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
| Runtime | Best fit | Sandbox boundary | Notes |
|---|---|---|---|
| Docker | Local development with Docker available. | Capability-free workload container. | Uses `network_mode=none`; a separate capability-free supervisor container mediates egress and access over a private daemon-local Unix socket volume. |
| Podman | Existing rootless driver. | Container. | Not converted by this isolation stack. |
| Podman | Local development with Podman available. | Capability-free workload container. | Uses `network=none`; a separate capability-free supervisor container mediates egress and access over a private Unix socket volume. |
| Kubernetes | Cluster deployment through Helm. | Capability-free sandbox Pod. | Always creates a namespace-wide empty-egress workload NetworkPolicy and a separate capability-free supervisor Pod over mutually authenticated TLS. It requires an enforcing CNI and trusted sandbox namespace; the Kubernetes API does not attest policy enforcement. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
Expand Down Expand Up @@ -390,7 +392,7 @@ Drivers deliver the two binaries to separate trust domains:
| Runtime | Delivery model |
|---|---|
| Docker | A digest-pinned daemon-local volume supplies `openshell-sandbox`; the companion image runs `openshell-supervisor`. |
| Podman | Existing driver behavior; not converted by this stack. |
| Podman | The driver pins `sandbox_runtime_image` and `supervisor_image` to image IDs. The former supplies `openshell-sandbox`; the latter is passed as the companion container's image and runs `openshell-supervisor`. |
| Kubernetes | A non-root init container stages `openshell-sandbox` into a memory volume; a directly managed Pod runs `openshell-supervisor`. |
| VM | `openshell-sandbox` is embedded in the guest rootfs; a separately digest-checked native `openshell-supervisor` runs on the host. |
| Extension | Defined by the out-of-tree driver. |
Expand All @@ -408,19 +410,31 @@ and supplementary-group set before creating the immutable workload:

- Docker pins the image ID, resolves policy selectors against the image's
`/etc/passwd` and `/etc/group`, and validates its OCI working directory.
- Podman pins the image ID, resolves policy selectors against the image's
`/etc/passwd` and `/etc/group`, and validates its OCI working directory.
- Kubernetes uses platform-resolved numeric values, including OpenShift
namespace ranges.
- VM uses the configured numeric guest identity.

UID/GID zero and `u32::MAX` are invalid. The sandbox and every child start with
the resolved identity and zero capability masks; neither process performs an
in-workload UID transition. Identity-changing policy updates require sandbox
recreation, while other policy updates remain live.
UID/GID zero and `u32::MAX` are invalid. Agent commands run as the resolved
non-root user. For Podman's managed `/sandbox` workspace, trusted setup briefly
starts as root to prepare the workspace, then switches to that user before
reading bootstrap material or accepting commands. Identity-changing policy
updates require sandbox recreation, while other policy updates remain live.

Docker uses an absolute OCI working directory as the workspace. Empty, root,
and explicit `/sandbox` values select `/sandbox`; other paths must already
exist without symlink or reserved-mount collisions and must be usable by the
resolved identity. Kubernetes and VM use `/sandbox`.
resolved identity.

Podman reads the image user and working directory from one pinned image. Empty,
`/`, and explicit `/sandbox` values use the managed `/sandbox` workspace. A
custom path must be absolute, normalized, and outside system and OpenShell
reserved paths. Image and driver mounts cannot cover the workspace. Podman
mounts the persistent workspace volume there and copies any existing image
files into it on first use. OpenShell preserves their ownership and permissions;
the final non-root user must be able to write the resulting workspace.
Kubernetes and VM use `/sandbox`.

### Executable Identity Binding

Expand Down
33 changes: 33 additions & 0 deletions crates/openshell-core/src/driver_mounts.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,18 @@ pub fn validate_container_mount_target(target: &str) -> Result<(), String> {
Ok(())
}

/// Validate a mount target against shared and driver-specific control paths.
pub fn validate_container_mount_target_with_control_paths(
target: &str,
control_paths: &[&str],
) -> Result<(), String> {
validate_container_mount_target(target)?;
for control_path in control_paths {
validate_mount_control_path(target, control_path)?;
}
Ok(())
}

/// Resolve an OCI image working directory to the internal workspace root used
/// by local container drivers.
///
Expand Down Expand Up @@ -360,6 +372,27 @@ mod tests {
validate_mount_control_path("/custom-other", "/custom/ssh.sock").unwrap();
}

#[test]
fn container_target_checks_shared_and_driver_control_paths() {
let control_paths = &["/.openshell/channel"];
assert!(
validate_container_mount_target_with_control_paths(
"/etc/openshell/tls/client",
control_paths,
)
.is_err()
);
assert!(
validate_container_mount_target_with_control_paths(
"/.openshell/channel/sandbox",
control_paths,
)
.is_err()
);
validate_container_mount_target_with_control_paths("/workspace/cache", control_paths)
.unwrap();
}

#[test]
fn workspace_rejects_malformed_runtime_control_paths() {
for control_path in [
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-core/src/resource_admission.rs
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ impl std::str::FromStr for DriverAdmissionConfig {
/// Reserved driver-owned runtime metadata; caller labels must never override it.
pub const CONFIG_USED_LABEL: &str = "openshell.ai/caller-driver-config-used";
pub const IDENTITIES_LABEL: &str = "openshell.ai/resource-admission-identities";
pub const PRIVATE_IMAGE_VOLUME_TARGETS_LABEL: &str = "openshell.ai/private-image-volume-targets";

pub fn check_config_provenance(allowed: bool, recorded: Option<&str>) -> Result<(), tonic::Status> {
match recorded {
Expand Down
19 changes: 12 additions & 7 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1215,7 +1215,7 @@ impl DockerComputeDriver {
.ok_or_else(|| Status::failed_precondition("sandbox lacks resource identity record"))?;
let mut actual = std::collections::BTreeMap::new();
let anonymous_targets: Vec<String> = labels
.get("openshell.ai/private-image-volume-targets")
.get(openshell_core::resource_admission::PRIVATE_IMAGE_VOLUME_TARGETS_LABEL)
.and_then(|value| serde_json::from_str(value).ok())
.unwrap_or_default();
for mount in container.mounts.as_deref().unwrap_or_default() {
Expand Down Expand Up @@ -5657,7 +5657,11 @@ fn build_container_create_body_for_image(
driver_mounts::validate_workspace_control_path(&workspace_root, BOUNDARY_MOUNT_PATH)
.map_err(Status::failed_precondition)?;
for volume in &image.volumes {
driver_mounts::validate_container_mount_target(volume).map_err(|error| {
driver_mounts::validate_container_mount_target_with_control_paths(
volume,
&[BOUNDARY_MOUNT_PATH],
)
.map_err(|error| {
Status::failed_precondition(format!(
"invalid image-declared volume '{volume}': {error}"
))
Expand All @@ -5667,8 +5671,6 @@ fn build_container_create_body_for_image(
"image-declared volume '{volume}' masks OCI WorkingDir '{workspace_root}' before workspace validation"
))
})?;
driver_mounts::validate_mount_control_path(volume, BOUNDARY_MOUNT_PATH)
.map_err(Status::failed_precondition)?;
}
for mount in &driver_config.mounts {
let target = match mount {
Expand All @@ -5679,8 +5681,11 @@ fn build_container_create_body_for_image(
};
driver_mounts::validate_workspace_mount_target(target, &workspace_root)
.map_err(Status::failed_precondition)?;
driver_mounts::validate_mount_control_path(target, BOUNDARY_MOUNT_PATH)
.map_err(Status::failed_precondition)?;
driver_mounts::validate_container_mount_target_with_control_paths(
target,
&[BOUNDARY_MOUNT_PATH],
)
.map_err(Status::failed_precondition)?;
}
let mut user_mounts = docker_driver_mounts(driver_config)?;
user_mounts.push(Mount {
Expand All @@ -5704,7 +5709,7 @@ fn build_container_create_body_for_image(
});
let mut labels = template.labels.clone();
labels.insert(
"openshell.ai/private-image-volume-targets".into(),
openshell_core::resource_admission::PRIVATE_IMAGE_VOLUME_TARGETS_LABEL.into(),
serde_json::to_string(&image.volumes)
.map_err(|error| Status::internal(error.to_string()))?,
);
Expand Down
25 changes: 23 additions & 2 deletions crates/openshell-driver-podman/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,27 @@ binary extraction path. `supervisor_image` supplies the dynamically linked
glibc `/openshell-supervisor` binary outside the workload. Image and request
environment belong to agent children, never the supervisor process.

## OCI working directory

OpenShell reads `WORKDIR` from the workload image. If it is unset, `/`, or
`/sandbox`, OpenShell uses its managed `/sandbox` workspace. A custom path must
be absolute, with no `.` or `..` segments. It cannot overlap `/proc`, `/sys`,
`/dev`, OpenShell-reserved paths, or the workload's private control and CA
mounts. For a custom path, image volumes and driver mounts cannot cover the
workspace or one of its parents; mounts nested below it remain valid. Podman
creates a private volume for each image-declared path nested below it.

For a custom path, Podman mounts a persistent workspace volume there. When the
volume is first created, Podman copies any files already in that image directory
into it. OpenShell keeps their ownership and permissions, starts as the final
non-root user, and rejects the image if that user cannot reach and write the
directory. Agent commands use the path as their working directory and `HOME`;
`filesystem.include_workdir` grants access to it when enabled.

For `/sandbox`, OpenShell prepares the managed workspace before switching to
the non-root user. The separate supervisor receives the path but does not mount
the workspace. Test custom images with the Podman configuration you will use.

## Lifecycle and readiness

Create builds both stopped containers and stages the private archives before
Expand All @@ -116,8 +137,8 @@ User `bind`, `volume`, `tmpfs`, and `image` mounts and CDI GPU selection remain
native Podman features and apply only to the workload. Bind mounts require the
operator's `enable_bind_mounts` opt-in and disabled label admission. Supplemental
image mounts also require disabled admission. Driver JSON requires
`allow_driver_config = true`. Reserved control paths and the workspace
root cannot be replaced. User-owned volumes are never created or deleted.
`allow_driver_config = true`. The workload's private mounts cannot be
replaced. User-owned volumes are never created or deleted.

See [gateway configuration](../../docs/how-it-works/gateways/configuration.mdx) for
operator settings and [NETWORKING.md](NETWORKING.md) for supervisor networking.
Expand Down
22 changes: 20 additions & 2 deletions crates/openshell-driver-podman/src/client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,10 @@ pub struct ImageConfig {
pub user: String,
#[serde(default)]
pub env: Vec<String>,
#[serde(default)]
pub working_dir: String,
#[serde(default)]
pub volumes: Option<HashMap<String, Value>>,
}

/// A container summary returned by the list API.
Expand Down Expand Up @@ -1158,12 +1162,12 @@ mod tests {
}

#[tokio::test]
async fn inspect_image_reads_immutable_id_and_oci_user() {
async fn inspect_image_reads_immutable_id_and_oci_config() {
let (socket_path, request_log, handle) = spawn_podman_stub(
"inspect-image",
vec![StubResponse::new(
StatusCode::OK,
r#"{"Id":"sha256:immutable","Config":{"User":"app:staff"}}"#,
r#"{"Id":"sha256:immutable","Config":{"User":"app:staff","Env":["A=one"],"WorkingDir":"/workspace/project","Volumes":{"/workspace/project/cache":{}}}}"#,
)],
);
let client = PodmanClient::new(socket_path.clone());
Expand All @@ -1178,6 +1182,20 @@ mod tests {
image.config.as_ref().map(|config| config.user.as_str()),
Some("app:staff")
);
assert_eq!(
image
.config
.as_ref()
.map(|config| config.working_dir.as_str()),
Some("/workspace/project")
);
assert!(
image
.config
.as_ref()
.and_then(|config| config.volumes.as_ref())
.is_some_and(|volumes| volumes.contains_key("/workspace/project/cache"))
);
handle.await.expect("stub task should finish");
assert_eq!(
request_log
Expand Down
Loading
Loading