Add the process shim and its broker - #348
Merged
Merged
Conversation
SaladDay
force-pushed
the
aos/shim-broker
branch
from
October 1, 2026 02:41
a646c3f to
9f650ff
Compare
The process broker reads local terminal modes with the same RFC 4254 table the Linux process service applies, so the table moves to sandboxprocess.
oac-process-shim hands its invocation and descriptors 0-2 to the Session's broker over /.oac/run/process.sock and exits the way the remote program did. The broker authenticates the peer uid, resolves the declared executable and environment, runs the program with the process protocol, pumps the passed descriptors without changing their flags, forwards signals and PTY resizes, survives link loss by re-attaching, and releases settled operations. processserve runs the Linux process service in its own process for the broker tests, so its reaper never competes with the tests' waits.
- Track each shim connection from accept, bound the handshake, and close the connection and its descriptors on Close. - Reopen pipes, FIFOs and character devices through /proc/self/fd as independent non-blocking descriptions, use sockets with MSG_DONTWAIT and regular files as they are, and refuse other descriptors, so no read or write blocks outside a poll the invocation can end. - Deliver output a gone reader discarded at once and close the remote output in the background, so settlement never waits behind it. - Keep resolving a Start that may have taken effect with the same ID and spec until it is definite; otherwise exit 255 and cancel the operation if it exists. - Release a settled operation right after re-attaching, and keep observing after scope observation is lost. - Take the PTY's TERM from the composed environment and free a partly allocated invocation. - Catch every signal a Go program can catch and document the remaining signal and descriptor limits.
sessionview.NewPTS creates the devpts instance in the daemon and reports its device number, so the process broker can accept terminals only from the view's own instance before anything in the view runs. Start passes the detached mount to the launcher, which attaches it at /dev/pts; a nil Spec.PTS gives the view a new instance.
- Accept as character devices only the memory devices, used as passed, and pty slaves of the view's devpts instance; refuse every other one with 126, so a reopen as root never revives a revoked description. - Send output on an AF_UNIX socket with SCM_CREDENTIALS naming the shim's pid, uid and gid, then the broker's pid once the shim is gone. - Require a run directory owned by the broker's user and writable by no one else, and bind, chmod, chown and unlink the socket relative to it; the view test mounts it read-only. - Keep resolving an uncertain Start until a refusal proves absence; Busy and a failed implicit Attach of an existing operation never end it. - Bound Start, its implicit Attach and every retry wait with one deadline. - Retry Busy acknowledgements and releases with backoff until the invocation halts. - Share a terminal's saved mode across invocations; the last to leave restores it.
The broker resolves shim names and refuses the private tree with agent.ViewPrivateRoot and agent.ViewShimName. The shim keeps SocketPath as a literal so that it does not link package agent, and a test checks it against agent.ViewPrivateRoot and agent.ViewRunName.
…ost shim's wait for a stream A Busy refusal no longer drops stdin, a re-Attach or a Cancel: every request on a live operation repeats after a backoff, and a stdin write resumes from the first byte the service did not accept. A Cancel that may have taken effect is not sent again, so the scope gets one TERM. Until a Start may have taken effect, losing the shim ends the wait for a stream and releases the shim's descriptors.
oac-process-shim gains a relay mode: one relay per Session, running as the Session user with no capabilities, receives each shim's request and descriptors and does every read, write and terminal ioctl on them with the Session's own authority. The broker keeps all process protocol work and talks to the relay over a socketpair it reads without a control buffer, so it never holds a view descriptor. Each descriptor has its own pump in the relay; the broker sends at most OutputWindow unreported bytes per stream and acknowledges output to the service only after the relay reports the write. The terminal registry moves into the relay and takes its snapshot under the registry lock. The broker no longer needs a root-only run directory, SO_PEERCRED, the character-device allowlist or explicit SCM_CREDENTIALS, so Config loses RunDir, UID and PTSDevice and gains Relay. A lost relay fails the broker: Done closes and Err wraps ErrRelayLost.
Only the process broker needed the instance's device number before the view, to accept terminals from that instance alone. The relay now opens terminals as the Session user, so the launcher creates the instance itself and Spec.PTS and NewPTS go away. This reverts commit 152056f6461df5af8b5f8e6fc30b381467fec8d4.
A declared shim named oac-process-shim would take the relay's path, so View.Validate rejects it and a processshim test checks RelayName against agent.ViewRelayName.
SaladDay
force-pushed
the
aos/shim-broker
branch
from
October 1, 2026 05:36
9f650ff to
292ff90
Compare
When the spec declares a shim, the launcher binds the Session's process socket on a read-only /.oac/run tmpfs owned by the Session user and starts the unprivileged relay with it and its end of the broker connection, after the launcher restricts itself and before the Harness. View.Relay is the broker's end. sessionview owns /.oac/run, so agenthost no longer presents a run directory from the Session directory. Teardown shuts the relay connection down, then stops the world while it reaps the view, because a process blocked on an unanswered world request cannot exit until the world ends it. It waits at most 30 seconds and reports ErrCleanup from Wait and Close past that. The launcher's drain no longer waits for the relay, and sessionview gives the Session user the pipes it creates.
The relay allocated an invocation's ID under its lock but wrote the Open later under the send lock, so a handshake that finished second could publish ID 2 before ID 1; the broker rejects a decreasing ID and fails the Session. Allocation and the Open write are now one step under the send lock.
The control goroutine makes the terminal raw on Started while the output pumps run on their own, so a pump could write first and the cooked terminal's OPOST|ONLCR turned the remote terminal's \r\n into \r\r\n. Output writes now wait for raw mode, as stdin reads do.
The broker treated a successful Start as started. A Start retried after an uncertain one can find the operation still Starting; its first stdin write then got NotRunning, forwarding stopped for good, and the program waited forever for input or end of file. The broker now tells the relay Started and begins forwarding stdin on the Started event, and a StartFailed event stays the typed start failure.
When a stdin write failed with an uncertain effect, such as a lost stream, the broker stopped reading local stdin but left the remote stdin open, so a program like cat hung once the stream was back. As the process protocol prescribes, the broker now learns the accepted offset with Inspect, on the next stream when the stream ended, and writes only the bytes after it; exact-offset writes keep any byte from arriving twice. When the offset cannot be learned or is inconsistent, the shim exits with 255 and the reason, and the program is cancelled.
Contributor
|
Found 2 test failures on Blacksmith runners: Failures
|
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light.svg)
A stdin write whose response was lost may find, through Inspect, that the leader exited before its Exited event arrived. A background process may still read stdin, so pipe stdin now closes at the offset the service accepted, as it does after the exit, and the background reader gets end of file instead of holding the output and the settlement open.
When the shim already has its Result or is gone, the path that answered or lost it owns the program's end. stdinLost cancelled the program anyway, so a shim lost while stdin recovery failed sent the program a second TERM.
stdinLost checked whether the exit was decided apart from reserving its Result, so an Exited event observed between the two had its status replaced by 255. replyBeforeExit does both under the invocation's lock.
The test paused the first handshake before its ID was allocated, so the second took ID 1 and a relay that allocated and published apart still passed. The seam now runs between allocation and publication: the second handshake must wait for the publication lock, which the test observes, and splitting the two steps publishes Open 2 before Open 1.
The raw-mode test held raw mode for 300 ms in the hope that the output reached its pump, and the fake service ran a late start 300 ms after the retried Start in the hope that its Attach saw Starting. A gating seam now marks the output pump holding output before the terminal is raw, and the test asserts nothing is written until it is raw. The fake keeps the operation Starting until an Attach found it so and the broker observes it without a stdin pump. Timers remain only as failure deadlines.
start launched the control goroutine before adding the output pumps to their WaitGroup. An End that arrived at once let finish wait on a zero count, close the descriptors and the end flag, and leave the pumps to start on descriptors that may already be reused.
The relay restores the terminal at the Exit and does not hold the shim's exit for the Terminal stream, as a native shell never waits for a background job that holds the terminal. Output that arrives after the Exit is therefore written in the restored mode.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lane L2b of the agent-outside-sandbox workstream. A Harness in a Session view runs declared programs through
oac-process-shim. The programs run in the sandbox through the Process protocol, and the shim mirrors the remote exit.Invariant: only an unprivileged per-Session relay inside the view receives or operates on descriptors the view supplies. The root broker holds only its own transport endpoints.
Shape
apps/daemon/cmd/oac-process-shimis a static multi-call binary./.oac/run/process.sockand passes its request and fds 0–2 to the relay.oac-process-shim relay): it is the only holder of view descriptors./proc/self/fd, checking the access mode and never changing shared descriptor flags.Openin one serialized step, so Opens reach the broker in ID order.PR_SET_DUMPABLE 0and ignores signals.apps/daemon/internal/processshimowns both private wires, shim↔relay and relay↔broker (ipc.go).apps/daemon/internal/processbroker(root, one per view) keeps all Process protocol work: Start with an idempotent retry, Observe and re-attach, acks, offsets, signals, cancel, and Link loss and resume.Startedreaches the relay, and stdin forwarding begins, only on the service'sStartedevent, so an idempotent Start retry that attaches while the program is still starting never writes stdin early.Inspectand resends only the unaccepted bytes it still holds. If the leader has already exited, pipe stdin closes at that offset, so a background reader gets end of file. If the offset cannot be learned, the broker cancels the program and the shim exits 255 with the reason, unless the exit is already decided.Exitedis forwarded without waiting for output to drain.ErrRelayLostand never replays output.processbroker.Start(Config{Relay, Executables, Environment, Scope, Dial, CancelGrace, Logger}), plusClose(idempotent, bounded, never waits on a relay syscall),DoneandErr.apps/daemon/internal/sessionview/.oac/runtmpfs, owned by the Session user with mode 0600.View.Relay()returns the broker's end.oac-process-shim(agent.ViewRelayName) and Privaterunare reserved.WorldServer.Stopwhile the view is reaped, then waits at most 30s. If cleanup is incomplete,WaitandClosereturnErrCleanup.internal/sandboxprocess/termios_linux.go: the PTY mode table moves out ofprocessservice, so the service and the relay share one definition.agenthost: the Privaterundirectory andbrokerConfig.RunDirare removed. The broker is still not wired; plan row 9c wires it, one broker per view.Hardening removed, because the relay's own uid now gives it: the character-device allowlist, the devpts check, the root-only run directory,
SO_PEERCRED, explicitSCM_CREDENTIALSand the cached shim pid.Checks
go build ./...andgo vet ./..., plus the darwin and windows builds and vet.go test -race -count=1on agenthost, agent, processshim, processbroker and sessionview.A 30s
FuzzDecoderun.Privileged container:
The processbroker suite includes these regression tests:
TestOutputWritesHaveTheSessionsAuthority: oom_score_adj.TestStuckOutputDoesNotHoldTeardownandTestTeardownIsBounded: a FUSE stdout that never answers.TestUnixSocketOutputNamesTheRelay.TestBlockedOutputHoldsOnlyItsInvocation.TestBackgroundOutputAfterShimExits.TestRelayDescriptorsAreDiscarded.TestViewRunsRemoteShell.TestRetriedStartWaitsForStarted,TestUncertainStdinWriteResumes,TestUnresolvedStdinEndsTheInvocationandTestBackgroundReaderGetsEOFAfterUncertainWrite, against an in-process fake Process service.The processshim tests
TestOpensReachTheBrokerInIDOrderandTestTerminalOutputWaitsForRawModecover the relay ordering.Residual
Terminal output that arrives after the program's exit is written after the relay has restored the terminal, so the local terminal processes it a second time (a remote
\r\nbecomes\r\r\n). Holding the exit until the remote terminal closes would instead keep the shim alive for as long as a background job holds the terminal.If the world stops answering while the launcher is still building the view,
sessionview.Startcan hang past its context. This predates the rework, and plan row 9c fixes it.Intermediate commits before 233e71d don't build after the rebase. Squash-merge only.