feat: support running Sessions as a jobRunAsUser on macOS by andychoquette · Pull Request #335 · OpenJobDescription/openjd-sessions-for-python

andychoquette · 2026-07-16T23:32:42Z

What was the problem/requirement? (What/Why)

openjd-sessions could not run a Session's actions as a separate user (a
"jobRunAsUser") on macOS. Three problems in the POSIX cross-user path:

setsid is missing on macOS. The cross-user command is
sudo -u <user> -i setsid -w <cmd>, but setsid(1) is a Linux/util-linux
tool that does not exist on macOS. Every impersonated action failed at spawn
with command not found (exit 127), so running actions as a jobRunAsUser was
completely broken on macOS.
find_child_process_id_pgrep never retried. On non-Linux POSIX hosts,
signal-target discovery uses pgrep -P <sudo_pid>. pgrep exits 1 when it
finds no match yet (the child hasn't spawned), but the code treated any
non-zero exit as a fatal FindSignalTargetError, which broke out of the
caller's retry loop. Because Linux uses procfs and only other POSIX platforms
use pgrep, this latent bug meant signal-target discovery never actually
retried off Linux.
No is_macos() helper. The MACOS = "darwin" constant existed in
_os_checker.py but was unused, so there was no way to branch on macOS.

What was the solution? (How)

Add is_macos() to _os_checker.py.
Reproduce setsid behavior on macOS with a pure-Python shim. On darwin,
the workload is launched via
sudo -u <user> -i /usr/bin/python3 -I -c '<shim>' <cmd>, where the shim
makes the process a new session/process-group leader and then execs the
real command:
```
import os,sys;os.getpgrp()==os.getpid() or os.setsid();os.execvp(sys.argv[1],sys.argv[1:])
```
- os.getpgrp() == os.getpid() or os.setsid() calls setsid() only when the
  process is not already a group leader, avoiding the EPERM that os.setsid()
  raises for an existing group leader.
- Single line so it passes cleanly through sudo -i argv without shell-quoting
  fragility.
- /usr/bin/python3 (the OS interpreter) rather than sys.executable, so the
  jobRunAsUser can execute it without traverse/read permission on the agent's
  virtual environment.
- -I (isolated mode) so the session working directory is not on sys.path,
  preventing a file such as os.py in that directory from being imported ahead
  of the standard library before os.execvp() runs.
This produces the same topology setsid -w gives on Linux — the workload is
sudo's direct child in a process group distinct from sudo's — so the
existing find_sudo_child_process_group_id discovery works unchanged.
Treat pgrep exit code 1 as "no match yet" (return None so the caller
retries) rather than fatal. Exit >1 is still a genuine error. This fixes the
discovery/cancellation path on all non-Linux POSIX hosts, not just macOS.

The Linux and Windows code paths are unchanged.

What is the impact of this change?

macOS hosts can now run Session actions as a jobRunAsUser via sudo, matching
the existing behavior on Linux and Windows. On Linux and Windows there is no
behavioral change. The pgrep fix additionally repairs signal-target discovery
(and therefore process cancellation/cleanup) for any non-Linux POSIX platform.

macOS prerequisite: because the shim runs via /usr/bin/python3 (the Command
Line Tools interpreter), impersonated Sessions on macOS require the Xcode Command
Line Tools (xcode-select --install). This is documented in the README.

How was this change tested?

Added unit tests: is_macos() (test_os_checker.py); the pgrep exit-code
handling — exit 1 -> None/retry, exit >1 -> raise, single/multiple/empty
matches (test_sudo.py, new); the macOS cross-user command construction and a
POSIX check that the shim creates a new process group (test_subprocess.py).
Ran the full test/openjd/sessions_v0 unit suite: 550 passed, 33 skipped,
16 xfailed, no failures.
Validated end-to-end on macOS 26.5 (arm64) with the AWS Deadline Cloud worker
agent against a live customer-managed fleet: install -> worker registration ->
running an action as a jobRunAsUser -> cancellation reaping the workload's
process group with no orphaned processes.
Yes, unit tests were run.

Was this change documented?

Yes. Added a macOS note to the POSIX impersonation section of the README
(Command Line Tools requirement) and expanded the code comments in
_subprocess.py explaining the shim, the -I isolation, and the discovery
assumption. The is_macos()/pgrep behavior is covered by the new tests.

Is this a breaking change?

No. No public interface changes; Linux and Windows behavior is unchanged.

Does this change impact security?

This touches the cross-user (jobRunAsUser) impersonation path, which is a
security boundary. The shim is constructed as an argv list (no shell, not
injectable), uses a fixed absolute interpreter path, and -I prevents importing
attacker-placed modules from the working directory before exec. All work after
sudo -u <user> runs as the target job user (same as the existing Linux path),
so control of the command only ever lets the job run as itself. Flagging with the
"security" label so maintainers can review the boundary; happy to work through a
threat-model discussion.

Cross-port to openjd-rs

A tracking issue has been filed in openjd-rs to port this change (link here):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

openjd-sessions could not run a Session's actions as a separate user (jobRunAsUser) on macOS. Three problems in the POSIX cross-user path: - setsid(1) does not exist on macOS, so the cross-user command 'sudo -u <user> -i setsid -w <cmd>' failed at spawn (exit 127), breaking all jobRunAsUser execution on macOS. - find_child_process_id_pgrep treated pgrep's exit code 1 (no match yet) as a fatal error, which broke out of the caller's retry loop. Since Linux uses procfs and only other POSIX platforms use pgrep, signal-target discovery never actually retried off Linux. - There was no is_macos() helper (the MACOS constant was unused). Changes: - _os_checker.py: add is_macos(). - _subprocess.py: on darwin, launch the workload via a pure-Python setsid shim ('sudo -u <user> -i /usr/bin/python3 -I -c <shim> <cmd>') that makes the workload a new session/process-group leader before exec, reproducing the new-session behavior 'setsid -w' provides on Linux. Runs under /usr/bin/python3 (so the job user needs no venv access) with -I (isolated mode, so the working directory is not on sys.path). - _linux/_sudo.py: treat pgrep exit code 1 as "no match yet -> return None" so the retry loop polls as intended; exit >1 is still fatal. Fixes discovery and cancellation on all non-Linux POSIX hosts. - Tests for is_macos(), the pgrep exit-code handling, and the macOS command construction (plus a POSIX check that the shim creates a new process group). - README: document the macOS Command Line Tools prerequisite for impersonation. Linux and Windows behavior is unchanged. Validated end-to-end on macOS 26.5 (arm64) via the AWS Deadline Cloud worker agent against a live customer-managed fleet: running an action as a jobRunAsUser and cancellation reaping the workload's process group with no orphans. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

The impersonation tests already exist but xfail everywhere the OPENJD_TEST_SUDO_* environment variables are unset; on Linux they run inside a purpose-built Docker container, and nothing runs them on macOS. macOS runners have passwordless sudo, so this workflow provisions the same user/group layout with Directory Services (sysadminctl/dseditgroup) and runs the existing tests for real, covering the macOS setsid-shim launch, signalling, and process-tree termination paths end to end. A guard step fails the job if the tests regress to xfail (e.g. the provisioning breaks), instead of silently passing an empty run. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

leongdl

Please also port this to OpenJD-RS.

We're in the middle of migrating from Python to the Rust implementation.

andychoquette · 2026-07-20T19:14:03Z

Please also port this to OpenJD-RS.

We're in the middle of migrating from Python to the Rust implementation.

Will do - I opened issue OpenJobDescription/openjd-rs#263 to track the work

andychoquette · 2026-07-27T20:13:54Z

@leongdl added CI tests to verify this already works as expected in openjd-rs: OpenJobDescription/openjd-rs#285

…ession_action_on_error The test asserted the parse-error log message immediately after observing the session leave RUNNING, but the message is emitted by the subprocess stdout-filter thread, which can still be draining the pipe at that point. The race is intermittent on Windows CI (observed repeatedly on python 3.9/3.14 windows-latest runners), where it fails with the message absent from caplog and then passes on re-run. Poll briefly for the message instead of racing the thread. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

# Conflicts: # src/openjd/sessions/_linux/_sudo.py # test/openjd/sessions_v0/test_subprocess.py

epmog

Do we have an example of what the error looks like for the user when they don't have a python reachable by the target user? Might be good to validate that it's clear/actionable to the user.

Seven changes, all to test/CI scaffolding; no change to the shim or to _subprocess.py behaviour. 1. Delete test/openjd/sessions_v0/test_sudo.py. It duplicated the pre-existing TestFindChildProcessIdPgrep in test_linux_sudo.py case for case (5 of 5), and the existing versions are better: the no-match case drives real pgrep against a real childless process instead of mocking the return, the error case is parametrized over three exit codes and asserts the message content, and the adjacent TestFindSudoChildProcessGroupId covers the late-child race the fix exists for. Nothing was lost; full suite still 936 passed. 2. Move test_setsid_shim_creates_new_process_group out of TestLoggingSubprocessMacOSSetsid into a new POSIX-scoped TestSetsidShimBehavior. It was macOS-named but gated on is_posix(), so it ran on Linux inside a macOS class. Kept POSIX rather than narrowed to macOS: the shim is portable stdlib (os.getpgrp/getpid/setsid/execvp, no platform branch) and macOS is only where it is *required*, so running it on Linux too catches a broken shim string on faster runners. Whether macOS selects the shim is a separate concern already covered by test_builds_setsid_shim_command_on_macos. 3. Add a class-level macOS skipif to TestMacOSShimInterpreter. Unlike the shim string, _macos_shim_interpreter() really is macOS-only selection logic. 4. Extract the workflow's ~80 lines of inline provisioning into scripts/run_macos_sudo_tests.sh, mirroring scripts/run_sudo_tests.sh so the job is reproducible on a developer's Mac. Because macOS cannot be containerized, the script is the counterpart to the Linux Dockerfile rather than to the docker command: it provisions, runs and then removes the users, groups, sudoers file, python symlink and temp root it created. Teardown is best-effort throughout so a partial provision is still removable, and --keep / --cleanup-only cover the CI and recovery cases. 5. Add cross-user-test / cross-user-test-macos hatch scripts so neither platform's cross-user suite requires remembering a script path. 6. Expand the Python matrix from ['3.11','3.13'] to 3.9-3.14, matching code_quality.yml and requires-python >=3.9. 7. Drop the paths: filter so the job runs on every PR. The cross-user path can break from more places than a four-file list can enumerate, and a filtered job that misses one reads as a pass. Also drops the workflow's final "run remaining tests" step: code_quality.yml already runs the whole suite on macos-latest across this same matrix, so that step only duplicated it. This job now covers the cross-user tests alone. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

_macos_shim_interpreter() returned /usr/bin/python3 without checking it. When the Command Line Tools are absent, or the fallback is itself unreachable, that path deferred the failure to Popen, and the operator saw only: Process failed to start: [Errno 2] No such file or directory: '/usr/bin/python3' on a workload that may have nothing to do with Python, naming neither the cause (cross-user execution needs an interpreter the job user can execute) nor the fix. The fallback is now verified the same way as the base interpreter, and when neither is usable a NoReachableInterpreterError names both candidates that were tried and the remedy. The caller is unchanged: _start_subprocess catches it, logs "Process failed to start: {message}" and returns None, so the launch still fails via the existing failed_to_start path rather than propagating. Also fixes the shim symlink in scripts/run_macos_sudo_tests.sh. `ln -sf` replaced whatever was at /usr/local/bin/python and cleanup() removed it unconditionally, so a default run would have deleted a developer's pyenv/Homebrew/python.org symlink, contradicting the script's promise to remove only what it created. It now creates the alias only when the path is empty, records whether it did, and removes it only in that case. --cleanup-only, which runs before provisioning, claims the alias only when it points at /usr/bin/python3. Tests: the existing fallback test now models a reachable fallback rather than patching every path unreachable. Two added: one asserting the raised message names both candidates plus 'xcode-select --install', and one driving _start_subprocess to pin that the text survives to the log, since the caller swallows the exception and the message is the operator's only diagnostic. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

andychoquette · 2026-08-10T15:23:18Z

Do we have an example of what the error looks like for the user when they don't have a python reachable by the target user? Might be good to validate that it's clear/actionable to the user.

Good q. It wasn't clear at all: Process failed to start: [Errno 2] No such file or directory: '/usr/bin/python3'. Fixed in latest commit.

The 3.9 leg failed at `hatch env create` with "Environment `default` is incompatible: module 'virtualenv.discovery.builtin' has no attribute 'propose_interpreters'". virtualenv 21 removed that API, and the hatch version resolvable on 3.9 still calls it. Provisioning had already succeeded, so the job failed after creating users and groups. This surfaced now because expanding the matrix to 3.9-3.14 added the only affected version; at 3.11/3.13 the job never installed a virtualenv that old. Every other workflow in this repo already carries the same constraint (code_quality.yml, both e2e workflows, release_publish.yml), so this brings the new job in line rather than inventing a fix. The environment marker scopes it to 3.9 alone, leaving 3.10+ on current virtualenv. Also surface the cause in scripts/run_macos_sudo_tests.sh: a developer running it locally on 3.9 hits the same failure, and hatch's message names neither virtualenv nor the remedy. `hatch env create` failing now prints the pin to try, gated on the interpreter actually being older than 3.10. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

Two problems in run_macos_sudo_tests.sh, both only dangerous on the developer machine the script exists to serve. 1. `chmod -R o+rX "$(pwd)"` made the entire working tree world-readable, including untracked files and anything credential-bearing a developer keeps under the repo root, and nothing put those bits back. The impersonated user only needs to read test/openjd/sessions_v0/support_files, so the grant is now scoped to that directory plus o+x (traverse only, no read) on the directories leading to it. Verified: the support file becomes world-readable, the path directories become 701 so they cannot be listed, and a .env at the repo root stays 600. These bits are not undone by the teardown, which the header now says explicitly rather than leaving it to be discovered. 2. TMPDIR was read from TMPDIR_OVERRIDE, an undocumented name that silently ignored a caller's own TMPDIR, and cleanup() does `rm -rf` on the result. So `TMPDIR_OVERRIDE=/tmp scripts/run_macos_sudo_tests.sh` would have ended in `sudo rm -rf /tmp`. The override is gone: the tests need any world-traversable already-resolved directory, so there was nothing to configure. Removal is also now gated on having created the directory, matching the treatment of the /usr/local/bin/python alias, so an existing one is reused and left in place. Also widens the --help line range, which the longer header had outgrown. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

Three review findings. 1. TEST_USER is interpolated into /etc/sudoers.d and comes from SUDO_USER, an inherited environment variable, so a caller controlled the content of that file. `visudo -cf` did not protect against it: the file is written before the check runs, so a rejected file still lands on disk, and a payload ending in '#' comments out the remainder and validates cleanly (confirmed: visudo exits 0 on "attacker ALL=(ALL) NOPASSWD: ALL\n# ..."). TEST_USER is now checked against ^[A-Za-z0-9._-]+$ and must resolve via `id -u` before it is used anywhere, which also guards the dseditgroup and chown calls that take it. 2. The class-level macOS skipif I added on TestMacOSShimInterpreter was swallowing the four _other_users_can_execute tests, whose own is_posix / is_windows markers had become unreachable. That function decides whether cross-user execution is possible at all and is plain permission-bit logic, so it was getting zero coverage on the Linux and Windows legs -- the opposite of the reasoning applied to TestSetsidShimBehavior. Moved to a POSIX-scoped TestOtherUsersCanExecute, which makes those per-test markers meaningful again. 3. Nothing exercised the shim through `sudo -i`. The shim contains ';', '(', ')', '[', ']' and '=', and sudo -i composes a login-shell command line, so quoting is the one link with real risk -- and every existing test bypassed it by passing an argv list to sys.executable or by asserting only the constructed list. Added test_shim_survives_sudo_login_shell_quoting, which self-sudos (target == current user, already permitted by the provisioning rule) and asserts both a zero exit and pid == pgid, since a mangled shim shows up as a failed launch rather than a wrong pgid. Verified by hand on macOS 26.5 that the real chain works. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

Three review findings. 1. The shim leaked CPython's signal ignores into the workload. CPython sets SIGPIPE and SIGXFSZ to SIG_IGN during startup, and SIG_IGN survives exec (installed handlers do not), so every impersonated macOS workload ran with SIGPIPE ignored: `producer | head` would get EPIPE write errors instead of dying on the signal. Popen's restore_signals does not help because the process it starts is the shim, which re-ignores them, and os.execvp has no equivalent. Linux's setsid(1) is a C binary that never touches these, so this was a real cross-platform divergence. Both are now reset to SIG_DFL before the exec. Confirmed with a C probe reading sigaction directly: through the old shim a workload saw SIGPIPE=SIG_IGN, plain it sees SIG_DFL, and with the fix it sees SIG_DFL again. The blocked-signal mask is also inherited across exec but CPython leaves it empty, so no pthread_sigmask reset is needed. Pinned by test_shim_restores_default_signal_dispositions, which probes with perl: a Python process reports SIG_IGN for SIGPIPE regardless of what it inherited, and /bin/sh's `trap -p` prints nothing for an inherited ignore, so neither can distinguish the two cases. Verified the test reports IGNORE against the pre-fix shim, so it is a real guard rather than a tautology. 2. TMPDIR is a fixed path under world-writable, sticky /private/tmp, and the reuse branch applied `chown`/`chmod 1777` to whatever was already there. `mkdir -p` succeeds silently on an existing symlink-to-directory and both chown and chmod follow symlinks, so a local user could pre-create it as a symlink and redirect those calls (verified: chmod through a symlink changes the target's mode). Now created with plain `mkdir` so an existing path is a hard error telling the operator to inspect it and re-run with --cleanup-only. 3. Documented that _other_users_can_execute() only inspects the interpreter file: it says nothing about whether the target user can read that interpreter's stdlib or framework dylib, so an interpreter with o+x on the binary but o-rx on .../lib passes and then fails at runtime with "Fatal Python error: init_fs_encoding". A live `-I -c ""` probe as the target user would cover it but is not worth the per-launch cost; the docstring now says what the check does and does not guarantee. Also records the discovery-timing change the shim introduces, which the previous "same assumption already holds for Linux" comment glossed over: the workload's pgid only differs from sudo's once the shim interpreter has booted and reached setsid(), which is a full CPython startup rather than Linux's tiny C binary. Measured on macOS 26.5 (arm64) with /usr/bin/python3: ~120-180ms idle, ~165-190ms with every core saturated, against the 1s default in find_sudo_child_process_group_id -- comfortable but not enormous, and the comment names raising that timeout as the fix if the margin ever proves thin. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

The hard error added in the previous commit told the user to re-run with --cleanup-only, but that path could not remove the directory: cleanup() only rm -rf's the temp root when TMPDIR_CREATED is "True", and the --cleanup-only branch set only PYTHON_SHIM_CREATED. So the documented recovery printed "Done", removed nothing, and the next run hit the same error, with no way forward that the message mentioned. --cleanup-only now re-establishes TMPDIR_CREATED from disk the same way it already does for the python alias: it claims a real directory (provision() only ever creates this fresh, so one found here is ours from an interrupted run) and refuses a symlink, which this script never creates and which would let the rm -rf be redirected at the target. A symlink gets a warning naming what to do instead. The hard error also distinguishes the two cases now, rather than pointing at --cleanup-only for a symlink it deliberately will not touch. Verified all three states: a leftover directory is claimed and removed, a symlink is refused with its target intact, and an absent path is a no-op. Signed-off-by: andychoquette <78888816+andychoquette@users.noreply.github.com>

github-actions Bot added the waiting-on-maintainers label Jul 16, 2026

andychoquette force-pushed the macos-support branch from 51dbf6e to a3128e6 Compare July 16, 2026 23:39

andychoquette marked this pull request as ready for review July 16, 2026 23:48

andychoquette requested a review from a team as a code owner July 16, 2026 23:48

This was referenced Jul 16, 2026

Feature: Add and validate macOS (darwin) support for cross-user Sessions OpenJobDescription/openjd-rs#263

Open

feat: add macOS (darwin) support to the worker agent installer aws-deadline/deadline-cloud-worker-agent#1012

Open

crowecawcaw reviewed Jul 17, 2026

View reviewed changes

Comment thread src/openjd/sessions/_subprocess.py Outdated

andychoquette force-pushed the macos-support branch from a3128e6 to fcfd47d Compare July 17, 2026 15:58

crowecawcaw previously approved these changes Jul 17, 2026

View reviewed changes

andychoquette dismissed crowecawcaw’s stale review via ce53771 July 17, 2026 17:43

andychoquette enabled auto-merge (squash) July 17, 2026 19:16

leongdl reviewed Jul 20, 2026

View reviewed changes

epmog requested changes Jul 20, 2026

View reviewed changes

Comment thread src/openjd/sessions/_subprocess.py

epmog disabled auto-merge July 20, 2026 18:55

epmog removed the waiting-on-maintainers label Jul 21, 2026

andychoquette added 2 commits July 22, 2026 11:14

Merge branch 'mainline' into macos-support

efe96e6

Merge branch 'mainline' into macos-support

ea87e9c

andychoquette requested a review from epmog July 27, 2026 19:28

andychoquette mentioned this pull request Jul 27, 2026

test(sessions): run POSIX cross-user tests on macOS in CI OpenJobDescription/openjd-rs#285

Open

andychoquette force-pushed the macos-support branch from d4b2b35 to e40eee5 Compare July 27, 2026 20:35

andychoquette added 3 commits July 29, 2026 09:40

Merge remote-tracking branch 'origin/mainline' into macos-support

449baa0

# Conflicts: # src/openjd/sessions/_linux/_sudo.py # test/openjd/sessions_v0/test_subprocess.py

Merge branch 'mainline' into macos-support

1ef2e51

Merge branch 'mainline' into macos-support

843cd05

crowecawcaw previously approved these changes Aug 7, 2026

View reviewed changes

epmog requested changes Aug 7, 2026

View reviewed changes

andychoquette dismissed crowecawcaw’s stale review via bdcffd5 August 10, 2026 14:45