Skip to content

processgroup: macOS reads process groups, so left-running services and the job bound work there - #2039

Merged
AbirAbbas merged 8 commits into
app/v3from
feat/darwin-process-inspection
Nov 3, 2026
Merged

AbirAbbas merged 8 commits into
app/v3from
feat/darwin-process-inspection

Conversation

@AbirAbbas

Copy link
Copy Markdown
Collaborator

Found in the macOS run of the engine suite (Apple M4 Pro, macOS 15.6): TestAShellCallTellsTheObserverWhatItLeftRunning and TestLocalRecordsAListeningServiceFully fail at app/v3 12dadd64a because darwin had no process-group inspection at all.

What was wrong (macOS)

  • processgroup.Members returned nil (members_other.go), Group.Usage returned "cannot say" (usage_other.go), and the executor's cmdline / listenPorts returned nil (service_other.go).
  • So on a Mac a server a shell command left running was never reported (executor.Left saw no members), a service record was named after the shell with no ports, and inventory refreshes had no ports.
  • And the job subtree bound could never cut (internal/session/jobbound.go: more than 128 processes, or more than three fifths of the machine's cores where nothing caps them, for 15 readings 2 s apart). A runaway or fork-storming bash job ran unbounded on macOS.

Reproduced in the real engine on this Mac (codeaf app spawned as the desktop spawns it, fake provider issuing one bash call):

app/v3 12dadd64a (shipped sidecar) this branch (garbled sidecar, CGO_ENABLED=0)
python3 -m http.server 0 --bind 127.0.0.1 & left running listening (lsof), but inventory.json has no services and no running services: [{name: python3, version: Python 3.10.19}], running: [{command: "python3 -m http.server 0 --bind 127.0.0.1", ports: [63451]}] — the port lsof shows
background job for i in $(seq 140); do sleep 90 & done; wait never cut; 140 sleepers until the session closed cut at 30.0 s (15 × 2 s); the run is told "the job subtree was not settled within its bound: job 1 was holding 141 processes"

The same checks in the real desktop app (UI2_CODEAF_BIN pointing at the branch sidecar) are queued behind the machine's performance baseline; results will be added here.

What changes

darwin readers, no cgo (new internal/darwinproc, darwin-only files):

  • Membership: kern.proc.pgrp through unix.SysctlRaw with a bounded retry (x/sys's SysctlKinfoProcSlice retries forever while the table grows); only the pid, state and pgid fields are read from each record; zombies excluded; sorted.
  • Subtree usage: a descendant walk from the recorded leader (proc_listpids(PROC_PPID_ONLY) + proc_pidinfo(PROC_PIDTASKALLINFO)), so a child that changed process groups is still the job's, with owned() checked before and after.
  • Argv: kern.procargs2, parsed at the kernel's eight-byte boundary after the executable path, stopping before the environment.
  • Listening ports: PROC_PIDLISTFDS + PROC_PIDFDSOCKETINFO, IPv4/IPv6 TCP in LISTEN only.

Why the raw proc_info trap and not a //go:cgo_import_dynamic libproc bridge. Both work without cgo. The trap (unix.Syscall6(unix.SYS_PROC_INFO, …), call 1 listpids, 2 pidinfo, 3 pidfdinfo) goes through Go's own, maintained per-architecture syscall stubs (arm64 x16 + svc 0x80; amd64 adds the 0x2000000 class), so there is no assembly to write or test per architecture and no //go:linkname into syscall for garble or the linkname rules to break. It also returns errno directly; libproc's wrappers return 0 and leave errno in thread-local storage, which the runtime's libc bridge does not capture for a 0 return, so "gone" and "empty" could not be told apart. Caveat: Apple documents neither the trap's call numbers (XNU's proc_info_private.h) nor libproc (its header calls the interface private and subject to change); both have been unchanged since macOS 10.5. Verified on this Mac natively on arm64, as a GOARCH=amd64 binary under Rosetta, and in the garbled, stripped sidecar (make app-sidecar).

CPU units. pti_total_user/pti_total_system are Mach absolute-time ticks, not nanoseconds: 0.267 s of CPU read as 6,406,888 ticks here. Under Rosetta the kernel still returns 24 MHz ticks while mach_timebase_info reports 1/1, so the conversion divides by hw.tbfrequency (24,000,000 here; 1e9 on Intel Macs, where ticks are ns). TestDarwinTaskMatchesSelf checks it against getrusage on both architectures.

Robustness of a reading (all from the adversarial review):

  • A queued parent is re-read after its children are listed and its start time compared, so a pid handed to another process cannot bring a stranger's children in.
  • The walk stops at 1,024 processes per reading; a child listing that fills the largest buffer is used as far as it goes. Both give a lower-bound count (a fork storm is still cut on the count) and mark the CPU sum partial.
  • New Usage.CPUPartial (zero value = complete, so Linux is unchanged): the job bound takes no rate from a partial sum and does not use it as the next baseline, so a process missing from one reading and present in the next cannot read as a burst and invent a strike.
  • A reading that fails outright is "cannot say", which never cuts.

Executor. describe reads each member's sockets once and derives both the naming member and the port set from that (it used to read every member twice). Naming, argv fallback, DataDir and the per-platform nil-vs-empty Ports are unchanged.

Tags. members_other.go, usage_other.go, service_other.go are now !linux && !darwin (Windows/other BSDs keep "cannot say"); the two service tests run on linux || darwin. TestLocalRecordsAListeningServiceFully's shell now waits (bounded) for the server's port before it exits, so the record the call returns is asserted directly; on macOS the call otherwise often returns while the backgrounded child is still the forked shell.

Behaviour change on macOS. A registered bash job (a background: true command, or a foreground command kept as a job) that sustains more than three fifths of the logical cores (7.2 of 12 on this M4 Pro) for 30 s, or holds more than 128 processes for 30 s, is now ended, as on Linux. Thresholds are unchanged. The manual gains a section on the bound (what-i-can-do.md) and the change entry says it.

Cost (benchmarks on this Mac, -cpu 1, arm64 / amd64 under Rosetta)

ns/op B/op allocs
Task (one TASKALLINFO) 312 / 894 0 0
Children (one PPID listing) 13,043 / 8,071 8 1
ListenPorts (test process) 1,825 / 1,994 392 2
Argv 9,186 / 8,152 152 6
Members (10-process group) 19,245 / 19,570 16416 5
Usage (10-process tree) 116,011 / 105,052 3656 14

Usage runs every 2 s per registered bash job; 16 such jobs cost about 1 ms of CPU per second (≈0.1% of one core). No ps, no polling beyond what callers already do (Left once per finished shell call, GroupPorts per live item at a seal).

Tests

  • internal/darwinproc: TestDarwinProcArgsStopsBeforeEnvironment (exact padding, empty argv[0] with environment after it, malformed layouts), TestDarwinArgvKeepsAnEmptyFirstArgument (live, four path lengths), TestDarwinArgvReadsALiveChild, TestDarwinTaskMatchesSelf, TestDarwinChildrenListsASpawnedChild, TestDarwinListeningPorts (v4 + v6 listeners; connected TCP, UDP and unix sockets excluded), benchmarks.
  • internal/processgroup: TestDarwinMembersExcludeZombies, TestDarwinMembersReadsALiveGroup, TestDarwinUsageReadsDescendants, TestDarwinUsageRefusesUnownedGroup, and scripted-tree walk tests (…CountsTheWholeTreeOnce, …KeepsAReusedParentsChildrenOut, …CountsATruncatedListingAndRefusesAFailedOne, …StopsAtTheWalkLimit, …MarksAnUnreadableChildPartial, …MarksAnUnreadableParentPartial), benchmarks.
  • internal/session: TestAPartialCPUSumNeverMakesARate (fails on the old strike code), TestAPartialReadingStillCountsTowardsTheProcessBackstop.
  • internal/executor: TestDescribeInspectsEachMemberOnce (portable).

Gates (on the Mac)

  • gofmt; go vet of ./internal/darwinproc ./internal/processgroup ./internal/executor ./internal/session ./internal/inventory (darwin), and of the first three for linux, windows and freebsd (session too for linux).
  • CGO_ENABLED=0 go build ./cmd/codeaf for darwin/arm64, darwin/amd64, linux/amd64, linux/arm64 and windows/amd64; the garbled, stripped desktop sidecar (make app-sidecar, darwin/arm64) builds and is the binary the manual checks used.
  • On the Mac: internal/darwinproc, internal/processgroup, internal/inventory pass; internal/executor passes except TestWorkspaceEnvironmentRoutesRealTemporaryFilesThroughBothRunnerDoors (fails at 12dadd64a too: BSD mktemp). The two R4 tests pass, 10/10 in a row. The same darwin tests pass as GOARCH=amd64 binaries under Rosetta.
  • internal/session: the job-bound tests and the two new ones; -run Manual in internal/session and internal/tui3; internal/manual, make test-packed-manual, internal/namelaw pass. internal/commentlaw fails only on the two files it fails on at 12dadd64a (factory_pipeline_copy.go, pipeline_copy.go).
  • Mac gate, full go test -count=1 -p 6 ./... compared by test name with the 12dadd64a Mac baseline: running (results will be added here).
  • CI, judged by content against app/v3 12dadd64a: PR gate 40934512927 fails exactly app/v3's 5 (commentlaw, guard, session ×2, taxonomy); its touched-packages job (processgroup, executor, session, …) passes on Linux. Full check 40934517251: first run: 3 tests outside the baseline failed, all intermittent (two reproduced at 12dadd64a locally, 1 in 30; the third is the known Mac/CI flake TestFactoryAssigneesRunnerPublishesTeamAndSolo); failed jobs re-run, results will be added here.
  • The branch is on 12dadd64a; git merge-tree onto the new app/v3 (1c4ec8a25, pair: merge a computer's own chats when it joins, instead of replacing them #2031) is clean and pair: merge a computer's own chats when it joins, instead of replacing them #2031 touches none of these files.

Codex review (adversarial, read-only, 3 rounds)

  • Round 1 (NOT SAFE): pid reuse of a queued parent → fixed (start-time re-read); a failed or truncated listing became a complete reading → fixed (unexpected failure = unknown; truncation = lower bound); no work budget for a fork storm → fixed (1,024-process walk limit); an empty argv[0] could shift an environment string into argv → fixed (kernel boundary parse, verified for path lengths 1–40 × three argv[0] values, 0/120 mismatches, both arches); the service test repaired its result outside production → fixed (fixture handshake); pr: 0 → kept.
  • Round 2 (NOT SAFE): a partial CPU sum could invent a strike → fixed (CPUPartial, baseline cleared; regression test).
  • Round 3 (NOT SAFE): a parent whose re-read failed (not "exited") left its children out without marking the sum partial → fixed after the round (…MarksAnUnreadableParentPartial); not re-reviewed by Codex. Everything else accepted.

The change entry follows app/v3's convention (docs/changes/unreleased/0-app-<slug>.md, pr: 0, as in #2014–#2016). codeaf-changes check reports "pr is missing" for every such entry on app/v3 (249 at the base).

Not covered

  • Descendants that leave the process group are counted by the bound but are not reached by group signals; this was true on Linux already.
  • TestWorkspaceEnvironmentRoutesRealTemporaryFilesThroughBothRunnerDoors still fails on macOS (BSD mktemp ignores TMPDIR; a test assumption, unrelated).

🤖 Generated with Claude Code

AbirAbbas added a commit that referenced this pull request Oct 29, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas force-pushed the feat/darwin-process-inspection branch from dfd4641 to 3406119 Compare October 29, 2026 23:17
@AbirAbbas

Copy link
Copy Markdown
Collaborator Author

Closing unmerged: the Mac push stopped early to save usage. What this fixes and what is left is tracked in #2055. The branch feat/darwin-process-inspection is kept.

AbirAbbas and others added 8 commits November 2, 2026 18:32
Members lists a group through kern.proc.pgrp with a bounded retry, and
Group.Usage walks the recorded leader's descendants through the kernel's
proc_info interface, converting task time with hw.tbfrequency. The new
darwinproc package owns the raw trap, the SDK layouts and the argv and
listening-socket readers the executor uses next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmdline reads kern.procargs2 and the ports come from each member's TCP
listeners. describe reads each member's sockets once for both the naming
member and the port set. The two service tests run on Linux and macOS, and
the listening-service test polls the record until the backgrounded shell
has exec'd its server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes:
- A queued parent is re-read after its children are listed and its start
  time compared, so a pid handed to another process cannot bring that
  process's children into the job's reading.
- A child listing that fills the largest buffer, and a walk that reaches
  1024 processes, count as lower bounds; any unexpected listing failure makes
  the reading unknown instead of partial.
- kern.procargs2 is parsed at the kernel's eight-byte boundary after the
  executable path, so an empty argv[0] can never shift an environment string
  into argv (checked for path lengths 1-40 on arm64 and under Rosetta).
- The listening-service test's shell now waits for the server's port before
  it exits, so the record the call returns is asserted directly; the
  membersOf helpers it no longer needs are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fix: a reading that left part of the tree out (the walk limit, a
truncated child listing, a process that could not be read) still counts
processes as a lower bound, but its CPU sum is marked partial. The job bound
takes no rate from it and does not use it as the next baseline, so a process
missing from one reading and present in the next cannot read as a burst of
its whole lifetime's CPU and invent a strike. Linux readings are always
complete, so nothing changes there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fix: when a queued parent read once but its re-read after the child
listing fails for a reason other than having exited, its children are left
out of this reading only, so the CPU sum is marked partial and no rate is
taken from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fix: the macOS walk now reads a parent's listed children first and
re-reads the parent after them. A pid is never handed on while its process
lives, so a parent with the same start time after its children were read was
their parent while they were read; a parent and child whose pids both changed
hands just after the parent's own re-read can no longer bring a stranger's busy
child into the job's reading.

The partial-sum regression now starts from one complete, quiet reading, so it
also fails if a partial reading ever keeps that earlier baseline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@AbirAbbas AbirAbbas reopened this Nov 2, 2026
@AbirAbbas
AbirAbbas force-pushed the feat/darwin-process-inspection branch from 096e996 to 0cf7152 Compare November 2, 2026 23:32
@AbirAbbas
AbirAbbas marked this pull request as ready for review November 3, 2026 00:33
@AbirAbbas
AbirAbbas merged commit 2c7f108 into app/v3 Nov 3, 2026
23 of 35 checks passed
@AbirAbbas
AbirAbbas deleted the feat/darwin-process-inspection branch November 3, 2026 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant