Skip to content

fix(supervision): stabilize grace probes and exited-group cleanup - #60

Merged
senamakel merged 11 commits into
mainfrom
fix-tinybox-ssh-supervisor-grace
Oct 12, 2026
Merged

senamakel merged 11 commits into
mainfrom
fix-tinybox-ssh-supervisor-grace

Conversation

@senamakel

@senamakel senamakel commented Oct 11, 2026 •

Copy link
Copy Markdown
Member

The SSH supervisor grace-period fixture combined 100 mandatory supervisor probes with an optional child probe interrupted by TERM. Count them separately while preserving the exit-status and cleanup assertions.

CI exposed two additional blockers: generated Docker executables can fail with ETXTBSY when concurrent forks retain a script writer, and jailed cleanup on macOS can reject EPERM after the process group has already exited. Docker fixtures now use an immutable launcher that sources generated commands. Jail cleanup accepts EPERM only after a fresh process-table probe confirms no live members; live groups and failed probes still fail cleanup.

Public APIs: None. Behavior: jailed cleanup acknowledges an already-exited process group after confirmed inspection, matching the existing local supervisor. No dependencies, release changes, or host gitlink changes.

Validation:

  • Docker writer-retention and jail exited-group regressions failed before their fixes and passed afterward. Jail coverage also checks live groups and inspection failures.
  • 500 consecutive Docker fixture suite runs passed.
  • cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, and cargo build --all-targets --all-features passed.
  • cargo test --all-features passed, including doctests, on rerun. The first run hit the existing SSH TERM-escalation fixture’s 10-second timeout under host load; that test passed unchanged in isolation and in the complete rerun.
  • Default-feature host and module unit suites passed before the base update (81 and 108 tests).
  • Merged the updated main branch, preserved its Docker probe/retry regression, and reran all four contract checks successfully. The merged module suite passed 137 tests.
  • Original SSH regression retained: exactly 100 supervisor probes and at most one child probe, with exit-status and directory-cleanup assertions.

Follow-up to #53; supports tinyhumansai/openhuman#7391 under tinyhumansai/openhuman#7390.

senamakel and others added 2 commits October 11, 2026 23:48
The fake ps stub now inspects the caller's process group and logs the
parent pid to a separate file when the call comes from the supervised
group, so the test can tell group-scoped inspection apart from other
invocations.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The observed host test now reads the child process inspection log and
checks that the child performs at most one process-group inspection
before TERM, while the supervisor grace probe count is adjusted to 100.
This separates the optional child probe from the supervisor's bounded
TERM grace period so the assertion reflects the intended behaviour.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7a30b0e6-ba62-49f2-b35c-7f4efbb47645





📥 Commits

Reviewing files that changed from the base of the PR and between 31181eb and 2081c24.






📒 Files selected for processing (1)
  • crates/tinybox-ssh/src/host/observed_tests.rs





Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.







📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough

Walkthrough

The process-inspection failure test now records supervisor and child process-group calls separately. It checks for exactly 100 supervisor probes and permits at most one child inspection.

Changes

Process inspection test

Layer / File(s) Summary
Separate process-group probe counts
crates/tinybox-ssh/src/host/observed_tests.rs
The fake ps command logs calls separately by process group. The test expects exactly 100 supervisor probes and permits at most one child inspection.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other












Merge Risk: ⚪ Minimal · up to 2081c

The test now distinguishes child inspections from the supervisor’s 100 grace probes. No production behavior change or concrete user-facing risk is identified, so the change appears ready to merge.

Architecture Summary

Architecture risk: 🔵 Low · up to 2081c

The change affects 1 system.

Changed systems: crates

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

  • observed — crates (service) was modified; 1 changed file maps to changed impact.

Before / after behavior

  • observed — Modified behavior in crates/tinybox-ssh/src/host/observed_tests.rs: The fake ps command now compares its process group with the supervised group and logs child-group calls separately from supervisor calls before returning failure.
  • observed — Modified behavior in crates/tinybox-ssh/src/host/observed_tests.rs: The test now reads and removes the optional child-call log, expects exactly 100 supervisor probes instead of 101 combined calls, and permits at most one child inspection.





Pre-merge checks | Passed 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly identifies the main change: stabilizing supervisor grace probes. The exited-group cleanup reference is not a primary change, but it remains related to the test behavior and does not …

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR









  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks the probes at night,
The supervisor count is right.
A child call gets its own track,
One may pass, but not a stack.
The test now keeps each count in sight.

Comment @coderabbitai help to get the list of available commands.

@senamakel senamakel self-assigned this Oct 12, 2026
senamakel and others added 6 commits October 12, 2026 07:48
The SIGKILL handling for a jailed process group now lives in a
kill_jail_group helper so the ESRCH and error paths can be exercised
directly. Tests cover the permission-error case, asserting cleanup
succeeds once the group is gone and still fails while it is alive.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Darwin can deny signalling a process group that exited between inspection
and SIGKILL, so kill_jail_group now checks whether the group still has live
members before reporting the EPERM as a failure. Tests cover the empty-group
case alongside the existing ESRCH handling.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Log Docker startup and container listing errors to stderr to aid
diagnosis of failures in the oneshot path. The logging is marked as
temporary and should be removed once the underlying issue is understood.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The docker tests now share a write_fake_cli helper that writes the script and
marks it executable, replacing the repeated permission boilerplate. Debug
eprintln calls were dropped from the oneshot error paths, and a new test covers
running a fixture while an inherited script writer is still open.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
The fake Docker CLI is now a symlink to a fixed launcher script that sources
the generated body, so a concurrently forked process holding a writer open no
longer makes the script unexecutable. The README also documents that
JailLocalHost rechecks the process table when a jailed group exits before
SIGKILL.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
Relocate the test module declaration to the bottom of the file so the
production code reads without an interruption. No behaviour change.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@senamakel senamakel changed the title test(ssh): count supervisor grace probes separately fix(supervision): stabilize grace probes and exited-group cleanup Oct 12, 2026
senamakel and others added 2 commits October 12, 2026 07:56
Reword the doc comment and inline comment on the fake CLI fixture helpers to describe publishing through an immutable launcher and confirming execution before the test starts, keeping the existing ETXTBSY retry contract.

Auto-committed-on: dragonfly
Co-authored-by: Medulla <medulla@tinyhumans.ai>
@senamakel
senamakel merged commit 01ac8ae into main Oct 12, 2026
14 checks passed
@senamakel
senamakel deleted the fix-tinybox-ssh-supervisor-grace branch October 12, 2026 05:04
@tinysweeper

tinysweeper Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

This pull request stabilizes supervision cleanup in three areas. First, the jailed-process group SIGKILL is extracted into a testable kill_jail_group helper that accepts an EPERM error only after a fresh process-table probe confirms the group has no live members, keeping cleanup unacknowledged when live members remain or the probe fails. Second, the Docker test fixture is rewritten to launch through an immutable symlinked fixture script (docker-cli.sh) that sources a generated body as data, so inherited writers on the script body can no longer cause execution failures, plus a new test covering execution while a writer is open. Third, the SSH supervisor test's fake ps is refined to separate supervisor grace probes from a child's own optional group inspection, tightening the call-count assertions. The limited-mode README documents the new exited-group cleanup behavior. No reporting lane raised a blocking defect. The tests lane confirmed the helper's branches are pinned; residual unverified concerns include whether docker-cli.sh's sourcing of $0.script behaves correctly in every fake-CLI scenario, and the description lane raised unsettled compile questions about the alive closure's FnOnce bound and the symlink API call site.

State: Incomplete
Priority: none
Reviewed head: f1800ffcd5cf
Updated: 2026-10-12T21:24:09Z

Review snapshot

Change surface Files Review signal Count
Production 1 Active findings 0
Tests 4 Noted findings 0
Documentation 1 Resolved findings 0
Configuration 0 Pending checks/questions 5

Completeness: Incomplete
Test assessment: Test coverage is assessed from changed tests and lane evidence; execution is not claimed without trusted check data.

Features

  • Modified — SSH supervisor ps-call accounting distinguishes supervisor grace probes from child inspections: The fake ps now records calls separately depending on whether the caller's process group matches the supervised group. The expected supervisor grace probe count is adjusted (from 101 to 100) with an updated assertion message, and a new assertion bounds the child's own optional group inspection to at most one before TERM, clarifying that the child's probe is separate from the supervisor's bounded grace period. (crates/tinybox-ssh/src/host/observed_tests.rs#fn generated_supervisor_keeps_grace_period_when_group_inspection_fails() -> Resu)

Tests

  • added — jail_permission_error_after_group_exit_is_successful_cleanup exercises every branch of the kill_jail_group helper: Ok from kill, ESRCH, EPERM with a dead group (accepted), EPERM with a live group (rejected), and EPERM with a failing liveness probe (rejected).: The tests lane confirmed the helper's error paths are exercised with asserting cases. The EPERM branch is covered via injected closures rather than a real Darwin process group, which cannot be exercised from this platform. (crates/tinybox-host/src/limited/jailed_tests.rs#async fn cleanup_retry_does_not_wait_for_or_cancel_an_active_jailed_execution())
  • added — docker_fixture_runs_while_an_inherited_script_writer_is_open publishes the fixture, opens a writer on the generated script body, and runs a Docker status request through the launcher, asserting the daemon reports as available despite the inherited writer.: Directly regression-tests the reason the fixture was rewritten to an immutable launcher; the tests lane found assert!(status.available) constrains behavior. The tests lane flagged one unverified concern: it did not confirm that docker-cli.sh's sourcing of $0.script behaves correctly in every fake-CLI scenario, e.g. when the module passes flags before the script path, but found nothing in the diff showing a failure. (crates/tinybox-module/src/tinybus_module/docker_tests.rs#use tinybox_docker::DockerCli;, crates/tinybox-module/src/tinybus_module/docker_tests.rs#fn no_workspace_request_rejects_an_ambiguous_workspace_path())
  • modified — generated_supervisor_keeps_grace_period_when_group_inspection_fails uses the refined fake ps to log supervisor and child ps calls to separate files, asserts exactly 100 supervisor grace probes occur when inspection fails, asserts the child performs at most one process-group inspection before TERM, and asserts the working directory is removed.: The separation of child calls into their own log makes the grace-period accounting more precise; the tests lane found the assertions (the 101 to 100 probe count and the child-probe bound) do constrain behavior. (crates/tinybox-ssh/src/host/observed_tests.rs#fn generated_supervisor_keeps_grace_period_when_group_inspection_fails() -> Resu)

Findings

No active actionable findings.

Could not review: crates/tinybox-host/src/limited/jailed.rs, crates/tinybox-host/src/limited/jailed_tests.rs, crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs

Before merge

  • Complete the critique review for crates/tinybox-host/src/limited/jailed.rs, crates/tinybox-host/src/limited/jailed_tests.rs, crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs.

How this fits together

flowchart LR
  n0["failure<br/>changed"]:::changed
  n1["terminate<br/>changed"]:::changed
  n2["..._for_or_cancel_an_active_jailed_execution<br/>changed"]:::changed
  n3["...quest_rejects_an_ambiguous_workspace_path<br/>changed"]:::changed
  n4["..._grace_period_when_group_inspection_fails<br/>changed"]:::changed
  n5["Result"]:::impacted
  n6["io"]:::impacted
  n7["output"]:::impacted
  n8["Error"]:::impacted
  n9["...ps_the_owned_group_after_its_wrapper_dies"]:::impacted
  n0 -->|uses| n8
  n1 -->|calls| n0
  n1 -->|uses| n5
  n1 -->|calls| n6
  n2 -->|uses| n5
  n2 -->|uses| n8
  n3 -->|uses| n5
  n3 -->|uses| n8
  n4 -->|uses| n5
  n4 -->|calls| n6
  n4 -->|uses| n6
  n4 -->|tests| n6
  n4 -->|calls| n7
  n4 -->|tests| n7
  n5 -->|uses| n8
  n6 -->|uses| n8
  n7 -->|uses| n5
  n7 -->|calls| n6
  n7 -->|uses| n6
  n7 -->|uses| n8
  n9 -->|uses| n5
  n9 -->|calls| n6
  n9 -->|tests| n6
  n9 -->|calls| n7
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading
Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: incomplete; unanswered: crates/tinybox-host/src/limited/jailed.rs, crates/tinybox-host/src/limited/jailed_tests.rs, crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs
  • Positive: The critique lane reviewed the README addition and confirmed the claim that JailLocalHost's group cleanup reuses the same fresh process-table check holds, with the EPERM arm guarded by !alive(group).await? and a failing probe propagating via ?. It raised no findings, though it could not review the other five files.
  • Lane summary: The change is a two-line documentation addition to the narrow-process README claiming that JailLocalHost's group cleanup reuses the same fresh process-table check as LocalHost. Reading `kill_jail_group` in `crates/tinybox-host/src/limited/jailed.rs`, the claim holds: the EPERM arm is guarded by `!alive(group).await?` (the same `super::process::group_alive` probe used by `kill_group_checked`), a live group falls through to an error, and a failing probe propagates via `?`, so cleanup stays unacknowledged in both cases. No defect found in this file. 5 files could not be reviewed: crates/tinybox-host/src/limited/jailed.rs, crates/tinybox-host/src/limited/jailed_tests.rs, crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs.

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The security lane reviewed 5 files with zero findings; the only unreviewed file was the README prose.
  • Lane summary: Reviewed 5 files; 0 findings. 1 file was not security-reviewed: crates/tinybox-host/src/limited/README.md (prose or tabular data).

tests

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The tests lane found the kill_jail_group helper directly pinned by the added test and the Docker/SSH fixture changes' assertions do constrain behavior; no test asserts on a mock or a value that cannot vary.
  • Lane summary: The new `kill_jail_group` helper in `crates/tinybox-host/src/limited/jailed.rs` is directly pinned by the added test in `jailed_tests.rs`, which exercises all four outcomes of the match (Ok, ESRCH, EPERM-with-dead-group, EPERM-with-live-group, and a failed liveness probe) with asserting cases — the error paths are exercised and the one assertion that cannot fail (`is_err()` on a genuinely failing branch) is fine. The docker fixture and ssh observed-test changes are test-harness adjustments whose assertions (`assert!(status.available)`, the `101` -> `100` probe count, and the child-probe bound) do constrain behaviour. No findings.

commits

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Positive: The commits lane found nothing sensitive in what the pull request commits.
  • Lane summary: Nothing sensitive found in what this pull request commits.

description

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The description lane found no reportable defect but left unsettled lookup points, including whether the alive closure bound (FnMut vs FnOnce) satisfies the call site, whether the symlink API call site in write_cli_fixture compiles, and whether the fixture script is tracked.
  • Lane summary: This turn requests lookups to settle three points before the final verdict: (1) whether `Probe` (the `alive`/`a` parameter) is `FnMut` or `FnOnce`, since the closure passed to `wait_for_exit` is called repeatedly in a loop and `kill_jail_group` would inherit that requirement; (2) whether `unix::fs::symlink` is the correct API name in std (I suspect it is actually `std::os::unix::fs::symlink`, which exists — `std::os::unix::fs` does not export `symlink`; the call site in `write_cli_fixture` may not compile), and whether the Docker launcher script `tests/fixtures/docker-cli.sh` is shipped/tracked under `crates/tinybox-module/tests/`; (3) whether `failure` and the `CallFail` type are defined as expected. These determine whether the new `kill_jail_group` helper and the Docker fixture actually compile and run, and whether the PR description's claim about
Evidence and run details
  • Models: ladder/vectors-oai3, deepseek-v4.1-flash
  • Spend: $0.002764
  • Tokens: 427009 input · 37223 output · 96640 cached · 477 embedding
Head State Pass summary
f1800ffcd5cf incomplete 0 active finding(s), 0 resolved finding(s) (at 2026-10-12T20:04:44Z)
f1800ffcd5cf incomplete 0 active finding(s), 0 resolved finding(s) (at 2026-10-12T21:17:11Z)
f1800ffcd5cf incomplete 0 active finding(s), 0 resolved finding(s) (at 2026-10-12T21:24:09Z)

tinysweeper 0.1.2

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking, but could not review everything, so this is not an approval: tinysweeper/description.

             $0.0098 · 621,481 in / 117,523 out · 84,736 cached (14%) · ladder/vectors-oai3, deepseek-v4.1-flash, glm-5.3-flash · 477 embedded
critique:    $0.0033 · 288,470 in / 33,761 out  · 23,552 cached (8%)  · deepseek-v4.1-flash
security:    $0.0051 · 247,851 in / 66,410 out  · 56,192 cached (23%) · deepseek-v4.1-flash
tests:       $0.0009 · 60,782 in  / 10,346 out  · 2,944 cached (5%)   · deepseek-v4.1-flash
description: $0.0004 · 19,553 in  / 4,889 out   · 2,048 cached (10%)  · deepseek-v4.1-flash

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Oct 12, 2026

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking, but could not review everything, so this is not an approval: crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs.

             $0.0050 · 509,146 in / 69,298 out · 151,424 cached (30%) · ladder/vectors-oai3, deepseek-v4.1-flash,  · 477 embedded
critique:    $0.0028 · 214,007 in / 40,170 out · 71,680 cached (33%)  · deepseek-v4.1-flash,
security:    $0.0018 · 210,897 in / 25,469 out · 64,896 cached (31%)  · deepseek-v4.1-flash
tests:       $0.0000 · 38,435 in  / 573 out    · 4,480 cached (12%)   · deepseek-v4.1-flash
description: $0.0001 · 33,636 in  / 880 out    · 10,368 cached (31%)  · deepseek-v4.1-flash

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking, but could not review everything, so this is not an approval: crates/tinybox-host/src/limited/jailed.rs, crates/tinybox-host/src/limited/jailed_tests.rs, crates/tinybox-module/src/tinybus_module/docker_tests.rs, crates/tinybox-module/tests/fixtures/docker-cli.sh, crates/tinybox-ssh/src/host/observed_tests.rs.

             $0.0028 · 427,009 in / 37,223 out · 96,640 cached (23%) · ladder/vectors-oai3, deepseek-v4.1-flash · 477 embedded
critique:    $0.0019 · 163,950 in / 27,392 out · 54,528 cached (33%) · deepseek-v4.1-flash
security:    $0.0005 · 184,713 in / 6,468 out  · 33,920 cached (18%) · deepseek-v4.1-flash
tests:       $0.0000 · 36,537 in  / 546 out    · 3,968 cached (11%)  · deepseek-v4.1-flash
description: $0.0000 · 21,822 in  / 566 out    · 4,224 cached (19%)  · deepseek-v4.1-flash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant