Skip to content

fix(sandbox): wait for managed gateway lease - #8262

Merged
prekshivyas merged 32 commits into
NVIDIA:mainfrom
harjothkhara:codex/fix-7429-hermes-policy-remove
Aug 6, 2026
Merged

fix(sandbox): wait for managed gateway lease#8262
prekshivyas merged 32 commits into
NVIDIA:mainfrom
harjothkhara:codex/fix-7429-hermes-policy-remove

Conversation

@harjothkhara

@harjothkhara harjothkhara commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

For recover and gateway restart, the managed gateway controller now acquires the expected-exit lock before it inspects the supervisor or gateway. Lock acquisition, gateway termination, and replacement health share one recovery deadline. If lock acquisition reaches that deadline, the controller returns SUPERVISOR_BUSY without publishing an expected-exit marker.

This serializes consecutive Hermes policy-remove calls that reach the controller while gateway recovery is active.

Related Issue

Refs #7429

PR #7498 addressed the separate Hermes version probe for upgrade-sandboxes --check. This PR addresses the remaining consecutive policy-remove behavior and does not close the issue.

Changes

  • Acquire the expected-exit lock before supervisor and gateway inspection, then inspect the gateway again after contention ends.
  • Use one recovery deadline for lock acquisition, preflight, marker publication, gateway termination, and replacement health.
  • Remove the marker when publication reaches the deadline.
  • Re-prove gateway identity before pidfd signaling, and send no signal after the deadline.
  • Apply the remaining deadline to loopback HTTP connection, request, status, header, and body reads.
  • Preserve root-owned lock checks, process identity checks, inode-safe cleanup, and marker authorization.
  • Add regression coverage for contention, publication cleanup, identity recapture, signal deadlines, and slow HTTP responses.
  • Update the OpenClaw restart-respawn harness for the shared recovery deadline.
  • Document the serialized gateway lifecycle and deadline behavior.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Current-revision security revalidation passed with no findings: fix(sandbox): wait for managed gateway lease #8262 (comment)
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: No waiver. Required GitHub Actions checks must pass.

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: Independent documentation review on the current revision verified the shared recovery deadline, lock/termination/replacement semantics, SUPERVISOR_BUSY behavior, marker cleanup, and existing retry guidance with no findings. The follow-up commit changes only test-harness isolation and clock wiring. The docs build passed with 0 errors and 2 existing warnings.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable; scripts/prepare-dgx-station-host.sh is unchanged.
  • Station profile/scenario: Not applicable.
  • Result: Not applicable.
  • Supporting evidence: Not applicable.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub — all 32 commits appear as Verified.
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable — normal commit and push hooks passed for the current follow-up.
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — 4 focused files and 10 tests pass. The CLI build and type check, Python compilation, documentation build, commit hooks, and push hooks also pass.
  • Applicable broad gate passed — fresh required GitHub Actions checks are running for the current branch revision.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed — the independent security review passed all nine categories.
  • npm run docs builds without warnings (doc changes only) — no local docs build ran; GitHub Actions is authoritative.
  • Doc pages follow the style guide (doc changes only) — the independent documentation writer review passed.
  • New doc pages include SPDX header and frontmatter (new pages only) — not applicable; no new page.

Sensitive-Path Security Review

Independent review found no security finding for PR commit 8b52f32ac37b4dc3e2832d8948c206ce9d0db1c1 against base SHA db31c286129e878c3356eed49f76ab259561e47e.

  1. Secrets and credentials: PASS. Deadline propagation does not widen credential access, and Hermes validation remains fail closed.
  2. Input validation and data sanitization: PASS. Request actions and nonces remain validated, and loopback health probes use bounded parsing.
  3. Authentication and authorization: PASS. Root-owned lock validation precedes process inspection, and marker publication remains bound to gateway and controller identities.
  4. Dependencies and third-party libraries: PASS. No dependency or artifact changes.
  5. Error handling and logging: PASS. Failed marker publication removes only the matching inode, and lifecycle cleanup runs from the outer finally block.
  6. Cryptography and data protection: PASS. No cryptographic mechanism or protected-data flow changes.
  7. Configuration and security headers: PASS. Runtime files retain owner, type, link-count, and mode checks.
  8. Security testing: PASS. Regression sources cover contention, deadline expiry, identity refresh, inode-safe cleanup, signal admission, and slow HTTP response phases.
  9. System security: PASS. Recovery holds the lock from discovery through cleanup, re-proves the pidfd target before signaling, and applies one deadline to all recovery phases.

The documentation records the existing shared-UID provenance and mutable-configuration TOCTOU limitations. Required GitHub Actions checks remain separate merge requirements.

GitHub Actions

GitHub Actions is authoritative for PR commit 8b52f32ac37b4dc3e2832d8948c206ce9d0db1c1 against base SHA db31c286129e878c3356eed49f76ab259561e47e.
Required checks are running.
No check has a maintainer waiver.


Signed-off-by: harjoth harjoth.khara@gmail.com

Summary by CodeRabbit

  • Bug Fixes
    • Improved managed gateway lifecycle handling during concurrent expected-exit requests.
    • Added consistent recovery time limits across expected-exit locking, gateway termination, and replacement health checks.
    • Prevented stale or conflicting exit markers when the recovery deadline expires.
    • Improved gateway identity refresh and safe restart behavior.
    • Requests now correctly return “supervisor busy” when recovery cannot complete in time.
  • Documentation
    • Clarified expected lifecycle, locking, and timeout behavior during recovery.
  • Tests
    • Added integration coverage for lifecycle locking, deadline enforcement, and marker safety.

Wait for an active expected-exit lease before reporting SUPERVISOR_BUSY.

Refs NVIDIA#7429.

Signed-off-by: harjoth <harjoth.khara@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The managed gateway controller acquires the expected-exit lock before discovery and uses one recovery deadline for lock acquisition, termination, and replacement health checks. Tests cover contention, cleanup, restart, inode safety, permissions, and deadline behavior.

Changes

Managed gateway recovery control

Layer / File(s) Summary
Expected-exit lock lifecycle
scripts/managed-gateway-control.py
The controller polls non-blocking lock acquisition until the recovery deadline. Lease publication reuses the acquired lock, and cleanup closes the lease or untransferred lock.
Shared recovery deadline
scripts/managed-gateway-control.py
The controller passes the remaining recovery time through candidate waits, gateway termination, and replacement health checks. Deadline expiry returns SUPERVISOR_BUSY during lock acquisition or GATEWAY_FAILED during termination.
Lifecycle behavior validation
docs/manage-sandboxes/gateway-lifecycle-control.mdx, test/managed-gateway-control-locking.test.ts, test/managed-gateway-control.test.ts
Documentation and tests cover serialized leases, timeout handling, inode-safe markers, permissions, controller restarts, gateway identity refresh, termination deadlines, and updated mock timing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ManagedController
  participant ExpectedExitLock
  participant Gateway
  participant ReplacementHealth
  ManagedController->>ExpectedExitLock: Acquire lock until recovery deadline
  ExpectedExitLock-->>ManagedController: Return lock or SUPERVISOR_BUSY
  ManagedController->>Gateway: Terminate within remaining deadline
  Gateway-->>ManagedController: Termination result
  ManagedController->>ReplacementHealth: Verify replacement within remaining deadline
  ReplacementHealth-->>ManagedController: Health result
Loading

Possibly related PRs

Suggested reviewers: brandonpelfrey

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: the managed gateway now waits for the expected-exit lease before recovery operations.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/managed-gateway-control.py`:
- Around line 476-484: Update the retry loop around the nonblocking fcntl.flock
call so every retry remains within lock_deadline: check the deadline before
retrying, compute the remaining duration, and cap time.sleep(POLL_SECONDS) to
that duration. Preserve the existing SUPERVISOR_BUSY ControlError behavior while
ensuring no flock attempt or sleep occurs after the recovery timeout.

In `@test/managed-gateway-control.test.ts`:
- Around line 740-744: Update the timeout test around
control._publish_expected_exit_lease to verify bounded waiting, not just the
ControlError code: inject a fake monotonic clock and sleep hook or measure
elapsed time at the controller boundary, then assert polling advances through
RECOVERY_TIMEOUT_SECONDS before returning SUPERVISOR_BUSY. Preserve the existing
error-code assertion while proving the configured deadline was reached.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: abe37f77-8764-42b9-84fd-9cc6a07fed67

📥 Commits

Reviewing files that changed from the base of the PR and between 0d1cb93 and 800605f.

📒 Files selected for processing (3)
  • docs/manage-sandboxes/gateway-lifecycle-control.mdx
  • scripts/managed-gateway-control.py
  • test/managed-gateway-control.test.ts

Comment thread scripts/managed-gateway-control.py Outdated
Comment thread test/managed-gateway-control.test.ts Outdated
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 3 warnings · 1 suggestion
  • Model comparison: normalized findings differ; normalized terminology decisions differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 3 more warnings, 1 more suggestion.
3 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

  • pidfd at scripts/managed-gateway-control.py:1788: selected only by the second-opinion lane as established.
  • expected-exit marker at docs/manage-sandboxes/gateway-lifecycle-control.mdx:26: selected only by the second-opinion lane as justified.
  • acquire-expected-exit-lock at scripts/managed-gateway-control.py:101: selected only by the second-opinion lane as established.
5 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

  • gateway-guard-recovery: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • hermes-e2e: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • rebuild-hermes: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • sandbox-operations: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • openclaw-discord-pairing: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • established — recovery deadline at docs/manage-sandboxes/gateway-lifecycle-control.mdx:25: Keep `recovery deadline`; it matches established repository usage and the stated controller behavior.
  • justified — expected-exit lock at docs/manage-sandboxes/gateway-lifecycle-control.mdx:24: Keep `expected-exit lock`; the modifier distinguishes the lock from the expected-exit marker and documents its concrete role.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: managed-image-multiarch-startup

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Keep expected-exit lock polling within the recovery deadline.

Signed-off-by: harjoth <harjoth.khara@gmail.com>
@wscurran wscurran added area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior platform: linux Affects non-Ubuntu Linux environments labels Aug 4, 2026
@wscurran

wscurran commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@apurvvkumaria apurvvkumaria self-assigned this Aug 5, 2026
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Security review for the current branch revision c29c022bc: PASS — no findings.

Category Result Evidence
Secrets and credentials PASS The three-file contributor diff adds no credential material or logging.
Input validation and data sanitization PASS The change does not add untrusted input parsing.
Authentication and authorization PASS Waiting occurs before marker publication and does not authorize a gateway exit.
Dependencies and third-party libraries PASS The contributor diff adds no dependency.
Error handling and logging PASS Timeout returns the existing SUPERVISOR_BUSY code and adds no diagnostic content.
Cryptography and data protection PASS The change adds no cryptographic operation or stored sensitive data.
Configuration and security headers PASS Root ownership, mode, inode, path, and controller-identity checks remain enforced.
Security testing PASS The integration test covers lock release and the configured timeout boundary.
System security PASS The lock remains fail-closed, and each retry and sleep remains within the recovery deadline.

Verification on c29c022bc:

  • Focused integration test: 4 passed.
  • Python compile check: passed.
  • Diff-scoped repository hooks: passed.
  • Pre-push TypeScript and version checks: passed.
  • All commits appear as GitHub Verified.

The branch now includes current main. This incorporates the merged managed-image privilege-transition fix that addresses the earlier Hermes image-build failure.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator

I reviewed the current fork diff and the trusted E2E risk plan. The changes are limited to bounded managed-gateway lease waiting, focused tests, and documentation, and the selected protected multi-architecture startup lane is appropriate. I authorized the trusted controller to run that plan; fork code does not receive repository credentials. I will reconcile the resulting evidence before merge.

cv added 2 commits August 5, 2026 04:44
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator

The authorized protected E2E plan has completed successfully. All 41 current checks pass, including the selected managed-image multi-architecture startup lane, aggregate CI, both advisor lanes, CodeQL, DCO, documentation receipt, and commit verification. There are no unresolved review threads. An independent approval is the only remaining merge gate.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
test/managed-gateway-control-locking.test.ts (2)

104-124: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider asserting a clock bound instead of an exact fake-clock total.

lock_timeout_result records the exact accumulated fake-clock value and line 302 asserts it equals 0.01. This pins the test to the internal retry-sleep schedule of _acquire_expected_exit_lock. If the implementation changes its retry interval or adds one extra sleep before the deadline check, the total overshoots 0.01 and this test fails even though the observable contract (SUPERVISOR_BUSY raised at or after the deadline) is unchanged.

Assert the error code exactly and the clock as a lower bound. This keeps the deadline-enforcement claim and removes the schedule coupling.

♻️ Proposed change to assert a bound
-    lock_timeout_result = [lock_timeout, timeout_clock[0]]
+    lock_timeout_result = [
+        lock_timeout,
+        timeout_clock[0] >= control.RECOVERY_TIMEOUT_SECONDS,
+    ]

Then update the Vitest assertion:

-      lock_timeout: ["SUPERVISOR_BUSY", 0.01],
+      lock_timeout: ["SUPERVISOR_BUSY", true],

Note that control.RECOVERY_TIMEOUT_SECONDS is restored at line 122, so capture the 0.01 value before the finally block if you use this form.

As per path instructions: "Prefer observable outcomes through the public boundary over source-text, private-shape, or mock-call assertions."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/managed-gateway-control-locking.test.ts` around lines 104 - 124, Update
the assertions for lock_timeout_result to require the exact SUPERVISOR_BUSY
error code while asserting the fake clock is at least the configured recovery
timeout, rather than matching an exact total. Capture the 0.01 timeout value
before the finally block restores RECOVERY_TIMEOUT_SECONDS, and use that value
in the lower-bound assertion.

Source: Path instructions


262-266: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Derive the first wait expectation from STOP_GRACE_SECONDS.

The first wait is min(STOP_GRACE_SECONDS, 3.0), so [3, 0] depends on the current value 5.0. Use a deadline above both grace values and advance the fake clock to that deadline, or derive the first expected value from control.STOP_GRACE_SECONDS. Keep the 0 assertion because it proves the shared deadline clamps the kill wait.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/managed-gateway-control-locking.test.ts` around lines 262 - 266, Update
the test around record_deadline_wait so the first wait expectation is
independent of the current STOP_GRACE_SECONDS value: either set the fake
deadline above both grace values and advance deadline_clock to it, or derive the
first expected wait from control.STOP_GRACE_SECONDS. Preserve the 0-second
second-wait assertion to verify the shared deadline clamps the kill wait.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/managed-gateway-control-locking.test.ts`:
- Around line 295-299: Update the spawnSync invocation in the locking harness
test to include an explicit test-scale timeout and killSignal. After spawning,
assert that result.error is not a timeout error before asserting result.status
is 0, so a hung child produces a clear failure while normal harness assertion
failures remain distinguishable.

---

Nitpick comments:
In `@test/managed-gateway-control-locking.test.ts`:
- Around line 104-124: Update the assertions for lock_timeout_result to require
the exact SUPERVISOR_BUSY error code while asserting the fake clock is at least
the configured recovery timeout, rather than matching an exact total. Capture
the 0.01 timeout value before the finally block restores
RECOVERY_TIMEOUT_SECONDS, and use that value in the lower-bound assertion.
- Around line 262-266: Update the test around record_deadline_wait so the first
wait expectation is independent of the current STOP_GRACE_SECONDS value: either
set the fake deadline above both grace values and advance deadline_clock to it,
or derive the first expected wait from control.STOP_GRACE_SECONDS. Preserve the
0-second second-wait assertion to verify the shared deadline clamps the kill
wait.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f7e76603-27a0-48ae-b30a-19d1ebd92403

📥 Commits

Reviewing files that changed from the base of the PR and between c29c022 and 5c5799f.

📒 Files selected for processing (4)
  • docs/manage-sandboxes/gateway-lifecycle-control.mdx
  • scripts/managed-gateway-control.py
  • test/managed-gateway-control-locking.test.ts
  • test/managed-gateway-control.test.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • docs/manage-sandboxes/gateway-lifecycle-control.mdx
  • test/managed-gateway-control.test.ts

Comment thread test/managed-gateway-control-locking.test.ts
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@cjagwani cjagwani added v0.0.104 Release target and removed v0.0.103 labels Aug 5, 2026

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security review — exact head 50eac324c3afd345f697851e23004ab48571e246 against base c176af5f08595ed748b4eeac8988c554e180183e: PASS with no findings.

  1. Secrets and credentials — PASS. No credential material, secret value, credential source, or sensitive logging changes.
  2. Input validation and injection resistance — PASS. No new untrusted parser or command interpolation. Runtime files retain descriptor-relative validation for links, ownership, modes, process identity, and inode replacement.
  3. Authentication and authorization — PASS. Mutating requests acquire the root-owned expected-exit lock before process inspection; marker publication remains bound to gateway and live-controller identity.
  4. Dependencies and supply chain — PASS. No dependency, lockfile, image source, download, or external repository change occurs in this effective patch.
  5. Error handling and information exposure — PASS. Lock timeout returns SUPERVISOR_BUSY, publishes no marker, and closes descriptors. The shared recovery deadline is rechecked before SIGKILL; exhausted operations send no later signal.
  6. Cryptography and data protection — PASS / not applicable. No cryptographic behavior changes. The lock remains mode 0600, the marker mode 0444, and both require the trusted runtime owner and a single link.
  7. Configuration and infrastructure — PASS. Lifecycle mutation is serialized without broadening network policy, ports, users, privileges, capabilities, or supported configuration.
  8. Security testing — PASS. Exact focused validation passes 3 files and 8 tests for contention, deadline enforcement, marker isolation, descriptor cleanup, refreshed process discovery, deadline propagation, and SIGTERM-only exhaustion. npm run check:diff, strict docs validation, and diff hygiene pass.
  9. System security — PASS. Process identity is refreshed after contention, pidfd signaling remains in place, and no signal occurs after the recovery deadline. Base PR #7894's health-streak reset is complementary post-recovery accounting and does not change expected-exit authorization or serialization.

The signed merges were conflict-free and preserve the effective five-file patch byte-for-byte (raw SHA-256 d21036f50ae0af21e9fed2b13167babb0f3c14d4e691b4b0b1accb082b7d3c9d; stable patch ID 84297edbceeeba90c29fb17b920290f599720578). Base PR #8371 changes internal vLLM model acquisition behind a preserved API and does not alter managed-gateway lifecycle state. Product scope remains established by accepted issue #7429. Fresh CI, protected E2E, documentation receipt, and independent post-push approval remain mandatory.

@cjagwani
cjagwani requested review from apurvvkumaria and cv August 5, 2026 18:05
@apurvvkumaria

apurvvkumaria commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

CI disposition for the current branch revision:

  • The Nemotron job in advisor run 31032650495 initially failed because the model omitted a required terminology-tool result and emitted text before that tool completed. The downloaded artifact preserved 0 canonical findings. This was an advisor model/tool-protocol infrastructure failure, not a PR code or test failure.
  • The run was on attempt 1, so I used the single permitted failed-job retry. Attempt 2 completed successfully: both GPT-5.6 and Nemotron report 0 blockers, 0 warnings, and 0 suggestions. The retry allowance for this branch revision is now consumed; no additional retry will be used.
  • The protected managed-image E2E child run passed on both amd64 and arm64. The E2E gate reports that every selected shard passed with no skips or pending tests.

No PR code change is indicated by the advisor failure.

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

Security review — exact head a22e40f493e34e442780551f149df170fc1f0359 against base 3b208d79e5d3bda4183704145ee5c28d79876ae1: PASS with no findings. The effective patch is safe from the reviewed security perspective, subject to every repository merge gate.

Findings Table

No findings.

Detailed Analysis

  1. Secrets and Credentials — PASS. No credential material, secret value, credential source, or sensitive logging changes.

  2. Input Validation and Data Sanitization — PASS. No new untrusted parser or command interpolation. Runtime files retain descriptor-relative validation for links, ownership, modes, process identity, and inode replacement.

  3. Authentication and Authorization — PASS. Mutating requests acquire the root-owned expected-exit lock before process inspection; marker publication remains bound to gateway and live-controller identity.

  4. Dependencies and Third-Party Libraries — PASS. No dependency, lockfile, image source, download, or external repository change occurs in this effective patch.

  5. Error Handling and Logging — PASS. Lock timeout returns SUPERVISOR_BUSY, publishes no marker, and closes descriptors. The shared recovery deadline is rechecked before SIGKILL; exhausted operations send no later signal.

  6. Cryptography and Data Protection — PASS / not applicable. No cryptographic behavior changes. The lock remains mode 0600, the marker mode 0444, and both require the trusted runtime owner and a single link.

  7. Configuration and Security Headers — PASS. Lifecycle mutation is serialized without broadening network policy, ports, users, privileges, capabilities, or supported configuration.

  8. Security Testing — PASS. Exact focused validation passes 3 files and 8 tests for contention, deadline enforcement, marker isolation, descriptor cleanup, refreshed process discovery, deadline propagation, and SIGTERM-only exhaustion. npm run check:diff, strict docs validation, and diff hygiene pass.

  9. System Security — PASS. Process identity is refreshed after contention, pidfd signaling remains in place, and no signal occurs after the recovery deadline. Base PR #7894's health-streak reset is complementary post-recovery accounting and does not change expected-exit authorization or serialization.

Files Reviewed

  • docs/manage-sandboxes/gateway-lifecycle-control.mdx
  • scripts/managed-gateway-control.py
  • test/managed-gateway-control-locking.test.ts
  • test/managed-gateway-control.test.ts
  • test/openclaw-managed-restart-respawn.test.ts

Provenance and Required Gates

The signed refresh merge is conflict-free and preserves the effective 5-file patch byte-for-byte (raw SHA-256 d21036f50ae0af21e9fed2b13167babb0f3c14d4e691b4b0b1accb082b7d3c9d; stable patch ID 84297edbceeeba90c29fb17b920290f599720578). Base PR #8372 changes the canonical security rubric and PR Review Advisor machinery only; it does not alter any effective PR file or runtime trust boundary. This review independently applies the new rubric.

Fresh CI, protected E2E, documentation receipt, unresolved-review resolution, and independent exact-head approval remain mandatory. This security review waives none of those gates.

@apurvvkumaria

apurvvkumaria commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

CI disposition after the latest branch refresh:\n\n- The newest refresh commit is signed off and GitHub-verified, and the fresh DCO check passed. It merges the current base into the branch. The PR patch retains the same stable patch identity, with no change to the five PR-owned files.\n- Advisor run 31038118964 was on attempt 1. GPT-5.6 completed the review with no blockers, warnings, or suggestions, but its final same-session synthesis was incomplete. Nemotron received an external provider 429 response before completing analysis.\n- I used the single permitted failed-job retry. Attempt 2 completed successfully: Nemotron reports no blockers, warnings, or suggestions, and all nine security categories passed. The retry allowance for this branch revision is consumed; no further retry will be used.\n- Fresh documentation review, Fern preview, DCO, commit lint, macOS and WSL E2E, security checks, and managed-image startup checks have passed. The remaining CI rerun and protected E2E gate are still in progress.\n- Protected E2E evidence from the previous branch refresh passed on both architectures, but it is superseded by this refresh and cannot satisfy the current gate.\n\nNo merge is appropriate until every current required gate finishes.

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

Security review — exact head f0ff49aada08a1e17e3d636c9d52e6db1dcebfc6 against base c6713c9dd17a9682654713ccb6829b8714f1b307: PASS with no findings. The effective patch is safe from the reviewed security perspective, subject to every repository merge gate.

Findings Table

No findings.

Detailed Analysis

  1. Secrets and Credentials — PASS. No credential material, secret value, credential source, or sensitive logging changes.

  2. Input Validation and Data Sanitization — PASS. No new untrusted parser or command interpolation. Runtime files retain descriptor-relative validation for links, ownership, modes, process identity, and inode replacement.

  3. Authentication and Authorization — PASS. Mutating requests acquire the root-owned expected-exit lock before process inspection; marker publication remains bound to gateway and live-controller identity.

  4. Dependencies and Third-Party Libraries — PASS. No dependency, lockfile, image source, download, or external repository change occurs in this effective patch.

  5. Error Handling and Logging — PASS. Lock timeout returns SUPERVISOR_BUSY, publishes no marker, and closes descriptors. The shared recovery deadline is rechecked before SIGKILL; exhausted operations send no later signal.

  6. Cryptography and Data Protection — PASS / not applicable. No cryptographic behavior changes. The lock remains mode 0600, the marker mode 0444, and both require the trusted runtime owner and a single link.

  7. Configuration and Security Headers — PASS. Lifecycle mutation is serialized without broadening network policy, ports, users, privileges, capabilities, or supported configuration.

  8. Security Testing — PASS. Exact focused validation passes 3 files and 8 tests for contention, deadline enforcement, marker isolation, descriptor cleanup, refreshed process discovery, deadline propagation, and SIGTERM-only exhaustion. npm run check:diff, strict docs validation, and diff hygiene pass.

  9. System Security — PASS. Process identity is refreshed after contention, pidfd signaling remains in place, and no signal occurs after the recovery deadline. Base PR #7894's health-streak reset is complementary post-recovery accounting and does not change expected-exit authorization or serialization.

Files Reviewed

  • docs/manage-sandboxes/gateway-lifecycle-control.mdx
  • scripts/managed-gateway-control.py
  • test/managed-gateway-control-locking.test.ts
  • test/managed-gateway-control.test.ts
  • test/openclaw-managed-restart-respawn.test.ts

Provenance and Required Gates

The signed refresh merge is conflict-free and preserves the effective 5-file patch byte-for-byte (raw SHA-256 d21036f50ae0af21e9fed2b13167babb0f3c14d4e691b4b0b1accb082b7d3c9d; stable patch ID 84297edbceeeba90c29fb17b920290f599720578). Base PR #8383 adds only the v0.0.103 release-note page and does not alter any effective PR file or reviewed runtime trust boundary. Base PR #8372 changes the canonical security rubric and PR Review Advisor machinery only; it does not alter any effective PR file or runtime trust boundary. This review independently applies the new rubric.

Fresh CI, protected E2E, documentation receipt, unresolved-review resolution, and independent exact-head approval remain mandatory. This security review waives none of those gates.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Current CI is blocked by an unrelated timeout in test/managed-bootstrap-trampoline.test.ts:305: the hostile loader fixture exceeded its 15-second limit after repeated loader warnings. This PR changes gateway lease handling, not the managed bootstrap trampoline or loader fixture, so the failure is not attributable to this change. The bounded transient retry has already been used; I will not rerun it again without new evidence. The earlier E2E result is not being carried forward to this revision. External CI blocker: https://github.com/NVIDIA/NemoClaw/actions/runs/31038121810/job/92421400025

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

Security revalidation — exact head e3f858d526c97de83321b02279d48409464f5835 against current base 1608281462923ff8282151c7be1a109ee262b4e9: PASS with no findings, subject to every repository merge gate.

Findings Table

No findings.

Detailed Analysis

  1. Secrets and Credentials — PASS. The effective PR patch is unchanged from the prior exact review. The base refresh adds no overlapping secret, credential, environment, or sensitive-logging path.

  2. Input Validation and Data Sanitization — PASS. The reviewed validation and command/path construction are unchanged. The new base delta does not alter an effective PR file.

  3. Authentication and Authorization — PASS. No identity, ownership, permission, or authorization boundary changed in the PR patch or its integration with the refreshed base.

  4. Dependencies and Third-Party Libraries — PASS. The PR's reviewed dependency and artifact identities are unchanged. Base PR #8387 adds verified llama.cpp GGUF acquisition in eight disjoint files and does not change this PR's dependency surface.

  5. Error Handling and Logging — PASS. The reviewed fail-closed behavior and diagnostics are unchanged; the base refresh introduces no overlapping error or logging path.

  6. Cryptography and Data Protection — PASS. The PR's reviewed cryptographic and protected-data behavior is unchanged. No new cross-patch data flow exists.

  7. Configuration and Security Headers — PASS. The effective PR patch preserves its reviewed configuration, network, privilege, capability, port, and header boundaries.

  8. Security Testing — PASS. Raw patch SHA-256 remains d21036f50ae0af21e9fed2b13167babb0f3c14d4e691b4b0b1accb082b7d3c9d and stable patch ID remains 84297edbceeeba90c29fb17b920290f599720578. Documentation writer revalidation passed at this exact head; the existing PR documentation remains accurate. Prior targeted test evidence remains applicable; fresh exact-head GitHub Actions is authoritative.

  9. System Security — PASS. The signed refresh is conflict-free, preserves the reviewed patch byte-for-byte, and introduces no new runtime trust-boundary interaction with the disjoint base delta.

Provenance and Required Gates

This exact-head revalidation supplements the full prior nine-category review at #8262 (review). It applies the current rubric from .agents tree 7320c9e331ddbcd9ff83e0fce9387a004e3bd1aa.

Fresh required CI, protected E2E, the exact documentation receipt, unresolved-review resolution, GitHub commit verification, and an independent exact-head approval remain mandatory. This review waives none of those gates.

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

Security review — exact head 651a19cd4ac3a02da4a18e93ac599741590248f2 against current base b148bc13ee53b7a1f612d07e42275ec732e71877: PASS with no findings, subject to every repository merge gate.

Findings Table

No findings.

Detailed Analysis

  1. Secrets and Credentials — PASS. The effective PR patch is unchanged and adds no secret, credential source, environment exposure, or sensitive logging. The refreshed base introduces no overlapping credential path.

  2. Input Validation and Data Sanitization — PASS. Reviewed validation, command construction, and path handling remain unchanged. The portable-profile base delta has no exact file overlap with this PR.

  3. Authentication and Authorization — PASS. No identity, ownership, permission, or authorization boundary is widened by the PR patch or its integration with the refreshed base.

  4. Dependencies and Third-Party Libraries — PASS. The PR's reviewed dependency and artifact identities are unchanged. Base PRs #8333 and #8376 add disjoint E2E artifact restoration and portable experimental onboarding behavior.

  5. Error Handling and Logging — PASS. Reviewed fail-closed behavior and diagnostics remain unchanged; the signed integrations add no overlapping error or logging path.

  6. Cryptography and Data Protection — PASS. Reviewed cryptographic and protected-data behavior remains unchanged, with no new cross-patch data flow.

  7. Configuration and Security Headers — PASS. The effective PR patch preserves its reviewed configuration, network, privilege, capability, port, and header boundaries. The hidden portable profile does not bypass this PR's authority checks.

  8. Security Testing — PASS. Raw patch SHA-256 remains d21036f50ae0af21e9fed2b13167babb0f3c14d4e691b4b0b1accb082b7d3c9d and stable patch ID remains 84297edbceeeba90c29fb17b920290f599720578. The prior exact lifecycle-control locking and deadline evidence remains applicable. Exact-head documentation writer revalidation passed; fresh GitHub Actions is authoritative.

  9. System Security — PASS. Both signed refreshes are conflict-free, preserve the effective patch byte-for-byte, and introduce no new runtime trust-boundary interaction. For #7853 specifically, the portable profile supplies onboarding defaults but neither stages messaging credentials nor bypasses registry-driven credential-drift validation.

Provenance and Required Gates

The exact head preserves contributor history through signed merge commits and applies the current rubric from .agents tree 7320c9e331ddbcd9ff83e0fce9387a004e3bd1aa. This supplements the full prior nine-category review at #8262 (review).

Fresh required CI, protected E2E, the exact documentation receipt, unresolved-review resolution, GitHub commit verification, and an independent exact-head approval remain mandatory. This review waives none of those gates.

cv and others added 4 commits August 5, 2026 14:20
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Current revision d957bcbb9 repairs the recovery-deadline harnesses added by the preceding signed update.

Focused validation exposed four PR-related failures before CI reached them:

  • the new deadline harness retained its forwarding stub instead of restoring the production termination function;
  • two existing harness doubles did not accept the newly forwarded recovery deadline;
  • the HTTP deadline scenario advanced a private fake clock without connecting it to the controller;
  • one recorded result retained a mutable list that was later cleared and reused.

The follow-up is test-only. It restores test isolation, snapshots observations at the assertion boundary, wires the simulated clock to the controller, and keeps the stubs compatible with the public recovery flow. Validation now passes 4 focused files and 10 tests. The CLI build and type check pass, the documentation build reports 0 errors and the 2 existing Fern warnings, and commit/push hooks pass.

Compliance and security disposition:

  • the follow-up is signed off and GitHub-verified; all 32 PR commits are verified and the PR description retains the contributor DCO declaration;
  • all review threads remain resolved;
  • the six-file branch change adds no credential, dependency, network-policy, privilege, cryptographic, or external-input surface;
  • the root-owned lock, marker ownership/mode/inode checks, process-identity proof, pidfd signaling, fail-closed timeout behavior, and security regression coverage remain intact;
  • the existing lifecycle page already documents that lock acquisition, termination, and replacement health share one recovery deadline. The test-only repair adds no new user-facing behavior.

After reviewing the fork scope, I authorized the repository standard CI and protected E2E workflows to run. Fresh CI, advisor, CodeQL, documentation review, and protected E2E results are still required. The merge freeze remains in effect, so no merge or main-branch write will occur.

@apurvvkumaria

Copy link
Copy Markdown
Collaborator

Final validation receipt for branch revision d957bcbb91:

  • The required E2E gate and rollup passed. Trusted controller 31053910556 and selected run 31053939402 completed successfully.
  • Protected managed-image startup passed on both linux/amd64 and linux/arm64. No retry was required.
  • Required CI, both advisor lanes, CodeQL, ShellCheck, DCO, commit lint, documentation review receipt, Fern preview, macOS/WSL E2E, managed-image startup, and CodeRabbit are settled green.
  • All 32 commits are GitHub Verified, the contributor DCO declaration remains in the PR description, no review threads are unresolved, and independent approval remains current.
  • Documentation review is current. The recorded security review remains applicable; the current test-only repair adds no credential, dependency, network-policy, privilege, cryptographic, or external-input surface.

Validation is complete. The merge freeze remains active, so this PR is intentionally not being merged or queued.

@prekshivyas
prekshivyas enabled auto-merge (squash) August 6, 2026 00:57
@prekshivyas
prekshivyas disabled auto-merge August 6, 2026 01:22
@prekshivyas
prekshivyas merged commit 422fe95 into NVIDIA:main Aug 6, 2026
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: cli Command line interface, flags, terminal UX, or output area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior platform: linux Affects non-Ubuntu Linux environments release-target v0.0.104 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants