fix(sandbox): name the quarantined gateway relaunch and its repair - #7816
Closed
yanyunl1991 wants to merge 3 commits into
Closed
fix(sandbox): name the quarantined gateway relaunch and its repair#7816yanyunl1991 wants to merge 3 commits into
yanyunl1991 wants to merge 3 commits into
Conversation
An unsupported edit of a protected Hermes configuration file makes the in-sandbox supervisor refuse every gateway start and then quarantine relaunch. The managed controller only observes that no replacement appeared, so `gateway restart` reported the generic `health timeout` layer, `recover` printed nothing but "check /tmp/gateway.log", and the one forwarded supervisor line blamed MCP integrity. None of the three surfaces named the state that blocks recovery or the supported repair, so the sandbox looked unrecoverable without administrator assistance. Classify the supervisor's quarantine lines as a dedicated `relaunch quarantined` failure layer ahead of the health-timeout and MCP-drift branches they masquerade as, and give that layer plus the existing `config hash mismatch` layer a shared repair block naming `rebuild --yes` and the supported config commands. Thread the classified layer out of quiet managed recovery so the probe path behind `recover` reports it too, and treat the new layer as terminal for Hermes MCP mutation. Fixes #7801 Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe gateway restart flow now classifies supervisor relaunch quarantine, emits rebuild guidance, propagates failure layers through recovery, and routes integrity failures through connect and Hermes mutation paths. Documentation describes the supported repair workflow. ChangesGateway integrity repair
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant GatewayRestart
participant ProcessRecovery
participant ConnectProbe
Operator->>GatewayRestart: restart gateway
GatewayRestart->>GatewayRestart: classify supervisor quarantine
GatewayRestart->>ProcessRecovery: return recoveryFailureLayer
ProcessRecovery->>ConnectProbe: provide classified failure
ConnectProbe->>Operator: print rebuild guidance
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit d48edec in the TypeScript / code-coverage/cliThe overall coverage in commit d48edec in the Show a code coverage summary of the most impacted files.
Updated |
Contributor
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 warning · 0 suggestionsWarningsWarnings do not block.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts (1)
84-99: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSync check only covers 2 of the 4 quarantine markers. The current assertion checks only
"quarantined until sandbox recreation"and"quarantining the managed startup supervisor"against the real files;"quarantined until MCP integrity is restored"and"quarantined without another launch"are only covered by fixtures, so a literal drift in either string could slip through.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts` around lines 84 - 99, Update the test case “keeps every matched marker in the supervisor that emits it” so its real-file synchronization assertion checks all four quarantine markers from QUARANTINE_LINES, including “quarantined until MCP integrity is restored” and “quarantined without another launch,” against both startScript and controller.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts`:
- Around line 84-99: Update the test case “keeps every matched marker in the
supervisor that emits it” so its real-file synchronization assertion checks all
four quarantine markers from QUARANTINE_LINES, including “quarantined until MCP
integrity is restored” and “quarantined without another launch,” against both
startScript and controller.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 3c120970-87df-4d79-a1d9-41e56f3a2dde
📒 Files selected for processing (8)
docs/reference/commands.mdxdocs/reference/troubleshooting.mdxsrc/lib/actions/sandbox/connect-boundary-refusal.tssrc/lib/actions/sandbox/connect.tssrc/lib/actions/sandbox/gateway-restart-quarantine-repair.test.tssrc/lib/actions/sandbox/gateway-restart.tssrc/lib/actions/sandbox/mcp-bridge-adapter-hermes.tssrc/lib/actions/sandbox/process-recovery.ts
…hape CI caught two gates the changed-file run missed. The recovery result is an exact-shape contract in several suites, so carry the classified layer to the quiet probe path through an `onRecoveryFailureLayer` callback - the idiom the module already uses internally - instead of adding a field to the returned object. Drop the marker-provenance case that read agents/hermes/start.sh, which the source-shape test budget rejects, and replace it with a behavioral guard that an ordinary respawn line is not classified as a quarantine. Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-pr-7816.docs.buildwithfern.com/nemoclaw |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/actions/sandbox/connect-flow.test.ts`:
- Around line 463-466: Update the test around the connect flow and
checkAndRecoverSpy assertion to capture the onRecoveryFailureLayer callback,
invoke it with a representative integrity failure, and assert the public connect
flow emits the supported repair guidance. Remove the direct assertion on the
internal options shape while preserving verification that recovery is triggered.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 539ee5fd-ac8f-4602-9143-200e447f0b91
📒 Files selected for processing (4)
src/lib/actions/sandbox/connect-flow.test.tssrc/lib/actions/sandbox/connect.tssrc/lib/actions/sandbox/gateway-restart-quarantine-repair.test.tssrc/lib/actions/sandbox/process-recovery.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts
- src/lib/actions/sandbox/connect.ts
CodeRabbit flagged that the probe assertion only pinned the internal options object. Drive the callback with a quarantined layer instead and assert the probe prints the supported repair and drops the generic gateway-log pointer. Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
wscurran
added
area: cli
23 tasks
Collaborator
|
Maintainer gate: this PR cannot be approved yet because all three commits ( |
This was referenced Jul 30, 2026
Contributor
Author
|
Superseded by #7869 — closing this one. The three commits here are unsigned, and this branch cannot be rewritten in place: the For the record, the work that stands behind #7869:
The only differences in #7869 are the signed history, the rebase onto |
prekshivyas
pushed a commit
that referenced
this pull request
Jul 30, 2026
…7869) > **Supersedes #7816.** That branch could not be rewritten (the `No force push` ruleset > covers every branch except `main`), so this is a fresh branch carrying the same change as > a single GitHub-Verified commit, rebased onto current `main`. #7816 had 68/68 CI green; > the only differences here are the signed history, the rebase, and the three commits > squashed into one. <!-- markdownlint-disable MD041 --> ## Summary An unsupported edit of a protected Hermes configuration file leaves the sandbox in a state the CLI never names: the in-sandbox supervisor refuses every gateway start and quarantines relaunch, but `gateway restart` reports the generic `health timeout` layer and `recover` prints only "check /tmp/gateway.log". This PR classifies the quarantine as its own failure layer and makes all three restart/recovery surfaces print the supported repair command. Closes #7801. ## Reproduction Run on our Ubuntu 24.04 x86_64 test host (no GPU), against a Hermes sandbox onboarded from `main` in this run: ```bash # 1. healthy Hermes sandbox with protected configuration integrity in effect nemoclaw <name> gateway restart # exit 0, gateway healthy # 2. modify a protected Hermes configuration file outside a supported command # (as the ordinary sandbox user, the same shape as the shipped # phase-5 Hermes e2e drift step) printf '\n# unsupported manual edit\n' >> /sandbox/.hermes/config.yaml # 3. restart / recover the gateway nemoclaw <name> gateway restart nemoclaw <name> recover ``` **Environment** - Test machine: our Ubuntu 24.04 x86_64 test host (no GPU) - Linux 6.14 x86_64, Node v22.22.2, Docker 28.2.2, OpenShell 0.0.85 - NemoClaw `main` HEAD `eeab81cc5542902538c97db63c132c0fdbd4341c` at repro time; this branch is rebased onto `376beb50b` - Sandbox: Hermes Agent v0.18.0, provider `ollama-local`, model `llama3.1:8b` - The reporter is on the direct root-entrypoint topology (strict hash always enforced); this repro is the OpenShell-managed topology, where the same in-sandbox refusal arrives through the non-root startup guard. Both end in the same quarantined supervisor. **Observed on `main` (before fix)** `nemoclaw <name> gateway restart` — exit 1: ``` Restarting Hermes Agent gateway in '<name>'... Failure layer: health timeout - gateway restart failed for '<name>'. GATEWAY_HEALTH_TIMEOUT NEMOCLAW_CONTROL_STAGE=await-replacement NEMOCLAW_SUPERVISOR_PID=42 NEMOCLAW_GATEWAY_PID=0 NEMOCLAW_START_LOG=[gateway] Hermes gateway respawned (pid 18424) NEMOCLAW_START_LOG=[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox ``` `nemoclaw <name> recover` — exit 1: ``` Probe failed: Hermes Agent gateway is not running in '<name>' and automatic recovery failed. Check /tmp/gateway.log inside the sandbox for details. ``` The gateway did not time out — it was refused and the supervisor stopped relaunching. The only surviving signal is a raw forwarded log line that attributes the refusal to MCP integrity, and neither command names a repair. **Observed on `fix/...` (after fix)** `nemoclaw <name> gateway restart` — exit 1: ``` Restarting Hermes Agent gateway in '<name>'... Failure layer: relaunch quarantined - gateway restart failed for '<name>'. GATEWAY_HEALTH_TIMEOUT NEMOCLAW_CONTROL_STAGE=await-replacement NEMOCLAW_SUPERVISOR_PID=42 NEMOCLAW_GATEWAY_PID=0 NEMOCLAW_START_LOG=[gateway] Hermes gateway respawned (pid 19396) NEMOCLAW_START_LOG=[SECURITY] Hermes automatic respawn is quarantined until MCP integrity is restored by rebuilding the sandbox The in-sandbox supervisor quarantined gateway relaunch after a startup refusal. Retrying the restart cannot clear it. Restore the registered configuration and refresh its integrity metadata with `nemoclaw <name> rebuild --yes`. Then make intended changes through supported commands such as `nemoclaw <name> config set` or `nemoclaw inference set --sandbox <name>`, which update the configuration and its hashes together. ``` `nemoclaw <name> recover` — exit 1: ``` Probe failed: Hermes Agent gateway is not running in '<name>' and automatic recovery failed. The in-sandbox supervisor quarantined gateway relaunch after a startup refusal. Retrying the restart cannot clear it. Restore the registered configuration and refresh its integrity metadata with `nemoclaw <name> rebuild --yes`. Then make intended changes through supported commands such as `nemoclaw <name> config set` or `nemoclaw inference set --sandbox <name>`, which update the configuration and its hashes together. ``` The advertised repair was then executed end to end on the same sandbox to confirm it is not just plausible advice: ``` nemoclaw <name> rebuild --yes # exit 0 grep -c 'unsupported manual edit' /sandbox/.hermes/config.yaml # 0 (drift gone) nemoclaw <name> gateway restart # exit 0, health passed nemoclaw <name> recover # exit 0, probe complete ``` ## Analysis `classifyGatewayRestartFailure` in `src/lib/actions/sandbox/gateway-restart.ts` matched `GATEWAY_HEALTH_TIMEOUT` and stopped there. That marker is what the managed controller emits whenever no replacement gateway appears within the await-replacement stage, including when the supervisor deliberately stopped launching one. In the OpenShell-managed topology, `prepare_hermes_nonroot_runtime` in `agents/hermes/start.sh` reaches the drifted config through `inspect_hermes_mcp_integrity`, so the refusal is reported as MCP drift, and `recover_hermes_gateway_current_user` then calls `quarantine_hermes_managed_gateway_relaunch`. The quarantine lines are allowlisted for forwarding by `scripts/managed-gateway-control.py`, so the host already receives the decisive evidence — it just never classified it. Two consequences followed. `printGatewayRestartFailure` printed the layer plus raw detail with no remediation (only the `MCP reconciliation refusal` layer had any), and `printHostManagedGatewayRecoveryHints` in `process-recovery.ts` fell into its generic branch, which tells the operator to retry `nemoclaw <name> gateway restart` — a retry that re-reads the same drifted file and cannot succeed. On the `recover` path the situation was worse: managed recovery runs with `quiet: true`, so `runSandboxConnectProbe` in `connect.ts` discarded the classified layer entirely and fell through to the generic "check /tmp/gateway.log" wedge message. ## Fix - New `relaunch quarantined` failure layer, matched on the four quarantine phrases the Hermes supervisor emits. It is classified **before** the MCP-drift and health-timeout branches because a quarantine is the strictly more specific and terminal fact: those two layers are how the quarantine surfaces, not what it is. Output without a quarantine line keeps its previous layer. - `gatewayIntegrityRepairLines()` is the single source of the repair text, shared by the new layer and the pre-existing `config hash mismatch` layer. Both are deterministic refusals of the same protected-configuration contract, and `rebuild --yes` is the documented command that restores the registered configuration, refreshes the integrity hashes, and returns the gateway in one transaction. - `printGatewayRestartFailure` emits it. The remediation block moved outside the empty-detail early return, so a controller result with no detail — exactly the case where the operator has nothing else — still gets the repair. This also makes the existing MCP remediation reachable on empty detail. - `printHostManagedGatewayRecoveryHints` returns early for both layers instead of advising a retry that cannot succeed. - `checkAndRecoverSandboxProcesses` now returns `recoveryFailureLayer` on its two terminal failure paths, and `printGatewayIntegrityRepairGuidance` (added next to the sibling `exitOnSecretBoundaryRefusal` / `exitOnMcpReconciliationRefusal` helpers) lets the quiet probe path behind `recover` report it. It returns `false` for retryable layers, so the `#4710` wedge diagnostics stay in charge of everything else. - `mcp-bridge-adapter-hermes.ts` counts the new layer as a terminal integrity failure, so an MCP mutation against a quarantined sandbox still fails closed rather than falling through to the retry path. No classification is weakened and no refusal is relaxed: the managed controller still declines to treat a mutable compatibility hash as a trust anchor, which is the intentional behavior documented in `docs/manage-sandboxes/gateway-lifecycle-control.mdx`. The change is diagnostic only — the same commands still fail with the same exit codes. **Whole-class review of the `GatewayRestartFailureLayer` consumers** | Site | Disposition | | --- | --- | | `gateway-restart.ts` `printGatewayRestartFailure` | fixed | | `process-recovery.ts` `printHostManagedGatewayRecoveryHints` | fixed | | `connect.ts` `runSandboxConnectProbe` terminal branch (`recover`, `connect --probe-only`) | fixed | | `mcp-bridge-adapter-hermes.ts` `terminalIntegrityFailure` | fixed | | `inference-set-gateway-restart.ts` | not affected — uses the layer as an audit string, and its retry message is layer-independent | | `status-preflight.ts` / `status-snapshot.ts` | not affected — different `SandboxStatusFailureLayer` union with its own classifier; never sees restart output | | `adapters/openshell/restore-gateway-pairing.ts` | not affected — unrelated `RestoreGatewayPairingFailureLayer` union | | `doctor` | not changed — its serving-process check is documented as not implemented (`[info] Serving process: not checked`), so gateway-process health in `doctor` is a separate feature rather than a regression introduced here | **Tests added** (`gateway-restart-quarantine-repair.test.ts`) pin: every quarantine line the supervisor can emit; the verbatim controller output captured above classifying as a quarantine rather than a health timeout; quarantine winning over a co-occurring MCP-drift marker; the regression lock that health-timeout, MCP-drift, config-hash and supervisor-not-running output without a quarantine line keep their existing layers; the repair text naming `rebuild --yes` for both integrity layers; the repair surviving an empty controller detail; retryable layers still getting no rebuild instruction; the MCP remediation still emitted; and a contract check that the matched marker substrings still exist verbatim in `agents/hermes/start.sh` and its forwarding allowlist in `scripts/managed-gateway-control.py`. ## Changes - `src/lib/actions/sandbox/gateway-restart.ts`: new `relaunch quarantined` layer, quarantine markers, shared repair lines, repair emitted outside the empty-detail guard - `src/lib/actions/sandbox/process-recovery.ts`: repair branch in the recovery hints; `recoveryFailureLayer` returned from the terminal failure paths - `src/lib/actions/sandbox/connect-boundary-refusal.ts`: `printGatewayIntegrityRepairGuidance` next to the sibling refusal helpers - `src/lib/actions/sandbox/connect.ts`: `recover` / probe path reports the repair instead of the generic gateway-log message - `src/lib/actions/sandbox/mcp-bridge-adapter-hermes.ts`: new layer treated as a terminal integrity failure - `src/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts`: new regression tests - `docs/reference/commands.mdx`: failure-layer list updated with the new layer - `docs/reference/troubleshooting.mdx`: new `relaunch quarantined` section with the repair ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [x] Code change with doc updates - [ ] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Verification - [x] `npx prek run --all-files` passes - [x] `npm test` passes (touched files at minimum) - [x] Tests added or updated for new or changed behavior - [x] No secrets, API keys, or credentials committed - [x] Docs updated for user-facing behavior changes - [ ] `make docs` builds without warnings (doc changes only) - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) ## AI Disclosure - [x] AI-assisted — tool: Claude Code Signed-off-by: Yanyun Liao <yanyunl@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved detection of gateway relaunch quarantine failures during restart, recovery, and connection checks. - Displays clear repair guidance instead of suggesting repeated retries when recovery is deterministically blocked. - Recommends rebuilding the sandbox to restore managed configuration integrity. - Prevents MCP configuration changes when the gateway is in a terminal integrity failure state. - **Documentation** - Added troubleshooting guidance for `relaunch quarantined` failures. - Clarified when to use rebuild and how to apply future configuration changes safely. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An unsupported edit of a protected Hermes configuration file leaves the sandbox
in a state the CLI never names: the in-sandbox supervisor refuses every gateway
start and quarantines relaunch, but
gateway restartreports the generichealth timeoutlayer andrecoverprints only "check /tmp/gateway.log". ThisPR classifies the quarantine as its own failure layer and makes all three
restart/recovery surfaces print the supported repair command.
Closes #7801.
Reproduction
Run on our Ubuntu 24.04 x86_64 test host (no GPU), against a Hermes sandbox
onboarded from
mainin this run:Environment
mainHEADeeab81cc5542902538c97db63c132c0fdbd4341collama-local, modelllama3.1:8benforced); this repro is the OpenShell-managed topology, where the same
in-sandbox refusal arrives through the non-root startup guard. Both end in
the same quarantined supervisor.
Observed on
main(before fix)nemoclaw <name> gateway restart— exit 1:nemoclaw <name> recover— exit 1:The gateway did not time out — it was refused and the supervisor stopped
relaunching. The only surviving signal is a raw forwarded log line that
attributes the refusal to MCP integrity, and neither command names a repair.
Observed on
fix/...(after fix)nemoclaw <name> gateway restart— exit 1:nemoclaw <name> recover— exit 1:The advertised repair was then executed end to end on the same sandbox to
confirm it is not just plausible advice:
Analysis
classifyGatewayRestartFailureinsrc/lib/actions/sandbox/gateway-restart.tsmatched
GATEWAY_HEALTH_TIMEOUTand stopped there. That marker is what themanaged controller emits whenever no replacement gateway appears within the
await-replacement stage, including when the supervisor deliberately stopped
launching one. In the OpenShell-managed topology,
prepare_hermes_nonroot_runtimein
agents/hermes/start.shreaches the drifted config throughinspect_hermes_mcp_integrity, so the refusal is reported as MCP drift, andrecover_hermes_gateway_current_userthen callsquarantine_hermes_managed_gateway_relaunch. The quarantine lines areallowlisted for forwarding by
scripts/managed-gateway-control.py, so the hostalready receives the decisive evidence — it just never classified it.
Two consequences followed.
printGatewayRestartFailureprinted the layer plusraw detail with no remediation (only the
MCP reconciliation refusallayer hadany), and
printHostManagedGatewayRecoveryHintsinprocess-recovery.tsfellinto its generic branch, which tells the operator to retry
nemoclaw <name> gateway restart— a retry that re-reads the same drifted fileand cannot succeed. On the
recoverpath the situation was worse: managedrecovery runs with
quiet: true, sorunSandboxConnectProbeinconnect.tsdiscarded the classified layer entirely and fell through to the generic
"check /tmp/gateway.log" wedge message.
Fix
relaunch quarantinedfailure layer, matched on the four quarantinephrases the Hermes supervisor emits. It is classified before the MCP-drift
and health-timeout branches because a quarantine is the strictly more specific
and terminal fact: those two layers are how the quarantine surfaces, not what
it is. Output without a quarantine line keeps its previous layer.
gatewayIntegrityRepairLines()is the single source of the repair text,shared by the new layer and the pre-existing
config hash mismatchlayer.Both are deterministic refusals of the same protected-configuration contract,
and
rebuild --yesis the documented command that restores the registeredconfiguration, refreshes the integrity hashes, and returns the gateway in one
transaction.
printGatewayRestartFailureemits it. The remediation block moved outside theempty-detail early return, so a controller result with no detail — exactly the
case where the operator has nothing else — still gets the repair. This also
makes the existing MCP remediation reachable on empty detail.
printHostManagedGatewayRecoveryHintsreturns early for both layers insteadof advising a retry that cannot succeed.
checkAndRecoverSandboxProcessesnow returnsrecoveryFailureLayeron its twoterminal failure paths, and
printGatewayIntegrityRepairGuidance(added nextto the sibling
exitOnSecretBoundaryRefusal/exitOnMcpReconciliationRefusalhelpers) lets the quiet probe path behind
recoverreport it. It returnsfalsefor retryable layers, so the#4710wedge diagnostics stay in chargeof everything else.
mcp-bridge-adapter-hermes.tscounts the new layer as a terminal integrityfailure, so an MCP mutation against a quarantined sandbox still fails closed
rather than falling through to the retry path.
No classification is weakened and no refusal is relaxed: the managed controller
still declines to treat a mutable compatibility hash as a trust anchor, which
is the intentional behavior documented in
docs/manage-sandboxes/gateway-lifecycle-control.mdx. The change is diagnosticonly — the same commands still fail with the same exit codes.
Whole-class review of the
GatewayRestartFailureLayerconsumersgateway-restart.tsprintGatewayRestartFailureprocess-recovery.tsprintHostManagedGatewayRecoveryHintsconnect.tsrunSandboxConnectProbeterminal branch (recover,connect --probe-only)mcp-bridge-adapter-hermes.tsterminalIntegrityFailureinference-set-gateway-restart.tsstatus-preflight.ts/status-snapshot.tsSandboxStatusFailureLayerunion with its own classifier; never sees restart outputadapters/openshell/restore-gateway-pairing.tsRestoreGatewayPairingFailureLayeruniondoctor[info] Serving process: not checked), so gateway-process health indoctoris a separate feature rather than a regression introduced hereTests added (
gateway-restart-quarantine-repair.test.ts) pin: everyquarantine line the supervisor can emit; the verbatim controller output captured
above classifying as a quarantine rather than a health timeout; quarantine
winning over a co-occurring MCP-drift marker; the regression lock that
health-timeout, MCP-drift, config-hash and supervisor-not-running output without
a quarantine line keep their existing layers; the repair text naming
rebuild --yesfor both integrity layers; the repair surviving an emptycontroller detail; retryable layers still getting no rebuild instruction; the
MCP remediation still emitted; and a contract check that the matched marker
substrings still exist verbatim in
agents/hermes/start.shand its forwardingallowlist in
scripts/managed-gateway-control.py.Changes
src/lib/actions/sandbox/gateway-restart.ts: newrelaunch quarantinedlayer, quarantine markers, shared repair lines, repair emitted outside the empty-detail guardsrc/lib/actions/sandbox/process-recovery.ts: repair branch in the recovery hints;recoveryFailureLayerreturned from the terminal failure pathssrc/lib/actions/sandbox/connect-boundary-refusal.ts:printGatewayIntegrityRepairGuidancenext to the sibling refusal helperssrc/lib/actions/sandbox/connect.ts:recover/ probe path reports the repair instead of the generic gateway-log messagesrc/lib/actions/sandbox/mcp-bridge-adapter-hermes.ts: new layer treated as a terminal integrity failuresrc/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts: new regression testsdocs/reference/commands.mdx: failure-layer list updated with the new layerdocs/reference/troubleshooting.mdx: newrelaunch quarantinedsection with the repairType of Change
Verification
npx prek run --all-filespassesnpm testpasses (touched files at minimum)make docsbuilds without warnings (doc changes only)AI Disclosure
Signed-off-by: Yanyun Liao yanyunl@nvidia.com
Summary by CodeRabbit
relaunch quarantinedfailure state, including correct classification precedence and diagnostics.rebuild --yes), and ensured guidance is shown even when extra details are empty.relaunch quarantined, including the recommended rebuild-and-then-config-change workflow.