fix(sandbox): name the quarantined gateway relaunch and its repair - #7869
Conversation
An unsupported edit of a protected Hermes configuration file makes the in-sandbox supervisor refuse every gateway start and then quarantine relaunch. The managed controller only observes that no replacement appeared, so `gateway restart` reported the generic `health timeout` layer, `recover` printed nothing but "check /tmp/gateway.log", and the one forwarded supervisor line blamed MCP integrity. None of the three surfaces named the state that blocks recovery or the supported repair, so the sandbox looked unrecoverable without administrator assistance. Classify the supervisor's quarantine lines as a dedicated `relaunch quarantined` failure layer ahead of the health-timeout and MCP-drift branches they masquerade as, and give that layer plus the existing `config hash mismatch` layer a shared repair block naming `rebuild --yes` and the supported config commands. Report the classified layer to the quiet probe path behind `recover` through an `onRecoveryFailureLayer` callback, so it explains why recovery is not retryable without changing the recovery result shape, and treat the new layer as terminal for Hermes MCP mutation. Fixes #7801 Signed-off-by: Yanyun Liao <yanyunl@nvidia.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughThe gateway restart flow classifies supervisor relaunch quarantine failures, provides rebuild-based integrity repair guidance, propagates failure layers through recovery and connect flows, blocks Hermes MCP mutations for terminal integrity failures, and documents the new behavior. ChangesGateway integrity repair
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant connectSandbox
participant checkAndRecoverSandboxProcesses
participant printGatewayIntegrityRepairGuidance
connectSandbox->>checkAndRecoverSandboxProcesses: register recovery failure callback
checkAndRecoverSandboxProcesses-->>connectSandbox: report relaunch quarantined
connectSandbox->>printGatewayIntegrityRepairGuidance: print rebuild guidance
Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 1f0b924 in the TypeScript / code-coverage/cliThe overall coverage in commit 1f0b924 in the Show a code coverage summary of the most impacted files.
Updated |
|
🌿 Preview your docs: https://nvidia-preview-pr-7869.docs.buildwithfern.com/nemoclaw |
PR Review Advisor — No blocking findings reportedAdvisor assessment: No blocking advisor findings reported Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
1 warning · 0 suggestionsWarningsWarnings do not block.
|
cjagwani
left a comment
There was a problem hiding this comment.
Exact-head maintainer security review for 1f0b9248f424efa7242f5f09142feb8d8589844c: PASS.
- Secrets and credentials: PASS — no credential material or logging changes.
- Input validation: PASS — classification matches fixed, allowlisted supervisor quarantine phrases and preserves sanitization.
- Authentication and authorization: PASS — no trust rule is relaxed; quarantined Hermes MCP mutation now fails closed.
- Dependencies: PASS — no dependency or supply-chain change.
- Error handling and logging: PASS — the terminal state is classified ahead of its generic timeout/MCP symptoms and emits supported repair guidance.
- Cryptography and data protection: PASS — no cryptographic or protected-data path changes.
- Configuration and defaults: PASS — commands and exit behavior are unchanged; guidance points to the existing transactional rebuild path and preserves shell quoting.
- Security testing: PASS with a non-blocking depth warning — marker coverage, precedence, retained legacy layers, remediation, empty-detail output, and probe-only behavior are covered. A direct process-recovery callback test would improve locality, but the production callback is a straight handoff from the tested classifier.
- System security: PASS — the change remains diagnostic and makes the terminal integrity boundary stricter for MCP mutation.
The deterministic gate reports allPass: true: 54 current checks green, current base, one GitHub-Verified commit, and no unresolved major/critical finding. No waiver is required.
|
Current-base handoff after main advanced to da1b103: exact head 1f0b924 remains approved with a PASS security review, but maintainer edits are disabled and the deterministic gate needs current-base evidence. Please refresh it; I will re-gate the next head after one hour of quiet. This is a plain status comment, not Changes Requested. |
|
Correction to my prior handoff: conflict-free base refreshes are explicitly waived. Please do not merge main solely for base currency; preserving exact-head CI/E2E and documentation receipts is preferred unless GitHub reports a real conflict or reviewed behavior requires a change. The existing approval/review evidence remains in force, and I will continue monitoring mergeability. This is a plain coordination comment, not Changes Requested. |
<!-- markdownlint-disable MD041 --> ## Summary Adds the canonical July 30 release entry for `v0.0.99` before the release tag is captured. The entry covers all 37 merged PRs since `v0.0.98` and bounds experimental or dormant work without presenting it as supported behavior. ## Changes - Adds `docs/changelog/2026-07-30.mdx` with the exact `## v0.0.99` heading, parser-safe MDX SPDX comment, summary, detailed release bullets, and published documentation routes. - Records user-visible recovery, snapshot, shared-route, Hermes, readiness, inference, image, documentation, and release E2E changes. - States that the managed-image selection and startup-profile contracts remain dormant and do not activate buildless onboarding. Source summary: - [#7972](#7972) -> `docs/changelog/2026-07-30.mdx`: Records restored managed OpenClaw configuration modes during recovery. - [#7834](#7834) -> `docs/changelog/2026-07-30.mdx`: Records clone-bound pairing verification after snapshot restore. - [#7975](#7975) -> `docs/changelog/2026-07-30.mdx`: Records managed startup recovery coverage. - [#7960](#7960) -> `docs/changelog/2026-07-30.mdx`: Records dormant startup-profile coordination without activating a supported surface. - [#7856](#7856) -> `docs/changelog/2026-07-30.mdx`: Records persistence of the credential-free OpenClaw startup command. - [#7959](#7959) -> `docs/changelog/2026-07-30.mdx`: Records dormant startup-profile construction without changing onboarding. - [#7946](#7946) -> `docs/changelog/2026-07-30.mdx`: Records the internal startup-profile schema and transport contract. - [#7951](#7951) -> `docs/changelog/2026-07-30.mdx`: Records platform-pull cleanup before managed-image validation. - [#7949](#7949) -> `docs/changelog/2026-07-30.mdx`: Records rejection of retained Hermes `uv` build cache metadata. - [#7597](#7597) -> `docs/changelog/2026-07-30.mdx`: Records separate command and agent first-turn latency evidence. - [#7931](#7931) -> `docs/changelog/2026-07-30.mdx`: Records focused E2E replacement evidence for retired selectors. - [#7950](#7950) -> `docs/changelog/2026-07-30.mdx`: Records exclusion of build-only BuildKit telemetry from the Deep Agents Code probe. - [#7665](#7665) -> `docs/changelog/2026-07-30.mdx`: Records consolidated priority 2 E2E coverage. - [#7911](#7911) -> `docs/changelog/2026-07-30.mdx`: Records the corrected NVIDIA DORI installation pin. - [#7934](#7934) -> `docs/changelog/2026-07-30.mdx`: Records the staging image-family wait before Brev Launchable deployment. - [#7772](#7772) -> `docs/changelog/2026-07-30.mdx`: Records dormant managed-image selection contracts without activating buildless onboarding. - [#7941](#7941) -> `docs/changelog/2026-07-30.mdx`: Records corrected agent-specific provider and policy guidance. - [#7819](#7819) -> `docs/changelog/2026-07-30.mdx`: Records removal of empty Deep Agents Code provider-switch sections. - [#7932](#7932) -> `docs/changelog/2026-07-30.mdx`: Records independent credential-generation E2E execution. - [#7840](#7840) -> `docs/changelog/2026-07-30.mdx`: Records shared-route preservation and pre-delete peer validation during upgrades. - [#7874](#7874) -> `docs/changelog/2026-07-30.mdx`: Records the split between pre-tag release entries and post-tag Announcements. - [#7876](#7876) -> `docs/changelog/2026-07-30.mdx`: Records the writable Hermes runtime root within lockdown. - [#7756](#7756) -> `docs/changelog/2026-07-30.mdx`: Records validated multi-platform managed-image publication. - [#7914](#7914) -> `docs/changelog/2026-07-30.mdx`: Records accepted `uv` version metadata in Hermes image validation. - [#7686](#7686) -> `docs/changelog/2026-07-30.mdx`: Records the explicitly experimental Microsoft Entra runtime identity reference. - [#7869](#7869) -> `docs/changelog/2026-07-30.mdx`: Records classified gateway relaunch quarantine and rebuild guidance. - [#7814](#7814) -> `docs/changelog/2026-07-30.mdx`: Records state restore into replacement sandboxes and SQLite write verification. - [#7839](#7839) -> `docs/changelog/2026-07-30.mdx`: Records quieter onboarding test execution without a user-facing behavior claim. - [#7854](#7854) -> `docs/changelog/2026-07-30.mdx`: Records generalized agent-selection guidance. - [#7845](#7845) -> `docs/changelog/2026-07-30.mdx`: Records isolated CDI test evidence without a user-facing behavior claim. - [#7843](#7843) -> `docs/changelog/2026-07-30.mdx`: Records the corrected Omni sub-agent model ID. - [#7908](#7908) -> `docs/changelog/2026-07-30.mdx`: Records reviewed Hermes and Deep Agents Code dependency pins. - [#7887](#7887) -> `docs/changelog/2026-07-30.mdx`: Records rejection of a symlinked DGX Station release marker. - [#7747](#7747) -> `docs/changelog/2026-07-30.mdx`: Records the internal compute-driver separation without a user-facing behavior claim. - [#7660](#7660) -> `docs/changelog/2026-07-30.mdx`: Records atomic publication of rebuild recovery manifests. - [#7661](#7661) -> `docs/changelog/2026-07-30.mdx`: Records bounded local inference health-response retention. - [#7654](#7654) -> `docs/changelog/2026-07-30.mdx`: Records state preservation across supervisor relaunch recovery. ## Type of Change - [ ] Code change (feature, bug fix, or refactor) - [ ] Code change with doc updates - [x] Doc only (prose changes, no code sample modifications) - [ ] Doc only (includes code sample changes) ## Quality Gates - [ ] Tests added or updated for changed behavior - [x] Existing tests cover changed behavior — justification: `test/changelog-docs.test.ts` validates the dated changelog contract, SPDX comment, version heading, and published routes. - [ ] Tests not applicable — justification: - [x] Docs updated for user-facing behavior changes - [ ] Docs not applicable — justification: - [ ] Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging) - [ ] Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: - [ ] Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue: ## Documentation Writer Review - [x] Documentation writer subagent reviewed the completed changes - Result: `docs-updated` - Evidence: `docs/changelog/2026-07-30.mdx`; the documentation-only diff passed review against `WRITING.md`, the controlled word list, and `docs/CONTRIBUTING.md`. The review covered terminology, structure, active voice, release meaning, product-scope boundaries, and link and code presentation. Changelog tests passed 6/6, and the docs build reported 0 errors with 2 pre-existing warnings. - Agent: Codex CLI <!-- docs-review-head-sha: 200940f --> <!-- docs-review-agents-blob-sha: c052d60 --> ## DGX Station Hardware Evidence - [ ] Tested on DGX Station - Tested commit: - Station profile/scenario: - Result: - Supporting evidence: ## Verification - [x] PR description includes a `Signed-off-by:` line and every commit appears as `Verified` in GitHub - [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or `npm run validate:pr` passed after refreshing `origin/main` when hooks were skipped or unavailable - [x] Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: `npx vitest run test/changelog-docs.test.ts` passed 6/6 tests. - [ ] Applicable broad gate passed — `npm test` for broad runtime/test-harness changes; `npm run check` for repo-wide validation/coverage changes — command/result: Not applicable to this documentation-only release entry. - [x] Quality Gates section completed with required justifications or waivers - [x] No secrets, API keys, or credentials committed - [ ] `npm run docs` builds without warnings (doc changes only) — result: Build passed with 0 errors and 2 pre-existing warnings. - [x] Doc pages follow the [style guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md) (doc changes only) - [ ] New doc pages include SPDX header and frontmatter (new pages only) --- Signed-off-by: San Dang <sdang@nvidia.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added release notes for v0.0.99 covering snapshot restoration, sandbox recovery, gateway route upgrades, and Hermes security updates. * Documented experimental Microsoft Entra runtime identity support and enhanced readiness checks. * Added details on managed image validation, trusted CI image promotion, and end-to-end release evidence. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Summary
An unsupported edit of a protected Hermes configuration file leaves the sandbox
in a state the CLI never names: the in-sandbox supervisor refuses every gateway
start and quarantines relaunch, but
gateway restartreports the generichealth timeoutlayer andrecoverprints only "check /tmp/gateway.log". ThisPR classifies the quarantine as its own failure layer and makes all three
restart/recovery surfaces print the supported repair command.
Closes #7801.
Reproduction
Run on our Ubuntu 24.04 x86_64 test host (no GPU), against a Hermes sandbox
onboarded from
mainin this run:Environment
mainHEADeeab81cc5542902538c97db63c132c0fdbd4341cat repro time; this branch is rebased onto376beb50bollama-local, modelllama3.1:8benforced); this repro is the OpenShell-managed topology, where the same
in-sandbox refusal arrives through the non-root startup guard. Both end in
the same quarantined supervisor.
Observed on
main(before fix)nemoclaw <name> gateway restart— exit 1:nemoclaw <name> recover— exit 1:The gateway did not time out — it was refused and the supervisor stopped
relaunching. The only surviving signal is a raw forwarded log line that
attributes the refusal to MCP integrity, and neither command names a repair.
Observed on
fix/...(after fix)nemoclaw <name> gateway restart— exit 1:nemoclaw <name> recover— exit 1:The advertised repair was then executed end to end on the same sandbox to
confirm it is not just plausible advice:
Analysis
classifyGatewayRestartFailureinsrc/lib/actions/sandbox/gateway-restart.tsmatched
GATEWAY_HEALTH_TIMEOUTand stopped there. That marker is what themanaged controller emits whenever no replacement gateway appears within the
await-replacement stage, including when the supervisor deliberately stopped
launching one. In the OpenShell-managed topology,
prepare_hermes_nonroot_runtimein
agents/hermes/start.shreaches the drifted config throughinspect_hermes_mcp_integrity, so the refusal is reported as MCP drift, andrecover_hermes_gateway_current_userthen callsquarantine_hermes_managed_gateway_relaunch. The quarantine lines areallowlisted for forwarding by
scripts/managed-gateway-control.py, so the hostalready receives the decisive evidence — it just never classified it.
Two consequences followed.
printGatewayRestartFailureprinted the layer plusraw detail with no remediation (only the
MCP reconciliation refusallayer hadany), and
printHostManagedGatewayRecoveryHintsinprocess-recovery.tsfellinto its generic branch, which tells the operator to retry
nemoclaw <name> gateway restart— a retry that re-reads the same drifted fileand cannot succeed. On the
recoverpath the situation was worse: managedrecovery runs with
quiet: true, sorunSandboxConnectProbeinconnect.tsdiscarded the classified layer entirely and fell through to the generic
"check /tmp/gateway.log" wedge message.
Fix
relaunch quarantinedfailure layer, matched on the four quarantinephrases the Hermes supervisor emits. It is classified before the MCP-drift
and health-timeout branches because a quarantine is the strictly more specific
and terminal fact: those two layers are how the quarantine surfaces, not what
it is. Output without a quarantine line keeps its previous layer.
gatewayIntegrityRepairLines()is the single source of the repair text,shared by the new layer and the pre-existing
config hash mismatchlayer.Both are deterministic refusals of the same protected-configuration contract,
and
rebuild --yesis the documented command that restores the registeredconfiguration, refreshes the integrity hashes, and returns the gateway in one
transaction.
printGatewayRestartFailureemits it. The remediation block moved outside theempty-detail early return, so a controller result with no detail — exactly the
case where the operator has nothing else — still gets the repair. This also
makes the existing MCP remediation reachable on empty detail.
printHostManagedGatewayRecoveryHintsreturns early for both layers insteadof advising a retry that cannot succeed.
checkAndRecoverSandboxProcessesnow returnsrecoveryFailureLayeron its twoterminal failure paths, and
printGatewayIntegrityRepairGuidance(added nextto the sibling
exitOnSecretBoundaryRefusal/exitOnMcpReconciliationRefusalhelpers) lets the quiet probe path behind
recoverreport it. It returnsfalsefor retryable layers, so the#4710wedge diagnostics stay in chargeof everything else.
mcp-bridge-adapter-hermes.tscounts the new layer as a terminal integrityfailure, so an MCP mutation against a quarantined sandbox still fails closed
rather than falling through to the retry path.
No classification is weakened and no refusal is relaxed: the managed controller
still declines to treat a mutable compatibility hash as a trust anchor, which
is the intentional behavior documented in
docs/manage-sandboxes/gateway-lifecycle-control.mdx. The change is diagnosticonly — the same commands still fail with the same exit codes.
Whole-class review of the
GatewayRestartFailureLayerconsumersgateway-restart.tsprintGatewayRestartFailureprocess-recovery.tsprintHostManagedGatewayRecoveryHintsconnect.tsrunSandboxConnectProbeterminal branch (recover,connect --probe-only)mcp-bridge-adapter-hermes.tsterminalIntegrityFailureinference-set-gateway-restart.tsstatus-preflight.ts/status-snapshot.tsSandboxStatusFailureLayerunion with its own classifier; never sees restart outputadapters/openshell/restore-gateway-pairing.tsRestoreGatewayPairingFailureLayeruniondoctor[info] Serving process: not checked), so gateway-process health indoctoris a separate feature rather than a regression introduced hereTests added (
gateway-restart-quarantine-repair.test.ts) pin: everyquarantine line the supervisor can emit; the verbatim controller output captured
above classifying as a quarantine rather than a health timeout; quarantine
winning over a co-occurring MCP-drift marker; the regression lock that
health-timeout, MCP-drift, config-hash and supervisor-not-running output without
a quarantine line keep their existing layers; the repair text naming
rebuild --yesfor both integrity layers; the repair surviving an emptycontroller detail; retryable layers still getting no rebuild instruction; the
MCP remediation still emitted; and a contract check that the matched marker
substrings still exist verbatim in
agents/hermes/start.shand its forwardingallowlist in
scripts/managed-gateway-control.py.Changes
src/lib/actions/sandbox/gateway-restart.ts: newrelaunch quarantinedlayer, quarantine markers, shared repair lines, repair emitted outside the empty-detail guardsrc/lib/actions/sandbox/process-recovery.ts: repair branch in the recovery hints;recoveryFailureLayerreturned from the terminal failure pathssrc/lib/actions/sandbox/connect-boundary-refusal.ts:printGatewayIntegrityRepairGuidancenext to the sibling refusal helperssrc/lib/actions/sandbox/connect.ts:recover/ probe path reports the repair instead of the generic gateway-log messagesrc/lib/actions/sandbox/mcp-bridge-adapter-hermes.ts: new layer treated as a terminal integrity failuresrc/lib/actions/sandbox/gateway-restart-quarantine-repair.test.ts: new regression testsdocs/reference/commands.mdx: failure-layer list updated with the new layerdocs/reference/troubleshooting.mdx: newrelaunch quarantinedsection with the repairType of Change
Verification
npx prek run --all-filespassesnpm testpasses (touched files at minimum)make docsbuilds without warnings (doc changes only)AI Disclosure
Signed-off-by: Yanyun Liao yanyunl@nvidia.com
Summary by CodeRabbit
Bug Fixes
Documentation
relaunch quarantinedfailures.