Skip to content

test(e2e): exercise model override startup path - #11003

Merged
rsliter merged 2 commits into
mainfrom
fix/model-override-startup-e2e
Sep 4, 2026
Merged

test(e2e): exercise model override startup path#11003
rsliter merged 2 commits into
mainfrom
fix/model-override-startup-e2e

Conversation

@ericksoa

@ericksoa ericksoa commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Outcome

The image-level model-override test now exercises the real container entrypoint and proves the complete startup transition: JSON rewrite, refreshed hash, sandbox ownership, and mutable modes.

Reason

PR #11001 repaired the deterministic current-main normalize_mutable_config_perms: command not found harness failure with local stubs. Its exact-head Delivery and Verification Advisors correctly found that those stubs bypassed production permission normalization, owner dispatch, and hash refresh. #11001 was merged externally before this correction could be published.

Related issues

Follow-up to #11001.

Changes

  • Remove the brittle extraction of apply_model_override and all three local helper stubs.
  • Run the built image through its real root entrypoint with NEMOCLAW_MODEL_OVERRIDE.
  • Require the one-shot command to observe the rewritten primary and model entries.
  • Require .config-hash to validate and the mutable directory, config, and hash to retain their sandbox ownership and expected modes.
  • Fail the inner command immediately if any JSON, hash, ownership, or mode check fails, before the success marker can be emitted.

Verification

  • shfmt — passed.
  • shellcheck — passed.
  • Codebase growth guardrail and repository hooks — passed.
  • git diff --check origin/main...HEAD — passed.
  • Exact-head managed image E2E on test(e2e): reduce Hermes Discord assertions #11001 proved the original 44/45 failure was resolved by the predecessor fixture update; this stronger real-entrypoint form requires its own CI run before merge.
  • Diff inspection — no secrets, API keys, or credentials.

Review notes

This is the already-prepared correction for the exact-head Advisor findings on #11001. It replaces the merged stubs rather than layering another test path. The first #11003 Advisor pass found that the new inner shell needed fail-fast behavior; the current head adds set -e before every required check.


Signed-off-by: Aaron Erickson aerickson@nvidia.com

Summary by CodeRabbit

  • Tests
    • Expanded end-to-end coverage for the model-override scenario.
    • Added validation that configuration changes are correctly applied and remain consistent.
    • Added checks for configuration integrity and expected file ownership and permission settings.
    • Updated test execution to stop immediately when a validation step fails.

@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5c46e0a9-a8e1-4699-98ec-6c04b6c994d1

📥 Commits

Reviewing files that changed from the base of the PR and between 968a039 and 9adb50f.

📒 Files selected for processing (1)
  • test/e2e-gateway-isolation.sh

Included review availability: Your plan provides up to 12 included reviews per hour; 5 remain after this review.


📝 Walkthrough

Walkthrough

The model-override end-to-end test now uses the image entrypoint with fail-fast behavior. It validates the overridden model data, configuration hash, and ownership and permission modes for mutable configuration paths.

Changes

Gateway override validation

Layer / File(s) Summary
Entrypoint override and configuration checks
test/e2e-gateway-isolation.sh
Test 27 runs through the image entrypoint, enables fail-fast behavior, and validates the model override, config hash, ownership, and mutable permission modes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 9adb5

This updates the end-to-end model-override test to exercise the normal container entrypoint and validate configuration integrity and permissions. No current merge-blocking risk remains.

Suggested reviewers: aasthajh

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: the E2E test now exercises the model-override startup path.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1 files.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/model-override-startup-e2e

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall line coverage in commit 9adb50f in the fix/model-override-s... branch remains at 96%, unchanged from commit 3d75441 in the main branch.


Updated September 03, 2026 21:26 UTC

@wscurran wscurran added area: e2e End-to-end tests, nightly failures, or validation infrastructure area: local-models Local model providers, downloads, launch, or connectivity area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery chore Build, CI, dependency, or tooling maintenance labels Sep 4, 2026

@rsliter rsliter left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 9adb50f.

I reviewed the complete one-file diff and the production entrypoint path. Test 27 now exercises the real root entrypoint and owner dispatch, fails closed before emitting its success marker, and validates the rewritten JSON, refreshed config hash, and sandbox ownership and modes. The exact-head gateway-isolation E2E passed, as did static checks, ShellCheck, CodeQL, DCO, and signed-commit verification. CodeRabbit reported no actionable findings. The completed Advisor migration and documentation reviews found no issues. Security review found no new credential, dependency, authorization, input, or configuration exposure; this strengthens coverage of the existing privilege and integrity boundary.

The failed statuses are infrastructure or aggregate fallout, not candidate-owned failures. Four Advisor specialists report transport errors with TCP connection refused, then produce no review artifacts. cli-test-shards (8) was canceled during Vitest with only "The operation was canceled" and no test failure. cli-tests and checks fail only because that shard did not complete. The same base commit, 3d75441, passed CI / Main Branch, including shard 8, cli-tests, and checks. The changed shell E2E is not part of that CLI shard, and its exact-head gateway-isolation job passed.

@rsliter
rsliter merged commit 3e4f701 into main Sep 4, 2026
90 of 102 checks passed
@rsliter
rsliter deleted the fix/model-override-startup-e2e branch September 4, 2026 16:26
@github-actions github-actions Bot added the v0.0.120 Release target label Sep 4, 2026
rsliter added a commit that referenced this pull request Sep 4, 2026
<!-- markdownlint-disable MD041 -->
## Outcome

Sandbox exec actions now call a transport-neutral streamed-command
interface. The CLI implementation owns OpenShell arguments and process
details while user-visible command behavior remains unchanged.

## Reason

#9804 requires NemoClaw command, session, and SSH consumers to stop
depending on OpenShell CLI details. This first capability slice
separates streamed execution without mixing buffered capture, SSH
credentials, or interactive terminal lifecycle into one review.

### Related issues

Part of #9804
Refs #10991
Refs #10992
Refs #10994

## Changes

- Add a typed streamed-command contract for the current exec consumer.
The new CLI adapter tests protect this transport-neutral boundary.
- Add a CLI implementation that owns executable resolution, gateway
arguments, working directory, TTY and standard-input selection, timeout
flags, process errors, exit status, and signal handling.
- Route `execSandbox` through the typed executor while preserving input
validation, policy guidance, OpenClaw permission cleanup, Google Chat
activation, and exit behavior.
- Remove the superseded buffered completion and raw workdir-probe action
APIs. Keep only the compatibility helpers required by #10991 and #10994,
with linked retirement criteria.
- Move standard-input behavior and tests from the obsolete action
wrapper to the OpenShell adapter.
- Ratchet the OpenShell runtime fan-in budget from 54 to 53 and the
sandbox action root-file budget from 179 to 178.

## Verification

- Candidate: signed DCO commit `f58d784c4c` integrates canonical `main`
commit `58732cfd84`, which contains the #11003 correction.
- Contributor validation: `npm run validate:pr` passed at `f58d784c4c`,
including pre-commit, commit-message, and pre-push checks.
- Focused CLI tests: eight files passed all 126 tests for the executor,
standard-input policy, gateway targeting, multiline arguments,
permission cleanup, Google Chat restart behavior, and launch
compatibility.
- Type safety and build: `npm --prefix nemoclaw run build` and `npm run
typecheck:cli` passed after installing the subproject's locked
dependencies.
- Repository guards: `npm run checks:repository` passed.
- Integration boundary: three cases in
`test/onboarding/onboard-installer-restore-intent.test.ts` passed. The
restored-volume case reached its unchanged Docker inspection boundary
and could not run because this host has no `/var/run/docker.sock`.
- Ready-state E2E: [run
33898752052](https://github.com/NVIDIA/NemoClaw/actions/runs/33898752052)
passed on `f58d784c4c`, including x64 and arm64 image builds plus
sandbox lifecycle, gateway isolation, port override, and non-root smoke
lanes.
- GitHub verification: GitHub marks all four published commits as
verified.
- Remote CI: the exact-head pull-request, code-scanning, and
repository-governance checks passed for `f58d784c4c`.
- Secrets review: the diff contains no secrets, API keys, or
credentials.

## Review notes

- All nine PR Review Advisor specialists found no issue requiring a
change at `f58d784c4c`.
- CodeRabbit reviewed `f58d784c4c` and found no actionable issue. All
three review threads are resolved.
- CodeRabbit's generic docstring-coverage warning does not require a
change: NemoClaw does not require function-level docstrings, and adding
them to touched existing test and support functions would create
non-functional noise outside the accepted refactor.
- `npm run review:local` produced no report for the integrated candidate
because its temporary OpenShell gateway refused connections. The
sandboxed attempt also could not reach the npm registry.
- #11003 merged, is integrated here, and the exact-head
gateway-isolation lane passed.
- The validator entry points, hook configuration, package manifests,
lockfile, and resolved validation surface match canonical `main` at
`58732cfd84`.

---
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Improvements**
- Improved sandbox command execution with more reliable streaming,
exit-status handling, cleanup, and signal management.
- Added clearer handling for unavailable sandboxes, invalid names,
timeouts, directory-probe failures, and command invocation errors.
- Preserved gateway targeting across workdir checks and command
execution.
- Improved standard-input behavior for interactive terminal sessions
while avoiding unintended input inheritance in non-interactive contexts.

- **Documentation**
- Updated sandbox execution guidance to reflect current command
behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Co-authored-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: e2e End-to-end tests, nightly failures, or validation infrastructure area: local-models Local model providers, downloads, launch, or connectivity area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery chore Build, CI, dependency, or tooling maintenance v0.0.120 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants