Skip to content

fix(hermes): trust corporate CA in final-stage downloads - #8228

Merged
cv merged 37 commits into
mainfrom
codex/hermes-final-stage-ca-ordering
Aug 5, 2026
Merged

fix(hermes): trust corporate CA in final-stage downloads#8228
cv merged 37 commits into
mainfrom
codex/hermes-final-stage-ca-ordering

Conversation

@cv

@cv cv commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Hermes custom images that receive a corporate CA now validate and install it in the final build stage before any registry or Python package download. Builds without a corporate CA keep the default uv and Python trust behavior.

Changes

  • Validate the operator-supplied CA after the final FROM instruction and install it as a root-owned, read-only trust anchor.
  • Set NODE_EXTRA_CA_CERTS before final-stage Node.js and npm operations.
  • Set CURL_CA_BUNDLE, SSL_CERT_FILE, and REQUESTS_CA_BUNDLE only when the decoded corporate CA exists.
  • Reject command-order decoys, duplicate reviewed commands, hidden shell substitutions, continuation-spliced commands, and non-shell Unicode whitespace.
  • Select only verified roots for the CI fixture, bound its encoded build argument, and verify the installed bundle bytes.
  • Open an existing output without following symlinks or truncating it. Verify descriptor and path identity before changing content or mode.
  • Require the corporate CA build argument before the bundle guard and image build.
  • Document the Hermes discovery installer, final-stage registry remediation, package-install trust, and default-trust behavior.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent review of the current PR commit, 24dd82d3a90a2affb564382b1f8153951dcd9665, against base commit 89e17bfe9a22ffc6b16c8407ce83a2c0a493ba74 passed all nine security categories with no findings. The merge from current main preserved the 15-file contribution with stable patch ID eee48b893c2158892e7ceb81925edc0067c5828a. The final Hermes stage validates the supplied CA before TLS operations and preserves default trust when it is absent. The CI fixture uses fixed endpoints, hostname and chain verification, bounded CA roots, disabled ambient CA stores, and installed-bundle SHA-256 equality. Output handling opens without following symlinks or truncating, verifies descriptor and path identity, requires a single-link regular file, and writes through the descriptor.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: Codex Desktop independently reviewed the complete 15-file contribution for the current PR commit, 24dd82d3a90a2affb564382b1f8153951dcd9665, against base commit 89e17bfe9a22ffc6b16c8407ce83a2c0a493ba74. The merge from current main preserved the contribution with stable patch ID eee48b893c2158892e7ceb81925edc0067c5828a. docs/security/configure-corporate-ca-trust.mdx accurately documents the Hermes discovery installer, final-stage CA ordering, conditional registry and Python trust, and default-trust behavior. The page is published in the OpenClaw, Hermes, and Deep Agents guide variants. The review found no changed text whose wording alters behavior, security, data safety, or test meaning. GitHub Fern Preview passed for this commit.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit: Not applicable; scripts/prepare-dgx-station-host.sh is unchanged.
  • Station profile/scenario: Not applicable.
  • Result: Not applicable.
  • Supporting evidence: Not applicable.

Verification

  • PR description includes a Signed-off-by: line and every published commit appears as Verified in GitHub — all 37 commits are Verified.
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable — pre-commit and commit-msg passed. The linked worktree lacks generated dist/ artifacts required by pre-push; GitHub Actions is the validation authority.
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — GitHub CLI tests, all eight CLI shards, static checks, build/typecheck, installer integration, managed image checks, WSL, macOS, CodeQL, and both automated advisors passed for commit 24dd82d3a90a2affb564382b1f8153951dcd9665.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Core GitHub checks passed for commit 24dd82d3a90a2affb564382b1f8153951dcd9665; the required E2E gate is still running.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — GitHub Docs / Fern Preview will validate the published commit.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only) — no new documentation pages were added.

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved corporate certificate authority handling during Hermes image builds.
    • Ensured npm registry remediation and package installation work with corporate TLS certificates.
    • Applied certificate trust configuration before dependency installation and npm operations.
    • Preserved default trust behavior when no corporate certificate is configured.
    • Added validation that corporate certificates are correctly included and verified in built images.
  • Documentation

    • Updated corporate CA configuration guidance to reflect the revised trust setup and npm behavior.
  • Tests

    • Expanded coverage for certificate setup order, conditional trust configuration, default trust behavior, and Docker build command validation.

Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Hermes configures corporate CA trust after the base image and before npm operations. Messaging installation uses conditional CA trust variables. Shared Dockerfile parsing helpers, contract tests, documentation, and boundary-build checks validate command structure, ordering, and certificate installation.

Changes

Hermes corporate CA trust

Layer / File(s) Summary
Early CA setup and registry remediation
agents/hermes/Dockerfile, docs/security/configure-corporate-ca-trust.mdx
The Dockerfile decodes and validates the corporate CA, sets NODE_EXTRA_CA_CERTS, conditionally sets CURL_CA_BUNDLE for npm remediation, and applies the CA during messaging installation. Documentation describes the updated order and trust variables.
Dockerfile command validation
test/helpers/dockerfile-run-commands.ts, test/dockerfile-run-commands.test.ts
New helpers parse Dockerfile instructions and validate unique reviewed RUN commands, shell structure, continuations, and required arguments.
Contract tests and source-shape checks
test/*dockerfile-contract.test.ts, test/hermes-final-image-layout.test.ts, test/corporate-ca-build-tls-anchor.test.ts, ci/source-shape-test-budget.json
Contract tests use parsed command validation for npm remediation ordering and npm-root arguments. Hermes tests verify CA setup and trust-variable behavior.
Boundary build CA validation
.github/workflows/sandbox-images-and-e2e.yaml, tools/e2e/sandbox-images-workflow-boundary.mts, test/e2e/support/sandbox-images-workflow-boundary.test.ts
The boundary build encodes the runner CA bundle, passes it to the Hermes build, verifies the installed certificate, and validates default-trust behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#8196: Both changes modify Hermes Dockerfile npm remediation commands and their contract tests.

Suggested labels: integration: hermes, area: security, security, bug-fix, platform: container

Suggested reviewers: apurvvkumaria, brandonpelfrey, dnandakumar-nv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding corporate CA trust for final-stage Hermes downloads.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/hermes-final-stage-ca-ordering

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 24dd82d in the codex/hermes-final-s... branch remains at 96%, unchanged from commit 89e17bf in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit 24dd82d in the codex/hermes-final-s... branch remains at 81%, unchanged from commit 89e17bf in the main branch.

Show a code coverage summary of the most impacted files.
File main 89e17bf codex/hermes-final-s... 24dd82d +/-
src/lib/shields/index.ts 74% 73% -1%
src/lib/sandbox...rce-identity.ts 87% 87% 0%
src/lib/inferen...er-discovery.ts 87% 88% +1%

Updated August 05, 2026 08:14 UTC

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Model comparison: normalized findings differ; normalized terminology decisions differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, the same number of suggestions.
5 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

  • default trust at .github/workflows/sandbox-images-and-e2e.yaml:224: selected only by the second-opinion lane as justified.
  • output identity at test/select-ci-endpoint-ca-roots.test.ts:193: selected only by the second-opinion lane as define.
  • registry remediation at ci/source-shape-test-budget.json:96: selected only by the second-opinion lane as justified.
  • cross-signed at test/select-ci-endpoint-ca-roots.test.ts:335: selected only by the second-opinion lane as justified.
  • compact CA bundle at scripts/checks/select-ci-endpoint-ca-roots.mts:296: selected only by the second-opinion lane as justified.

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — default trust at .github/workflows/sandbox-images-and-e2e.yaml:223: Keep `default trust`; the corporate-CA contrast defines its meaning.
  • replace — compact CA root at test/e2e/support/sandbox-images-workflow-boundary.test.ts:592: Rename the test concept to `compact CA root bundle` or `compact CA roots` to match the multi-root output.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-inference, cloud-onboard, full-e2e, hermes-e2e, hermes-inference-switch, managed-image-multiarch-startup, security-posture

1 optional E2E recommendation
  • managed-image-protected-runtime

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

cv added 8 commits August 4, 2026 08:12
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
…age-ca-ordering

Signed-off-by: Carlos Villela <cvillela@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/helpers/dockerfile-run-commands.ts`:
- Around line 85-103: Update the scanner around the command parsing logic in
test/helpers/dockerfile-run-commands.ts:85-103 to recognize executable commands
inside $() and backtick substitutions, and avoid treating ${`#PATH`} as a comment;
alternatively reject these unsupported shell constructs before accepting a
reviewed command. Add executable-behavior fixtures in
test/dockerfile-run-commands.test.ts:13-24 containing one valid direct
invocation plus substitution and ${`#PATH`}; variants, and assert that each
produces an unreviewed RUN error.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a0166195-29a5-4f6e-8647-411063c5fea1

📥 Commits

Reviewing files that changed from the base of the PR and between bbf46c1 and 0c6994c.

📒 Files selected for processing (10)
  • agents/hermes/Dockerfile
  • ci/source-shape-test-budget.json
  • docs/security/configure-corporate-ca-trust.mdx
  • test/bundled-npm-brace-expansion-dockerfile-contract.test.ts
  • test/bundled-npm-ip-address-dockerfile-contract.test.ts
  • test/corporate-ca-build-tls-anchor.test.ts
  • test/dockerfile-run-commands.test.ts
  • test/helpers/dockerfile-run-commands.ts
  • test/hermes-final-image-layout.test.ts
  • test/node-tar-dockerfile-contract.test.ts
🚧 Files skipped from review as they are similar to previous changes (4)
  • docs/security/configure-corporate-ca-trust.mdx
  • agents/hermes/Dockerfile
  • ci/source-shape-test-budget.json
  • test/corporate-ca-build-tls-anchor.test.ts

Comment thread test/helpers/dockerfile-run-commands.ts
cv and others added 3 commits August 4, 2026 09:54
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/support/sandbox-images-workflow-boundary.test.ts`:
- Around line 431-444: The sandbox image workflow boundary test currently
removes CA-related fragments without validating command ordering. Extend the
mutation setup around hermes.run to move CA preparation after docker build and
place OpenSSL verification before CA preparation, then invoke the validator for
each mutated workflow and assert both are rejected. Keep the assertions focused
on invalid behavioral ordering rather than implementation details.

In `@tools/e2e/sandbox-images-workflow-boundary.mts`:
- Line 600: Make the CA boundary contract order-aware: in
tools/e2e/sandbox-images-workflow-boundary.mts at lines 600-600, define
pre-build and post-build fragment groups; at lines 618-618, validate each
fragment’s position relative to docker build; and at lines 682-686, classify the
Hermes CA fragments into the appropriate phase. In
test/e2e/support/sandbox-images-workflow-boundary.test.ts at lines 431-444,
mutate command ordering, and at lines 453-455, assert rejection for each invalid
placement so tests verify boundary ordering rather than presence alone.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 80d14d78-ae2e-46b5-8755-ecea359d556c

📥 Commits

Reviewing files that changed from the base of the PR and between 3c776ce and 4c9e97c.

📒 Files selected for processing (3)
  • .github/workflows/sandbox-images-and-e2e.yaml
  • test/e2e/support/sandbox-images-workflow-boundary.test.ts
  • tools/e2e/sandbox-images-workflow-boundary.mts

Comment thread test/e2e/support/sandbox-images-workflow-boundary.test.ts
Comment thread tools/e2e/sandbox-images-workflow-boundary.mts
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
@wscurran wscurran added area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior labels Aug 4, 2026
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Comment thread scripts/checks/select-ci-endpoint-ca-roots.mts Fixed
jyaunches and others added 3 commits August 4, 2026 16:18
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
cv added 2 commits August 4, 2026 14:11
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Comment thread scripts/checks/select-ci-endpoint-ca-roots.mts Fixed
cv added 6 commits August 4, 2026 14:45
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
@prekshivyas prekshivyas self-assigned this Aug 5, 2026
@prekshivyas
prekshivyas enabled auto-merge (squash) August 5, 2026 08:11
@cv
cv disabled auto-merge August 5, 2026 08:20
@cv
cv merged commit 7e844ef into main Aug 5, 2026
79 of 80 checks passed
@cv
cv deleted the codex/hermes-final-stage-ca-ordering branch August 5, 2026 08:20
@github-actions github-actions Bot added the v0.0.103 Release target label Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior v0.0.103 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants