Skip to content

test(inference): compare PR 12626 with isolated native NVIDIA fix - #12854

Closed
jyaunches wants to merge 29 commits into
mainfrom
codex/test-12626-with-12826
Closed

jyaunches wants to merge 29 commits into
mainfrom
codex/test-12626-with-12826

Conversation

@jyaunches

@jyaunches jyaunches commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Outcome

Temporary test candidate for the NVIDIA inference failure blocking #12626. This draft combines #12626 at 3d7e25fb805365ff3db4eda38df0e023ed5eb334 with only the 31-file PR delta from #12826 at 57ff4ca0fca1a85a8db53891a16fe8a3b188846f.

Do not merge. Neither original PR branch nor main is changed. No newer main history was merged into this candidate.

Reason

The retained Brev run passed image creation, SSH access, and commit verification, then failed native NVIDIA onboarding verification with curl exit 56. This candidate tests whether the scoped native policy and supervisor-placeholder changes resolve that failure. The prior run did not retain a proxy response that establishes CONNECT 403 as its cause.

Related issues

Refs #12622, #12822. Temporary comparison of #12626 and #12826.

Changes

Verification

  • git diff --cached --check — passed before commit; exactly 31 files changed from fix(onboard): reuse retained gateway registration #12626.
  • Focused CLI tests — 254 passed across 8 files, covering native policy, inference probes, initial policy, create plans, rebuild handoff, and inference setup.
  • Focused Deep Agents integration and E2E helper tests — 350 passed across 6 files after correcting disposable-container prerequisites (non-root user and Python venv support).
  • Normal pre-push publication validation and all three TypeScript checks — passed using the documented validation environment below.
  • GitHub commit signatures — all 28 preserved baseline commits and the combined commit verified.
  • Live NVIDIA onboarding verification — passed on 40ca6b0 in run 37835993189. The baseline failed with curl exit 56. The full combined run failed afterward in Launchable routing; no full-suite pass is claimed.
  • Diff reviewed for credentials: no secrets, API keys, or credentials added.

The narrow native NVIDIA onboarding comparison is complete. The full Brev suite currently assumes a global inference route that native NVIDIA setup deliberately does not create (full-e2e.test.ts:1208-1260 versus setup-inference.ts:968-1014). That is a separate harness blocker; no full-suite pass is claimed. The trusted full workflow supplied the narrow comparison. It failed after onboarding and before these stale assertions.

Live comparison result

The isolated fix passed the previously failing native NVIDIA onboarding verification on the preserved #12626 baseline. Both runs used nvidia-prod with nvidia/nemotron-3-super-120b-a12b.

  • Baseline 3d7e25f: run 37801673014 failed native onboarding verification with curl exit 56.
  • Combined 40ca6b0: run 37835993189 recorded healthy deployment verification and completed onboarding at 21:03:57 UTC. The success message requires the native sandbox invocation to return a successful HTTP status and an API-shaped response.
  • The later brev-quickstart routing step failed: selected sandbox agent does not match the requested route. The full E2E assertion at full-e2e.test.ts:1103 therefore failed. Later global-route assertions were not reached.
  • Image and runtime identity checks passed: candidate 40ca6b0a747204e22d6797e11fe8f70471b3e4da, clean baked checkout, no runtime overrides, image producer 37836564832, image repository 37f6f7e5177753105b1a6bc2a7d324312645e2dc.
  • Workspace nclaw-e2e-37835993189-1 (6hdmi5ye1) was deleted and confirmed ABSENT at 21:06:22 UTC. Retained evidence includes the child log, identity checks, and cleanup record.

Comparison limitation: the image repository advanced from ae1af18af08d9297fc56c6b4ad35a1d9618dbacc to 37f6f7e5177753105b1a6bc2a7d324312645e2dc. Its two changes cover terminal readiness and selected-sandbox routing. Routing runs after onboarding. This is evidence that the combined candidate passes the original probe, not a claim of fully controlled A/B testing or full qualification.

General PR CI also reports an assertion-budget increase in concurrent-gateway-ports.test.ts. That file is unchanged by the isolated 31-file fix. The temporary candidate remains unchanged and is not merge-ready.

Review notes

Sensitive paths include agents/langchain-deepagents-code/**, src/lib/inference/**, and src/lib/onboard/** in NVIDIA/NemoClaw. The combined commit received source inspection and the focused checks above; no independent local Advisor clearance exists for this combined tree. These paths remain awaiting independent review.

The user authorized this temporary publication and the trusted comparison runs. The candidate is not submitted for merge readiness. The known foreign-placeholder validation and stale native-grant findings from #12826 remain outside this comparison's repair scope; a passing inference test cannot clear them.

Validation used the documented isolated alternative because the preserved baseline predates the canonical validation tooling. Candidate tree: a42d1ab887a7d18f2b824b2364b3ccca361aa5d3; publication commit: 40ca6b0a747204e22d6797e11fe8f70471b3e4da. Canonical validation tooling came from cd95b77c26dac10ad02f2cde88de1c9436174d00, unchanged in refreshed main 9966f01e83f2e17456e763dc778b1827f55a0c52.

Repository JavaScript checks and tests ran as a non-root user in a task-owned Linux container, with no contributor home, credentials, provider key, or Docker socket mounted. Only the disposable validation copy received canonical manifests, Vitest configuration, and test helpers; publication source retains the approved baseline. Normal Git hooks remain enabled. Pinned canonical Mac Oxlint parses source with the canonical configuration because the Linux copy hit an allocator failure. The canonical TypeScript compiler also parsed a disposable source copy on the Mac after the Linux VM exhausted memory; it passed with the verified SDK declarations installed. Candidate-controlled validators are not claimed as independent evidence. The live evidence below supplements this local validation.


Signed-off-by: Julie Yaunches jyaunches@nvidia.com

rsliter and others added 29 commits October 5, 2026 10:58
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
Registration observation and registration can await while the declared gateway owner changes.
Three regression cases reproduced registration or selection after that change.

Reload the bound owner immediately before each registration and selection effect.
Preserve registrations when authority changes so cleanup cannot mutate a different owner.

Validation: 74 focused gateway runtime and observer tests pass. CLI type checking passes
with an 8 GiB Node heap. Repaired Advisor finding F-security-built-in-quality-6daa530b0405f96b2753.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Integrate main b769619, including the resumed OpenRouter inference repair in #12681,
so the next exact-candidate Brev image includes that work. The merge had no conflicts.

Validation: 112 focused gateway and OpenRouter CLI tests pass, plus four OpenRouter
onboarding integration tests. The retained-gateway PR delta remains limited to 11 files.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Registration metadata can remain after its gateway becomes unavailable. Carry the existing
observer health result into the retained-gateway capture and require it after final cleanup.
This addresses CodeRabbit review 5444316020 without changing lifecycle behavior or retries.

Validation: 14 observer tests, the E2E assertion and semantic-phase checks, and CLI type
checking pass. The real post-cleanup health observation remains pending trusted Brev E2E.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Advisor found that post-registration authority drift retained a new local
registration without identifying it or explaining recovery. Preserve the
authority guard, report the registration name and inspection action, and
recheck authority before publishing the selected gateway to the client.

The recovery-message regression failed before the repair. A sibling
selection-drift case exposed a missing final authority check and also failed
before that check was added. All 75 runtime and observer tests now pass;
CLI typecheck passes. No changed-owner cleanup or gateway destruction occurs.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Integrate main through #12765 to accept verified dual SDK archives in
trusted E2E installation. Preserve retained-gateway behavior and the
required #12681 ancestry.

Validation: 75 gateway runtime tests, 32 SDK installer tests, and 5 SDK
package tests passed after the conflict-free merge. Brev SSH readiness
remains unresolved and is not claimed as passing.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
Temporary validation candidate based on 3d7e25f.
Apply only the 31-file PR delta from PR #12826 at
57ff4ca; retain the original baseline
and exclude newer main history. Not intended for merge.

Co-authored-by: San Dang <sdang@nvidia.com>
Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
@jyaunches jyaunches self-assigned this Oct 8, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

@github-code-quality

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall line coverage in commit 40ca6b0 in the codex/test-12626-wit... branch is 97%. The line coverage in commit 63002cd in the main branch is 96%.

Show a line coverage summary of the most impacted files.
File main 63002cd codex/test-12626-wit... 40ca6b0 +/-
nemoclaw/src/onboard/config.ts 98% 96% -2%
nemoclaw/src/index.ts 94% 93% -1%
nemoclaw/src/bl...print/runner.ts 95% 95% 0%
nemoclaw/src/bl...ime-identity.ts 97% 97% 0%
nemoclaw/src/bl...t-management.ts 100% 100% 0%
nemoclaw/src/co.../config-show.ts 100% 100% 0%
nemoclaw/src/commands/slash.ts 100% 100% 0%
nemoclaw/src/on...native-route.ts 0% 100% +100%

@jyaunches

Copy link
Copy Markdown
Contributor Author

Closing this do-not-merge comparison experiment. #12626 and #12826 have both merged, and the experiment answered its narrow question: NVIDIA onboarding passed. The full run (https://github.com/NVIDIA/NemoClaw/actions/runs/37835993189) subsequently failed at Launchable routing, addressed separately by merged brevdev/nemoclaw-image#202. Remaining Brev harness work belongs in #12907.

@jyaunches jyaunches closed this Oct 12, 2026
@wscurran wscurran added area: providers Inference provider integrations and provider behavior area: security Security controls, permissions, secrets, or hardening integration: dcode LangChain Deep Code integration behavior provider: nvidia NVIDIA inference endpoint, NIM, or NVIDIA provider behavior labels Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: providers Inference provider integrations and provider behavior area: security Security controls, permissions, secrets, or hardening integration: dcode LangChain Deep Code integration behavior provider: nvidia NVIDIA inference endpoint, NIM, or NVIDIA provider behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants