fix(e2e): retry transient OpenClaw provider turns by cjagwani · Pull Request #11077 · NVIDIA/NemoClaw

cjagwani · 2026-09-04T19:22:39Z

Outcome

OpenClaw security E2E launch turns now retry one positively recognized transient provider-availability failure in a fresh session after a short backoff. Exhausted provider attempts report provider unavailability explicitly, while security-evidence, authentication, policy, invalid-response, and unknown failures remain fail-closed without retry.

Reason

A transient NVIDIA hosted-inference ServiceUnavailableError can leave the launch session with no assistant evidence, causing the OpenClaw security target to look like a posture failure even though the provider was unavailable.

Related issues

Fixes #10978

Changes

Emit an exact provider-unavailable marker from the missing-turn launch boundary only when its bounded PTY diagnostic positively identifies ServiceUnavailable or HTTP 5xx output without terminal auth, policy, or invalid-response evidence.
Add a two-attempt retry keyed to that structured marker, with a one-second delay before the second attempt.
Give each retry a fresh run identifier, prompts, host session lifecycle, cleanup, and artifact name.
Preserve immediate failure for invalid security evidence, authentication and authorization failures, network-policy and routing failures, malformed or invalid responses, cleanup failures, and unknown errors.
Add focused E2E-support coverage for transient-then-success, exhausted provider attempts, and non-retryable failure classes.
Register the shared live launch helper and focused test in the E2E mock-parity manifest.

Verification

./node_modules/.bin/vitest run --project e2e-support test/e2e/support/launch-agent-turn.test.ts test/e2e/support/launch-agent-turn-provider-availability.test.ts — 28 tests passed, 25 skipped.
npx tsx scripts/checks/e2e-mock-parity.mts --base origin/main --head HEAD — passed.
npm run e2e:assertions:check — passed with no assertion-ratchet growth.
Normal pre-commit hooks — passed, including repository checks, E2E assertion census, growth guardrails, Oxfmt, Oxlint, gitleaks, and commitlint.
Normal pre-push hooks — affected CLI TypeScript check passed.
GitHub commit verification — commits 5bf10d6590ba35f4169f1fb9762c324781562537, 2c6c6870208b2c0c64ff8bc30e3dab3f04f20c60, and 65677babd90a36043ebc8bf1c21bd9d08dcc2848 are Verified (valid).
PR Review Advisor — replaced combined-console positive classification with a structured launch marker and added a fixed retry delay; Retry-After is not parsed because this PTY boundary exposes no structured header.
Documentation writer review — no documentation change is needed because this only changes live E2E resilience and diagnostics.
Secret review — the diff contains no secrets, API keys, or credentials; the gitleaks hook passed.

Review notes

This PR is intentionally a draft so the implementation is not held only in a local worktree. Before it is marked ready, it must be rebased and revalidated on the exact origin/main baseline after #11071 and #10996 merge. The supported local PR Review Advisor must also run when PR_REVIEW_ADVISOR_API_KEY is available; it is not currently exported in the task environment.

Signed-off-by: Charan Jagwani cjagwani@nvidia.com

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

copy-pr-bot · 2026-09-04T19:22:42Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-09-04T19:23:34Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

github-code-quality · 2026-09-04T19:23:43Z

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall line coverage in commit 65677ba in the codex/fix-10978-prov... branch remains at 96%, unchanged from commit 4b254b9 in the main branch.

TypeScript / code-coverage/cli

The overall line coverage in commit 65677ba in the codex/fix-10978-prov... branch remains at 83%, unchanged from commit 4b254b9 in the main branch.

Show a line coverage summary of the most impacted files.

File	`main` 4b254b9	`codex/fix-10978-prov...` 65677ba	+/-
`src/lib/onboard...ght-messages.ts`	82%	72%	-10%
`src/lib/onboard...on-authority.ts`	88%	81%	-7%
`src/lib/onboard...ed-lifecycle.ts`	77%	75%	-2%
`src/lib/onboard.../application.ts`	71%	69%	-2%
`src/lib/onboard...-transaction.ts`	70%	69%	-1%
`src/lib/onboard...mage-runtime.ts`	51%	52%	+1%
`src/lib/onboard...er-gpu-patch.ts`	81%	82%	+1%
`src/lib/onboard...-diagnostics.ts`	96%	99%	+3%
`src/lib/onboard...ure-evidence.ts`	88%	95%	+7%
`src/lib/inferen...ocal-runtime.ts`	87%	97%	+10%

_{Updated September 04, 2026 20:13 UTC}

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

prekshivyas · 2026-09-04T19:33:58Z

+): boolean {
+  const output = resultText(result);
+  return (
+    TRANSIENT_PROVIDER_AVAILABILITY_RE.test(output) && !TERMINAL_LAUNCH_FAILURE_RE.test(output)


Could we avoid deciding whether to retry from regexes over the combined console output? Provider wording can change, and unrelated output could contain one of these strings. A structured error or exit code from the launch layer would make this classification more reliable.

Addressed in 65677ba. The launch script now emits an exact nemoclaw.e2e.launch-failure=provider-unavailable marker only from the missing-turn boundary after checking its bounded PTY diagnostic. The TypeScript retry decision requires that exact marker; raw provider wording in combined output no longer triggers a retry, and terminal auth/policy/invalid-response diagnostics still veto it.

prekshivyas · 2026-09-04T19:33:59Z

+    if (result.exitCode === 0) return result;
+    finalFailure = result;
+    providerUnavailable = isTransientProviderAvailabilityFailure(result);
+    if (!providerUnavailable) break;


Could we add a short backoff here, and honor Retry-After when it is available, before starting the second attempt? Retrying immediately is likely to hit the same correlated provider outage, so this may only help with isolated per-request failures.

Addressed in 65677ba with a fixed one-second delay before the second attempt, covered with fake timers. This PTY boundary does not expose a structured Retry-After header, so the change intentionally does not parse arbitrary console text for one; doing that would recreate the classification fragility from the adjacent finding.

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

github-actions · 2026-09-04T19:48:14Z

PR Review Advisor finished for commit 65677ba. Include the Advisor findings in the complete PR feedback collection. Verify and group valid findings before repair.

All previous runs

fix(e2e): retry transient OpenClaw provider turns

5bf10d6

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

cjagwani self-assigned this Sep 4, 2026

test(e2e): register provider retry parity

2c6c687

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

prekshivyas reviewed Sep 4, 2026

View reviewed changes

fix(e2e): structure provider retry signals

65677ba

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

This was referenced Sep 4, 2026

e2e(security): transient hosted-inference outage fails OpenClaw posture qualification #10978

Open

[Epic] Restore full E2E qualification on main #10979

Open

Navigation Menu

Uh oh!

fix(e2e): retry transient OpenClaw provider turns - #11077

fix(e2e): retry transient OpenClaw provider turns#11077
cjagwani wants to merge 3 commits into
mainfrom
codex/fix-10978-provider-availability-20260904

cjagwani commented Sep 4, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 4, 2026

Uh oh!

coderabbitai Bot commented Sep 4, 2026

Draft PR not reviewed

Uh oh!

github-code-quality Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

prekshivyas Sep 4, 2026

Uh oh!

cjagwani Sep 4, 2026

Uh oh!

prekshivyas Sep 4, 2026

Uh oh!

cjagwani Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cjagwani commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Outcome

Reason

Related issues

Changes

Verification

Review notes

Uh oh!

copy-pr-bot Bot commented Sep 4, 2026

Uh oh!

coderabbitai Bot commented Sep 4, 2026

Draft PR not reviewed

Uh oh!

github-code-quality Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Overview

TypeScript / code-coverage/plugin

TypeScript / code-coverage/cli

Uh oh!

prekshivyas Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

cjagwani Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

prekshivyas Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

cjagwani Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cjagwani commented Sep 4, 2026 •

edited

Loading

github-code-quality Bot commented Sep 4, 2026 •

edited

Loading