fix(native-eval): benchmark genuine OpenClaw code mode

[vincentkoc]

What does this PR do?

Makes the OpenClaw native-eval arms represent genuine direct, directory, and
Code Mode execution, then exports and validates their public trajectory bundles
before a run can be scored.

Why?

Fixes #61.

The earlier code arm selected the legacy tool_search_code bridge rather than
OpenClaw Code Mode. The harness also masked setup failures, forced thinking off,
terminated the runtime before lifecycle cleanup settled, and reconstructed
delegated traces from private session files. Those failures made the released
direct/code comparison invalid and disproportionately erased Code Mode's nested
tool calls.

Changes

• Map direct, directory, and code to explicit, mutually exclusive
  OpenClaw tool surfaces.
• Propagate the requested reasoning effort through planning, dispatch,
  OpenClaw defaults, subagents, CLI execution, and manifests.
• Fail setup immediately and let openclaw agent exit naturally.
• Export root and child trajectories through
  openclaw sessions export-trajectory.
• Capture child runs at the canonical pre-cleanup terminal hook, including
  nested and repeated runs, without blocking the Gateway.
• Validate audit initialization, trace/session identity, event counts, terminal
  status, complete Code Mode snapshots, and the exact provider-visible
  exec/wait surface.
• Reconstruct Code Mode nested calls and complete session-tree usage from the
  public export bundles.
• Bound child-export stabilization and reject missing or failed evidence rather
  than publishing partial traces.

Tests

• Blacksmith Testbox full suite: 454 passed, 5 skipped
• Blacksmith Testbox focused runner suite after final fixes: 61 passed
• Ruff clean
• Python compile, git diff --check, generated shell syntax for all three
  modes, and generated audit-plugin node --check
• Fresh Codex autoreview: clean
• Fresh matched direct/Code Mode ShellBench batch

The July 29, 2026 released code result remains legacy Tool Search bridge
evidence only. It is not evidence about genuine OpenClaw Code Mode.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 22, 2026, 7:08 PM ET / 23:08 UTC.

ClawSweeper review

What this changes

The PR updates ShellBench’s OpenClaw native-evaluation runner to configure direct, directory, and Code Mode runs explicitly and validate exported trajectories before scoring.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open: the PR’s explicit mode handling is not reached by normal matrix generation, so its claimed Code Mode cohort is never planned.

Priority: P2
Reviewed head: f5b31cb1260b631b811bf0ede9f4eba51a94b98c

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The patch contains substantial, targeted harness work, but a P1 planning gap prevents its central benchmark claim from being true.
Proof confidence	🌊 off-meta tidepool	Not applicable: This MEMBER-authored PR is exempt from the external-contributor real-behavior-proof gate; the remaining blocker is a source-visible planning defect.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This MEMBER-authored PR is exempt from the external-contributor real-behavior-proof gate; the remaining blocker is a source-visible planning defect.
Evidence reviewed	5 items	Missing cohort expansion: Normal matrix generation creates exactly one RunSpec for each harness/model/repetition combination and never assigns openclaw_tool_mode; an OpenClaw row therefore reaches dispatch with an empty mode. Default becomes direct mode: The OpenClaw harness maps an unset tool mode to direct, so every normally planned OpenClaw run uses direct tools rather than producing directory and Code Mode cohorts. Regression test preserves incomplete plan: The changed test asserts a 96-row matrix and checks reasoning effort only; it does not require distinct direct, directory, and code OpenClaw entries.
Findings	1 actionable finding	[P1] Generate planned runs for every OpenClaw tool mode
Security	None	None.

Live Verification

Command: python -m scripts.native_eval.plan --help

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

• FAIL expect_output: --reasoning-effort

How this fits together

ShellBench builds a benchmark matrix, dispatches each run to a native harness, then scores exported trajectories. This change affects how OpenClaw runs receive tool-mode configuration and how their evidence is accepted.

flowchart LR
  A[Benchmark matrix] --> B[Planned OpenClaw runs]
  B --> C[Tool-mode dispatch]
  C --> D[OpenClaw harness]
  D --> E[Trajectory export]
  E --> F[Validation]
  F --> G[Scored result]

Loading

Before merge

• Generate planned runs for every OpenClaw tool mode (P1) - Normal planning appends one OpenClaw run per model/repetition without setting openclaw_tool_mode; dispatch therefore passes an empty value and the harness defaults to direct. The requested directory and Code Mode cohorts are never run. Expand the matrix and assert distinct labelled entries for all three modes.
• Resolve merge risk (P1) - Merging would publish a harness that validates explicit modes only when externally injected, while normal benchmark plans still run OpenClaw in direct mode and can misrepresent a direct-versus-Code-Mode comparison.
• Complete next step (P2) - The remaining P1 defect is a narrow mechanical planning repair with a clear regression-test boundary.

Findings

• [P1] Generate planned runs for every OpenClaw tool mode — scripts/native_eval/models.py:180

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch scope	9 files affected; production +699/-477, tests +1116/-475	The patch spans planning, dispatch, runtime capture, and validation, making an end-to-end matrix assertion important.

Merge-risk options

Maintainer options:

1. Generate all OpenClaw cohorts (recommended)
   Expand matrix generation and add focused plan tests so each OpenClaw model/repetition produces direct, directory, and code entries before dispatch.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Expand normal OpenClaw matrix generation into distinct direct, directory, and code cohorts; add focused plan-generation regression coverage and run the native-evaluation tests.

Technical review

Best possible solution:

Expand standard OpenClaw plans into distinct, uniquely labelled direct, directory, and code runs, preserve each mode through reruns and manifests, and prove the generated matrix before running a matched batch.

Do we have a high-confidence way to reproduce the issue?

Yes: source inspection shows normal planning leaves openclaw_tool_mode unset, and the harness deterministically defaults that value to direct.

Is this the best way to solve the issue?

No: explicit dispatch support is insufficient until the normal matrix creates one distinct OpenClaw run for every requested mode.

Full review comments:

• [P1] Generate planned runs for every OpenClaw tool mode — scripts/native_eval/models.py:180
  Normal planning appends one OpenClaw run per model/repetition without setting openclaw_tool_mode; dispatch therefore passes an empty value and the harness defaults to direct. The requested directory and Code Mode cohorts are never run. Expand the matrix and assert distinct labelled entries for all three modes.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: The defect invalidates benchmark cohort selection but does not directly alter the OpenClaw production runtime.
• merge-risk: 🚨 other: A merged harness can generate misleading comparative benchmark results because the requested Code Mode cohort is absent.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This MEMBER-authored PR is exempt from the external-contributor real-behavior-proof gate; the remaining blocker is a source-visible planning defect.

Evidence

Acceptance criteria:

• [P1] python -m pytest tests/test_native_eval_runner.py -k 'matrix_plan or run_index' -q.
• [P1] python -m pytest tests/test_native_eval_fleet.py -k 'openclaw_tool_mode' -q.

What I checked:

• Missing cohort expansion: Normal matrix generation creates exactly one RunSpec for each harness/model/repetition combination and never assigns openclaw_tool_mode; an OpenClaw row therefore reaches dispatch with an empty mode. (scripts/native_eval/models.py:180, f5b31cb1260b)
• Default becomes direct mode: The OpenClaw harness maps an unset tool mode to direct, so every normally planned OpenClaw run uses direct tools rather than producing directory and Code Mode cohorts. (scripts/native_eval/harnesses.py:367, f5b31cb1260b)
• Regression test preserves incomplete plan: The changed test asserts a 96-row matrix and checks reasoning effort only; it does not require distinct direct, directory, and code OpenClaw entries. (tests/test_native_eval_runner.py:61, f5b31cb1260b)
• Prior finding remains unresolved: The affected planning files are byte-identical to the earlier reviewed head, so the prior P1 finding remains applicable. (scripts/native_eval/models.py:180, f5b31cb1260b)
• Feature provenance: Current main history attributes the auditable native matrix runner and subsequent native-harness stabilization to Vincent Koc. (scripts/native_eval/models.py:140, 69f75c6629c4)

Likely related people:

• Vincent Koc: Git history attributes the native matrix runner, stabilization work, and this branch’s affected planning implementation to Vincent Koc. (role: native-evaluation feature author and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, f5b31cb1260b; files: scripts/native_eval/models.py, scripts/native_eval/plan.py, scripts/native_eval/fleet.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Generate and test distinct direct, directory, and code OpenClaw plan entries.
• Run a matched direct/Code Mode batch after the corrected plan is accepted.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (72 earlier review cycles; latest 8 shown)

• reviewed 2026-08-09T19:46:21.666Z sha f5b31cb :: found issues before merge. :: [P1] Generate planned runs for every OpenClaw tool mode
• reviewed 2026-08-09T22:02:01.596Z sha f5b31cb :: needs changes before merge. :: [P1] Generate planned runs for every OpenClaw tool mode
• reviewed 2026-08-09T23:12:37.716Z sha f5b31cb :: needs changes before merge. :: [P1] Generate planned runs for every OpenClaw tool mode
• reviewed 2026-08-11T23:11:15.062Z sha f5b31cb :: needs changes before merge. :: [P1] Expand normal plans across OpenClaw tool modes
• reviewed 2026-08-12T01:18:05.700Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a plan entry for every OpenClaw tool mode
• reviewed 2026-08-12T06:19:16.517Z sha f5b31cb :: needs changes before merge. :: [P1] Expand normal OpenClaw plans across tool modes
• reviewed 2026-08-15T11:06:50.762Z sha f5b31cb :: needs changes before merge. :: [P1] Generate planned runs for every OpenClaw tool mode
• reviewed 2026-08-15T18:44:55.752Z sha f5b31cb :: needs changes before merge. :: [P1] Generate a planned run for each OpenClaw tool mode