perf(native-eval): install only the active harness on fleet leases

[vincentkoc]

What happened

An OpenClaw-only native fleet run installs OpenClaw, Codex, Claude Code, and Hermes on every fresh lease.

In a two-repetition OpenClaw run, both machines cloned and installed Hermes, synchronized its skills, and installed the unused Codex and Claude Code npm packages before dispatching the OpenClaw task.

Expected behaviour

Fleet bootstrap should install only the harness assigned to that lease. Shared runtime dependencies such as Docker, Node, uv, and LiteLLM should remain common.

Steps to reproduce

python -m scripts.native_eval.fleet \
  --run-index <openclaw-only-run-index.json> \
  --local-root <output-dir> \
  --runner-root . \
  --task-archive <tasks.tar.gz> \
  --env-file <provider.env> \
  --openclaw-package-tarball <openclaw.tgz>

Relevant output

Installing to /opt/shellbench-native/hermes-agent...
Installing dependencies...
Syncing bundled skills to ~/.hermes/skills/ ...

The same bootstrap also installs @openai/codex and @anthropic-ai/claude-code.

Environment

• Python version: 3.14
• OS: macOS controller, Ubuntu 26.04 leases
• ShellBench commit: 42ceda028876f5134697e4473ecfe691e0af3ca6

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 23, 2026, 9:10 AM ET / 13:10 UTC.

Summary
Keep open: current main still bootstraps every lease with all four harnesses, so an OpenClaw-only fleet run incurs the reported unused installs. The MEMBER-authored, maintainer-review-marked issue needs an explicit maintainer choice before work is dispatched.

Reproducibility: yes. —source provides a high-confidence reproduction path: create an OpenClaw-only run index and hydrate a fresh lease; current bootstrap unconditionally installs Codex, Claude Code, and Hermes. A live lease was not executed in this read-only review.

Maintainer decision needed

Question	Recommendation
Should the fleet bootstrap now be changed to receive each run’s harness and install only that selected harness?	Sponsor a scoped bootstrap repair: Pass the run harness into the bootstrap script, retain shared dependencies, and add selection/manifest coverage.

Why: The source proves the inefficiency and a narrow repair is apparent, but this MEMBER-authored issue is explicitly marked for maintainer review and no-new-fix-pr handling.

Next step
The source-proven repair is small, but the MEMBER authorship and explicit maintainer/no-new-fix-pr signals require human routing rather than automated dispatch.

Review details

Best possible solution:

Pass the planned run’s harness into bootstrap, install only that harness plus shared dependencies, and add focused coverage for each supported harness selection.

Do we have a high-confidence way to reproduce the issue?

Yes—source provides a high-confidence reproduction path: create an OpenClaw-only run index and hydrate a fresh lease; current bootstrap unconditionally installs Codex, Claude Code, and Hermes. A live lease was not executed in this read-only review.

Is this the best way to solve the issue?

Yes—the narrowest maintainable repair is to carry the already-planned harness into bootstrap and conditionally install its tool, while leaving common runtime setup unchanged.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• A selective installer must keep the manifest and shared Docker, Node, uv, and LiteLLM setup valid for every supported harness; current fleet tests do not inspect the bootstrap script’s installed-harness set.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label changes:

• remove clawsweeper:no-new-fix-pr: Current issue advisory state no longer selects this label.
• remove clawsweeper:needs-maintainer-review: Current issue advisory state no longer selects this label.

Label justifications:

• P2: Fresh native fleet runs perform avoidable package and Hermes installation work, but task execution remains functional.
• impact:other: The concrete impact is fleet lease bootstrap cost and duration, outside the more specific impact categories.

Evidence reviewed

What I checked:

• Current fleet invocation lacks harness selection: Current main hydrates a lease and invokes the bootstrap script without passing the run harness, even though dispatch later supplies the run's harness to the remote runner. (scripts/native_eval/fleet.py:871, 884dd1bb5511)
• Bootstrap installs all harnesses unconditionally: The bootstrap always installs OpenClaw, Codex, Claude Code, and Hermes, then records versions for all four; it has no active-harness argument or conditional installer path. (scripts/native_eval/bootstrap_beast.sh:83, 884dd1bb5511)
• Run planning already establishes the intended selection boundary: The plan accepts one or more --harness selections and writes the selected harness names into the run index, so the active harness is available before a lease is provisioned. (scripts/native_eval/plan.py:36, 884dd1bb5511)
• Harness set and provenance: Blame attributes the unconditional npm and Hermes install lines to the native matrix runner introduction; current main still contains that implementation. (scripts/native_eval/bootstrap_beast.sh:83, 69f75c6629c4)
• Current-main and release check: Current main is 884dd1b and git tag --contains 884dd1bb55112c93292e1633081d62504ba49905 returned no local release tag; the behavior remains present on main rather than being fixed in a release. (scripts/native_eval/bootstrap_beast.sh:155, 884dd1bb5511)
• Maturity helper unavailable: The required shortlist helper could not classify the surface because this checkout has no qa/maturity-scores.yaml, so no stable-maturity label is supported.

Likely related people:

• vincentkoc: Introduced the native matrix runner commit that added the bootstrap script and remains the sole visible author across the current fleet/bootstrap history. (role: native-evaluation feature introducer and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/bootstrap_beast.sh, scripts/native_eval/fleet.py, tests/test_native_eval_fleet.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.