feat(native-eval): support OpenClaw candidate packages

[vincentkoc]

What does this PR do?

Adds exact local OpenClaw package artifacts to native evaluation campaigns.

Why?

ShellBench needs reproducible pre-publication baseline/candidate runs without
publishing temporary npm versions or mutating warmed machines.

Fixes #56

Changes

• add --openclaw-package-tarball campaign input and OpenClaw-only validation
• stage, upload, verify, and install the exact package artifact
• bind resume identity to package name, version, and SHA-256
• record candidate provenance in run manifests and research audits
• document the candidate workflow and failure modes

Exact proof

• Candidate tarball SHA:
  933616f90607b43970d2aaaf1de051a5fe8a751e514882f69c557d56e15755d2
• Exact c7a.8xlarge AWS run installed the package and exported three results.
• Exported manifests preserved package name openclaw, version 2026.7.2,
  artifact filename, and SHA-256.
• The run later exposed an independent Tool Search control bug tracked in fix(native-eval): make OpenClaw Tool Search controls explicit #61;
  that invalidates its score comparison, not the package staging proof.

Tests

• 108 focused native-eval tests pass

• Ruff passes on all touched Python files

• fresh Codex autoreview reports no actionable findings

• python -m pytest -q passes locally

• python -m ruff check clawbench app.py scripts tests passes locally, or the change is docs-only

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 5, 2026, 8:32 AM ET / 12:32 UTC.

ClawSweeper review

What this changes

The branch adds an OpenClaw-only native-evaluation option that stages a local npm tarball, verifies its identity, installs it on leased hosts, and records provenance in run and audit artifacts.

Merge readiness

⚠️ Ready for maintainer review - 4 items remain

Keep open for maintainer security-boundary approval. Current main remains registry-only, while this member-authored PR adds a useful optional path that executes an operator-supplied package during remote root bootstrap.

Priority: P2
Reviewed head: 3da8ec53b7964b64469dafd3090abcf318d942c8
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The patch has substantial focused coverage and a coherent design, but its remote-run claim is not independently inspectable and the install trust policy remains unresolved.
Proof confidence	🦪 silver shellfish (2/6)	Not applicable: The external-contributor proof gate does not apply to this MEMBER-authored PR; its stated AWS run would nevertheless be stronger with redacted manifest or terminal evidence.
Patch quality	🐚 platinum hermit (4/6)	Security review found an item that needs attention.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The external-contributor proof gate does not apply to this MEMBER-authored PR; its stated AWS run would nevertheless be stronger with redacted manifest or terminal evidence.
Evidence reviewed	5 items	Current main behavior: Current main installs OpenClaw from the registry by version, so it does not already provide the proposed local-candidate package capability. Candidate install path: The branch verifies the selected tarball's name, version, and SHA-256, then supplies it as the npm install input during bootstrap. Remote transfer boundary: The controller copies the staged candidate to the leased host and invokes bootstrap with its path, hash, and version before provider environment installation.
Findings	None	None.
Security	Needs attention	Define trust for remote npm tarballs: The hash verifies that the selected bytes arrived intact, but it does not establish a trusted producer. npm installation can execute package lifecycle code during root bootstrap on each leased host.

How this fits together

Native evaluation stages campaign inputs locally, bootstraps leased AWS hosts, runs OpenClaw against benchmark tasks, and exports manifests for audit. This change introduces an unpublished package artifact into that host-bootstrap path.

flowchart LR
A[Candidate package tarball] --> B[Campaign controller]
B --> C[Identity validation and staging]
C --> D[Leased host bootstrap]
D --> E[OpenClaw package install]
E --> F[Benchmark run]
F --> G[Run manifests and audit]

Loading

Decision needed

Question	Recommendation
Should native evaluation accept a locally supplied OpenClaw tarball for root-level remote installation under a trusted-operator contract, or require an approved build/provenance boundary first?	Require approved provenance: Accept candidate packages only from a documented approved build or provenance path before installing them on leased hosts.

Why: This is an intentional new remote code-execution path; integrity verification alone cannot choose the repository's trust policy.

Before merge

• Resolve security concern: Define trust for remote npm tarballs - The hash verifies that the selected bytes arrived intact, but it does not establish a trusted producer. npm installation can execute package lifecycle code during root bootstrap on each leased host.
• Resolve merge risk (P1) - Selecting the option transfers an operator-supplied archive into a root-level npm install; its SHA-256 proves byte identity but not a trusted producer, and package lifecycle scripts may execute.
• Resolve merge risk (P1) - The cited candidate run cannot establish valid score comparison until the independent Tool Search control defect is resolved in fix(native-eval): make OpenClaw Tool Search controls explicit #61.

Findings

• [medium] Define trust for remote npm tarballs — scripts/native_eval/bootstrap_beast.sh:95

Agent review details

Security

Needs attention: The optional candidate path introduces remote root-level npm package execution, so maintainers need to approve the intended artifact trust boundary.

Review metrics

Metric	Value	Why it matters
Patch surface	production +397, docs +16, tests +410	The capability spans fleet staging, bootstrap, manifests, audit exports, runbook guidance, and focused regression coverage.

Merge-risk options

Maintainer options:

1. Require an approved artifact source (recommended)
   Define and enforce the permitted producer or provenance boundary before enabling remote installation of a candidate tarball.
2. Accept the trusted-operator contract
   Approve the option as an operator-only capability and document that the chosen npm package can execute code during bootstrap.
3. Pause candidate installation
   Leave registry-only evaluation in place if maintainers do not want this remote code-execution boundary.

Technical review

Best possible solution:

Retain exact-artifact evaluation only with an explicit approved-producer or trusted-operator policy, document that lifecycle code can run during bootstrap, and attach redacted leased-host output plus exported provenance for the claimed run.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a proposed capability, not a report of broken current behavior. The PR describes a real campaign run, but no inspectable run artifact is attached.

Is this the best way to solve the issue?

Unclear: the identity, resume, and audit controls are focused, but maintainers must choose whether integrity-only validation is sufficient for the new remote installation trust boundary.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The external-contributor proof gate does not apply to this MEMBER-authored PR; its stated AWS run would nevertheless be stronger with redacted manifest or terminal evidence.
• remove status: 📣 needs proof: Current PR status label is status: ⏳ waiting on author.
• remove rating: 🧂 unranked krab: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P2: This is a bounded native-evaluation capability with operational impact but no demonstrated user-facing outage.
• merge-risk: 🚨 security-boundary: The new option sends an operator-selected npm package to a leased host for root-level installation, where lifecycle scripts may execute.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The external-contributor proof gate does not apply to this MEMBER-authored PR; its stated AWS run would nevertheless be stronger with redacted manifest or terminal evidence.

Evidence

Security concerns:

• [medium] Define trust for remote npm tarballs — scripts/native_eval/bootstrap_beast.sh:95
  The hash verifies that the selected bytes arrived intact, but it does not establish a trusted producer. npm installation can execute package lifecycle code during root bootstrap on each leased host.
  Confidence: 0.96

What I checked:

• Current main behavior: Current main installs OpenClaw from the registry by version, so it does not already provide the proposed local-candidate package capability. (scripts/native_eval/bootstrap_beast.sh:83, 884dd1bb5511)
• Candidate install path: The branch verifies the selected tarball's name, version, and SHA-256, then supplies it as the npm install input during bootstrap. (scripts/native_eval/bootstrap_beast.sh:95, 3da8ec53b796)
• Remote transfer boundary: The controller copies the staged candidate to the leased host and invokes bootstrap with its path, hash, and version before provider environment installation. (scripts/native_eval/fleet.py:486, 3da8ec53b796)
• Feature history: Native-evaluation fleet and bootstrap behavior appears to originate with Vincent Koc's auditable matrix runner and has been maintained through subsequent current-main fixes. (scripts/native_eval/fleet.py:1, 69f75c6629c4)
• Canonical work: The PR explicitly implements the open issue requesting exact candidate-package support; the Tool Search issue is a separate benchmark-control defect.

Likely related people:

• vincentkoc: Current-main history shows sustained authorship of the fleet/bootstrap/audit path, including its introduction and recent trace fixes. (role: native-evaluation feature owner; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 569b5c39c783; files: scripts/native_eval/fleet.py, scripts/native_eval/bootstrap_beast.sh, scripts/native_eval/run_job.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Attach redacted bootstrap or manifest output for the claimed leased-host run.
• Obtain maintainer approval for the candidate-artifact trust boundary.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (19 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T06:14:50.813Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T10:56:42.938Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T12:21:53.837Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-01T17:57:08.552Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T13:17:17.964Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T14:58:07.329Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T17:06:04.651Z sha 3da8ec5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-02T19:15:07.792Z sha 3da8ec5 :: needs maintainer review before merge. :: none