fix(native-eval): reject invalid harness runs

[vincentkoc]

What does this PR do?

Records structured native-eval execution outcomes and rejects runs where every
trial failed at the harness, infrastructure, or verifier layer.

Why?

Verifier rewards are useful diagnostics, but they are not proof that the
benchmark harness executed correctly. Fully invalid runs were being presented
as completed score data, and fleet recovery could infer success without an
archived exit status.

Fixes #63

Changes

• classify clean, agent, harness, infrastructure, and verifier outcomes
• preserve per-trial rewards while recording run-level execution acceptance
• exclude rejected runs from score-eligible aggregates
• infer historical OpenClaw exit 70/71 outcomes correctly
• remove fleet success inference based only on result count and checkpoint text

Tests

• python -m pytest -q passes on Blacksmith Testbox
• python -m ruff check clawbench app.py scripts tests passes on Blacksmith Testbox

Focused proof: 117 tests passed; Ruff clean. Autoreview reported no actionable
findings.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 23, 2026, 5:56 AM ET / 09:56 UTC.

ClawSweeper review

What this changes

The PR classifies native-evaluation failures, excludes fully invalid runs from score eligibility, and makes fleet recovery require a recorded terminal exit status.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open: the branch removes recovery’s fallback before its producer actually archives the required exit status, so a successful exported run can be failed and rerun.

Priority: P1
Reviewed head: 9f4490e8e4370cbc752404a1becce42d5a6a69f8

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The patch has a focused goal, but the unresolved producer/consumer mismatch blocks merge confidence.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is exempt from the external contributor proof gate; its stated tests remain supplemental and do not resolve the artifact-contract defect.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is exempt from the external contributor proof gate; its stated tests remain supplemental and do not resolve the artifact-contract defect.
Evidence reviewed	4 items	Exit status is written too late and not packaged: The remote script writes exit_status only in its EXIT trap, after creating the tar archive; the archive inputs do not include the run-state directory. Recovery now requires the missing artifact: The PR removes the full-coverage/checkpoint fallback, leaving recovery to complete only when an existing run_exit_code or archived exit_status is available. Producer remains unchanged from main: The PR does not modify remote_run.sh, so it cannot supply the newly mandatory archived status; current main retains the older inferred-success fallback.
Findings	1 actionable finding	[P1] Archive the exit status before requiring it for recovery
Security	None	None.

Live Verification

Command: python -m pytest -q tests/test_native_eval_fleet.py -k recovery

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

• FAIL expect_output: passed

How this fits together

Native evaluation runs benchmark tasks through a harness and exports trial artifacts. Aggregation turns those artifacts into scores, while fleet recovery uses them to decide whether a remote campaign completed.

flowchart LR
A[Evaluation tasks] --> B[Harness execution]
B --> C[Trial results]
C --> D[Score aggregation]
C --> E[Exported archive]
E --> F[Fleet recovery]
F --> G[Campaign status]

Loading

Before merge

• Archive the exit status before requiring it for recovery (P1) - The new recovery path only accepts an exit_status inside the final archive, but remote_run.sh writes that file in its EXIT trap after it creates the archive and never adds the run-state directory to the tar inputs. A completed run that reaches recovery is therefore marked failed and scheduled for a full rerun. Persist and package the status before this consumer reads it; this repeats the prior unresolved blocker on unchanged producer code.
• Resolve merge risk (P1) - Merging as written can mark a completed exported campaign as failed and schedule a full rerun whenever recovery lacks the status file that this producer never packages.
• Complete next step (P2) - A narrow, source-proven producer/consumer repair is available on this PR branch.

Findings

• [P1] Archive the exit status before requiring it for recovery — scripts/native_eval/fleet.py:1267-1274

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	production +186/-28; tests +93/-7	The change spans the native-eval artifact producer, score aggregation, and fleet recovery contract.

Merge-risk options

Maintainer options:

1. Package terminal status before recovery consumes it (recommended)
   Write the terminal exit status into export metadata before tar creation and cover recovery from that actual archive shape.
2. Pause the recovery behavior change
   Do not remove the existing fallback until the archive producer and consumer share a tested status contract.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Persist the final exit status before tar creation, include it in export metadata, and add recovery coverage that reads the packaged artifact.

Technical review

Best possible solution:

Persist the terminal status before final packaging, include it in the export metadata, and retain fail-closed recovery for archives that genuinely lack it.

Do we have a high-confidence way to reproduce the issue?

Yes—the source establishes the path: final packaging happens before the EXIT trap writes exit_status, while the revised recovery path requires that status from the archive.

Is this the best way to solve the issue?

No—the fail-closed policy is appropriate only once the producer persists and packages the status before recovery consumes it.

Full review comments:

• [P1] Archive the exit status before requiring it for recovery — scripts/native_eval/fleet.py:1267-1274
  The new recovery path only accepts an exit_status inside the final archive, but remote_run.sh writes that file in its EXIT trap after it creates the archive and never adds the run-state directory to the tar inputs. A completed run that reaches recovery is therefore marked failed and scheduled for a full rerun. Persist and package the status before this consumer reads it; this repeats the prior unresolved blocker on unchanged producer code.
  Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• remove rating: 🦪 silver shellfish: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.

Label justifications:

• P1: The introduced recovery path can fail a completed campaign and consume another fleet attempt.
• merge-risk: 🚨 compatibility: The PR removes recovery compatibility for exports without the newly required terminal-status artifact.
• merge-risk: 🚨 availability: The missing status can make fleet recovery fail and rerun otherwise completed work.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This member-authored PR is exempt from the external contributor proof gate; its stated tests remain supplemental and do not resolve the artifact-contract defect.

Evidence

Acceptance criteria:

• [P1] python -m pytest -q tests/test_native_eval_fleet.py tests/test_native_eval_runner.py tests/test_native_eval_aggregate.py.
• [P1] python -m ruff check scripts/native_eval tests.

What I checked:

• Exit status is written too late and not packaged: The remote script writes exit_status only in its EXIT trap, after creating the tar archive; the archive inputs do not include the run-state directory. (scripts/native_eval/remote_run.sh:57, 9f4490e8e437)
• Recovery now requires the missing artifact: The PR removes the full-coverage/checkpoint fallback, leaving recovery to complete only when an existing run_exit_code or archived exit_status is available. (scripts/native_eval/fleet.py:1267, 9f4490e8e437)
• Producer remains unchanged from main: The PR does not modify remote_run.sh, so it cannot supply the newly mandatory archived status; current main retains the older inferred-success fallback. (scripts/native_eval/remote_run.sh:158, 884dd1bb5511)
• Native-eval provenance: The native matrix and fleet surface appears to originate with the feature commit, with subsequent stabilization work by the same contributor. (scripts/native_eval/fleet.py:1217, 69f75c6629c4)

Likely related people:

• vincentkoc: Introduced the native matrix/fleet feature, followed with stabilization work, and authored this repair branch. (role: native-eval feature author and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 9f4490e8e437; files: scripts/native_eval/remote_run.sh, scripts/native_eval/fleet.py, scripts/native_eval/aggregate.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Persist and package terminal status before recovery consumes it.
• Add a regression that recovers from the actual exported archive shape.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (72 earlier review cycles; latest 8 shown)

• reviewed 2026-08-09T13:37:03.806Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before final packaging
• reviewed 2026-08-09T14:51:41.543Z sha 9f4490e :: needs changes before merge. :: [P1] Package the exit status before enforcing its presence
• reviewed 2026-08-09T20:13:42.955Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery
• reviewed 2026-08-09T22:00:34.458Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery
• reviewed 2026-08-11T23:11:34.595Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery
• reviewed 2026-08-12T01:18:09.867Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery
• reviewed 2026-08-12T09:12:19.212Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery
• reviewed 2026-08-15T13:57:26.553Z sha 9f4490e :: needs changes before merge. :: [P1] Archive the exit status before requiring it for recovery