fix: accept Claude native trajectories

[vincentkoc]

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

• add claude-code to the real-trajectory harness allowlist
• assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

• coverage: 4/4
• score: 0.75
• exact passes: 3
• trajectory complete: true
• trajectory status: real for all four tasks
• infrastructure failures: 0
• exclusion reason: exploratory_subset

Tests

• python -m pytest -q passes locally (444 passed, 5 skipped)
• python -m ruff check clawbench app.py scripts tests passes locally
• live Claude Code native run aggregates as real harness events

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 21, 2026, 7:55 PM ET / 23:55 UTC.

ClawSweeper review

What this changes

The PR adds Claude Code to ShellBench’s real-trajectory harness policy and tests that all pinned native harnesses report real harness events.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for an explicit maintainer decision: current main reconstructs Claude Code traces but still classifies the harness as unsupported, and this PR changes that benchmark-evidence policy to make qualifying runs eligible for aggregation.

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is focused and consistent with the existing converter safeguards; benchmark-policy acceptance remains the merge decision.
Proof confidence	🌊 off-meta tidepool	Not applicable: This member-authored PR is exempt from the external-contributor proof gate; its body nevertheless reports an after-fix four-task native Claude Code control run with real trajectories.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: This member-authored PR is exempt from the external-contributor proof gate; its body nevertheless reports an after-fix four-task native Claude Code control run with real trajectories.
Evidence reviewed	5 items	Current-main policy excludes Claude Code: Current main pins Claude Code as a native harness but omits it from the real-trajectory allowlist, so its manifest mode is unsupported. Current main already reconstructs and validates Claude traces: The Claude Code converter retains raw event provenance, requires a successful terminal event, and records canonical-model identity before returning a real trajectory. Aggregation makes the policy consequential: Native aggregation requires both real_harness_events in the manifest and real per-task trajectories before marking a run trajectory-complete and eligible.
Findings	None	None.
Security	None	None.

Live Verification

Command: python -m pytest -q tests/test_native_eval_runner.py -k all_native_harnesses_emit_real_trajectories

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.22.0.tgz

Assertions:

• FAIL expect_output: 1 passed

How this fits together

ShellBench’s native evaluator converts each harness’s raw run output into a normalized trajectory and records its evidence mode in the run manifest. Aggregation uses that mode plus per-task trajectory checks to decide whether a run can contribute to native benchmark results.

flowchart LR
A[Claude Code native run] --> B[Stream JSON log]
B --> C[Trajectory reconstruction]
C --> D{Evidence-mode policy}
D --> E[Aggregate eligibility]
E --> F[Native benchmark results]

Loading

Decision needed

Question	Recommendation
Should qualifying Claude Code stream-json reconstructions be admitted as real harness evidence and therefore become eligible for native aggregation?	Admit validated Claude trajectories: Approve the existing reconstruction, terminal-state, and canonical-model safeguards as sufficient evidence and merge the policy update.

Why: The patch is mechanically narrow, but the allowlist is an evidence-policy boundary rather than a purely technical parser correction.

Before merge

• Resolve merge risk (P1) - Merging changes the benchmark-evidence policy: completed Claude Code runs with reconstructed stream-json traces can enter native aggregation, so maintainers must affirm that this reconstruction meets the same comparability standard as the existing accepted harnesses.
• Complete next step (P2) - A maintainer must decide whether reconstructed Claude Code stream-json traces satisfy the benchmark’s real-evidence standard; no mechanical repair is warranted.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production/test delta	production +1/-1, tests +12 across 2 files	The implementation is narrowly scoped to one evidence-policy entry and one four-harness assertion.

Merge-risk options

Maintainer options:

1. Confirm the evidence standard (recommended)
   Explicitly approve that Claude Code’s reconstructed stream-json events satisfy the native benchmark’s real-trajectory policy before merging.
2. Keep Claude Code excluded
   Pause or close this policy change if reconstructed traces should not yet affect aggregate eligibility.

Technical review

Best possible solution:

Approve the existing Claude Code reconstruction as real benchmark evidence only if its retained raw events, terminal-state check, and canonical-model gate meet the intended comparability standard; otherwise retain the conservative exclusion.

Do we have a high-confidence way to reproduce the issue?

Yes, from current source: a Claude Code run can reconstruct to a real trajectory, but current main assigns its manifest an unsupported mode, which aggregation treats as trajectory-unavailable.

Is this the best way to solve the issue?

Unclear pending maintainer policy judgment; adding Claude Code is the narrowest implementation if its reconstructed events meet the same evidence standard as the other accepted harnesses.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

• remove proof: sufficient: Current real behavior proof status is not_applicable, not sufficient.

Label justifications:

• P2: This changes native benchmark eligibility for one harness but does not indicate an active runtime outage or data-loss event.
• merge-risk: 🚨 other: The allowlist changes which completed benchmark runs can be accepted as evidence, a policy impact outside the specific compatibility, delivery, session, auth, security, availability, and automation categories.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This member-authored PR is exempt from the external-contributor proof gate; its body nevertheless reports an after-fix four-task native Claude Code control run with real trajectories.

Evidence

What I checked:

• Current-main policy excludes Claude Code: Current main pins Claude Code as a native harness but omits it from the real-trajectory allowlist, so its manifest mode is unsupported. (scripts/native_eval/models.py:102, 884dd1bb5511)
• Current main already reconstructs and validates Claude traces: The Claude Code converter retains raw event provenance, requires a successful terminal event, and records canonical-model identity before returning a real trajectory. (scripts/native_eval/harness_trajectories.py:316, 884dd1bb5511)
• Aggregation makes the policy consequential: Native aggregation requires both real_harness_events in the manifest and real per-task trajectories before marking a run trajectory-complete and eligible. (scripts/native_eval/aggregate.py:906, 884dd1bb5511)
• Existing regression coverage supports the converter: Current-main tests already exercise Claude stream conversion, successful terminal state, tool-call reconstruction, and canonical-model rejection for mixed identities. (tests/test_native_eval_runner.py:1705, 884dd1bb5511)
• Feature history: The native evaluator and its trajectory policy were introduced in the native-matrix work and subsequently stabilized by the same area contributor; this PR is a narrow follow-up to that policy. (scripts/native_eval/models.py:102, b9acd9f7a010)

Likely related people:

• vincentkoc: Introduced the native matrix evaluator, stabilized its trajectory handling, authored the current-main trace work, and authored this focused policy change. (role: native-evaluation feature author and recent area contributor; confidence: high; commits: 69f75c6629c4, b9acd9f7a010, 884dd1bb5511; files: scripts/native_eval/models.py, scripts/native_eval/harness_trajectories.py, scripts/native_eval/aggregate.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Confirm that Claude Code stream-json reconstruction meets the intended real-trajectory comparability standard.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (64 earlier review cycles; latest 8 shown)

• reviewed 2026-08-09T12:41:32.200Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T15:19:11.621Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T19:45:59.678Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T22:01:21.887Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-11T23:10:51.821Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-12T01:22:50.827Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T10:11:27.098Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T11:15:30.965Z sha ac1c974 :: needs maintainer review before merge. :: none