Guidance for a reproducible ClawBench task-authoring comparison

[Shamratha]

Context

I am building a reproducible comparison of a native, non-proxy Codex control
against deepseek/deepseek-v4-flash-0731 in two CLI harnesses. The first two
checks create office artifacts; the final check should author and validate a
complete task in the OpenClaw/ClawBench ecosystem. Every arm receives identical
bytes and we record deterministic score, blind quality score, wall time, token
usage, cost, failures, and interventions over at least three matched runs.

The infrastructure is adapted from
https://github.com/lakshitsachdeva/deepseek-trial, but the task/data and final
results will be new.

Proposed Check 3

Have each arm author a complete ClawBench task package from one frozen seed:

• working title: Background Task Ledger Export Repair
• target: Tier 3 repo/multi_tool
• fixture: a small Python task-ledger exporter with independent normalization,
  terminal-state filtering, and completion-deduplication bugs
• expected agent trajectory: inspect multiple files, implement fixes, run the
  deterministic verifier, and report the result
• required output: canonical task YAML, asset pack, tests/verifier, and any
  manifest entry required by the current public pipeline
• scoring: deterministic completion first; an LLM judge must never rescue a
  deterministic failure

This makes the measured work task authoring—not merely replaying a contaminated
public task—and gives all three arms the same frozen OpenClaw/ClawBench source
snapshot.

Guidance requested

Could a maintainer confirm:

1. the current canonical task/asset-pack schema and directory layout;
2. the exact validation command we should treat as official;
3. whether there is an oracle/reference-solution check for newly authored tasks;
4. the OpenClaw and ClawBench versions/commits that should be pinned together;
5. whether this proposed seed is useful, or what seed would better exercise an
   end-to-end OpenClaw workflow without duplicating a public Core v1 task; and
6. whether three runs per arm is adequate for a small study, or whether your
   current methodology requires a different minimum.

I will keep the final source lock ready: false until these details are fixed
and will record the maintainer response and exact commit in the result package.

If an issue is not the right coordination channel, please point me to the
preferred OpenClaw/ClawBench contact.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 23, 2026, 7:57 PM ET / 23:57 UTC.

Summary
Keep open for one maintainer methodology decision: current main already documents the native campaign contract, pins, validation gates, and n=3-to-n=6 policy, but only a task owner can approve or replace the proposed seed after checking it against the pinned public suite.

Reproducibility: not applicable. this is a methodology and task-selection request, not a report of broken behavior.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Include redacted logs or terminal output.

Maintainer decision needed

Question	Recommendation
Should maintainers approve “Background Task Ledger Export Repair” as a distinct seed for the pinned native public suite, or select a different seed?	Approve after contamination review: Check the proposed seed against the pinned public suite, then author it as a native task package if it provides distinct coverage.

Why: The repository now answers the schema, validation, pinning, oracle-material, and repeat-count questions, but seed usefulness and non-duplication require judgment over the intended benchmark distribution.

Next step
Most requested guidance is already available in the native runbook; a maintainer must only decide whether the seed is distinct and worth adding.

Review details

Best possible solution:

Use the native task-package contract and campaign runbook, then record a maintainer-approved seed selection and contamination check against the pinned public-suite revision.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a methodology and task-selection request, not a report of broken behavior.

Is this the best way to solve the issue?

Yes, mostly: the native runbook is the narrowest existing source for the requested contract and repeat policy; only the proposed seed needs maintainer approval.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• The proposed seed has not been checked against the pinned public suite for overlap, so approving it without a maintainer contamination review could duplicate existing task coverage.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P3: This is low-urgency research-methodology coordination, not a demonstrated product failure.

Evidence reviewed

What I checked:

• Current native task package contract: The native validator requires task.toml, instruction.md, an environment Dockerfile, solution/solve.sh, and tests/test.sh for each task directory. (scripts/native_eval/tasks.py:117, 69f75c6629c4)
• Validation and plan entry point: The campaign planner validates the selected suite before creating a run index and enforces exactly ten named tasks for r0 qualification. (scripts/native_eval/plan.py:35, 884dd1bb5511)
• Reproducibility methodology: The checked-in research runbook requires pinned task and runner revisions, an r0 route qualification, three independent full-suite repetitions before audit, and six total repetitions for a research result. (.agents/skills/shellbench-research-runbook/references/runbook.md:20, 569b5c39c783)
• Current toolchain pins: The current native models source pins OpenClaw, Hermes, Codex, Claude Code, Node, and LiteLLM versions used by the campaign contract. (scripts/native_eval/models.py:93, 884dd1bb5511)

Likely related people:

• Vincent Koc: Introduced the native task validator, campaign planner, and reproducible research runbook that define the current methodology. (role: native evaluation runner author and recent area contributor; confidence: high; commits: 69f75c6629c4, 569b5c39c783; files: scripts/native_eval/tasks.py, scripts/native_eval/plan.py, .agents/skills/shellbench-research-runbook/references/runbook.md)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.