Proposal: map TaskDefinition/TaskRunResult/BenchmarkResult onto EvalPort's TestCase/Result/ResultSet

[adhabnr-ux]

I've been reading through clawbench/schemas.py and the trace-based scoring pipeline (completion/trajectory/behavior/judge axes, pass^k reliability, the 13-mode failure taxonomy) and think ClawBench's result data maps cleanly onto EvalPort — an open interchange format for portable eval datasets/results (JSON schema + a Python openeval SDK + a TS SDK). Wanted to check the mapping against the real field names before proposing anything, rather than show up with a guess.

Mapping, based on sdk/python/openeval/types.py and clawbench/schemas.py as they actually are today:

• TaskDefinition → EvalPort TestCase: id → id; simulated-user turns (SessionPhase.user.turns[].message via TaskDefinition.normalized_phases()) → input (list of turn strings); completion.execution_checks → code-type Graders (command + expected_exit_code as params); judge.rubric → an llm_judge Grader with params.rubric; tier/family/scenario/capabilities/etc. → metadata.

• TaskRunResult → EvalPort Result: task_id → test_case_id; transcript.assistant_text → actual_output; duration_ms → duration_ms; delivery_outcome.value == "pass" → passed; error → error. The four sub-results (completion_result, trajectory_result, behavior_result, judge_result) each become their own GraderResult (grader_id = gr_completion/gr_trajectory/gr_behavior/gr_judge, type="custom", score, passed = score >= pass_threshold) — the same pattern adapters/ragas-openeval-adapter in the EvalPort repo already uses to turn each independent metric into its own grader instead of collapsing everything into one number.

• BenchmarkResult → EvalPort ResultSet: submission_id → run_id; {model, provider} → provider; timestamp → started_at; task_results/tier_results/scenario_results → summary.

Sketch — this would live as a standalone clawbench-openeval-adapter package (same playbook as the existing ragas-openeval-adapter/autogen-openeval-adapter: works against ClawBench's public Pydantic models from the outside, nothing needs to merge into ClawBench core):

from openeval.types import TestCase, Grader, Result, GraderResult, ResultSet
from clawbench.schemas import TaskDefinition, TaskRunResult

def task_to_testcase(task: TaskDefinition) -> TestCase:
    turns = [t.message for phase in task.normalized_phases() for t in phase.user.turns]
    graders: list[Grader] = [
        Grader(
            id=f"gr_exec_{c.name}",
            type="code",
            params={"command": c.command, "expected_exit_code": c.expected_exit_code},
        )
        for c in task.completion.execution_checks
    ]
    if task.judge:
        graders.append(Grader(id="gr_judge", type="llm_judge", params={"rubric": task.judge.rubric}))
    return TestCase(
        id=task.id,
        input=turns or [""],
        graders=graders,
        metadata={
            "tier": task.tier.value,
            "family": task.family.value,
            "scenario": task.scenario.value if task.scenario else "",
        },
    )

def run_to_result(run: TaskRunResult) -> Result:
    def gr(name: str, sub, score: float) -> GraderResult:
        return GraderResult(
            grader_id=f"gr_{name}", type="custom", score=score,
            passed=score >= 0.7, reason=getattr(sub, "reason", ""),
        )
    return Result(
        test_case_id=run.task_id,
        passed=run.delivery_outcome.value == "pass",
        grader_results=[
            gr("completion", run.completion_result, run.completion_result.score),
            gr("trajectory", run.trajectory_result, run.trajectory_result.score),
            gr("behavior", run.behavior_result, run.behavior_result.score),
            gr("judge", run.judge_result, run.judge_result.score),
        ],
        actual_output=run.transcript.assistant_text,
        duration_ms=run.duration_ms,
        error={"message": run.error} if run.error else None,
    )

Happy to build this out as an adapter if it's useful. Two things I'd rather ask than assume: (1) is ResultSet.summary the right place for the tier/scenario rollups, or should those stay ClawBench-specific and only per-run Results cross over? (2) Partner Trace Spec already defines a JSONL interchange for traces — is EvalPort's Result/ResultSet meant to be complementary to that (scores/verdicts) rather than overlapping with it (raw trace), or would this proposal be redundant with where PARTNER_TRACE_SPEC.md is headed?

— Sahi, independent contributor (not affiliated with this project)

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed August 25, 2026, 6:24 PM ET / 22:24 UTC.

Summary
Keep open for a maintainer product-boundary decision. Current main supports agent-agnostic task adapters and a separate trace intake format, but does not endorse an external EvalPort result contract.

Reproducibility: not applicable. this is a request for a new interoperability contract, not a report of broken behavior.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Add expected vs actual behavior.
• Include redacted logs or terminal output.

Maintainer decision needed

Question	Recommendation
Should ShellBench endorse and version an EvalPort score-export adapter, and which task, per-run, and aggregate fields are authoritative exports versus ShellBench-specific metadata?	Endorse a scoped external adapter: Define a versioned out-of-tree contract that exports portable task and run data while retaining ShellBench reliability, provenance, and trace-specific fields as namespaced metadata.

Why: Current code establishes a canonical task view and a trace interchange, but no public external result-format contract or loss policy for ShellBench-specific semantics.

Next step
A maintainer must decide whether to own this external score-export contract before adapter work can be scoped safely.

Review details

Best possible solution:

If endorsed, publish a versioned out-of-tree adapter built on the canonical task boundary, preserve ShellBench-specific score and provenance fields as namespaced metadata, and keep raw traces in the existing JSONL interchange.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a request for a new interoperability contract, not a report of broken behavior.

Is this the best way to solve the issue?

No—not as a direct TaskDefinition mapping; if the direction is accepted, the existing CanonicalTask boundary is the narrower supported source for an out-of-tree adapter.

AGENTS.md: not found in the target repository.

Remaining risk / open question:

• Directly flattening ShellBench’s state, multi-turn, reliability, hidden-task, and provenance semantics into generic results could misrepresent official scores without an explicit export policy.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Label changes

Label justifications:

• P3: This is an optional interoperability proposal, not a report of broken ShellBench behavior.

Evidence reviewed

What I checked:

• Canonical adapter boundary: Current main explicitly separates agent-agnostic task intent from execution-specific adapters and preserves unsupported capabilities rather than flattening them. (clawbench/canonical/schema.py:20, 884dd1bb5511)
• Current score surface: Per-run results contain independent completion, trajectory, behavior, judge, transcript, efficiency, delivery, and failure fields; benchmark results separately carry reliability, provenance, task, tier, and scenario aggregates. (clawbench/schemas.py:562, 884dd1bb5511)
• Trace boundary: The partner specification defines JSONL for full prompts, transcripts, tool calls, artifacts, redaction, and provenance, so it is complementary to score exports rather than a generic ResultSet contract. (PARTNER_TRACE_SPEC.md:446, 884dd1bb5511)
• Roadmap scope: The v0.4 plan targets eventual cross-harness evaluation through canonical tasks and adapters, while explicitly not shipping a cross-harness benchmark in one release. (CLAWBENCH_V0_4_SPEC.md:13, 884dd1bb5511)
• Feature provenance: The canonical conversion layer was introduced by the adapter-canonicalization commit, and blame attributes its mapping design to scoootscooob. (clawbench/canonical/convert.py:1, 56531fbf432b)
• No current EvalPort integration: Current source and project metadata contain no EvalPort or OpenEval integration, so adopting its schema would create a new supported boundary rather than extend an existing one. (pyproject.toml:1, 884dd1bb5511)

Likely related people:

• scoootscooob: Introduced the canonical task conversion layer that is the natural boundary for any external adapter. (role: introduced canonical adapter boundary; confidence: high; commits: 56531fbf432b, 0f1b45e4674b, 4f752b617a75; files: clawbench/canonical/convert.py, clawbench/canonical/schema.py, clawbench/adapters/openclaw.py)
• Vincent Koc: Git blame attributes the partner trace specification’s purpose and JSONL interchange to this contributor. (role: introduced trace interchange documentation; confidence: medium; commits: fc86dd615523; files: PARTNER_TRACE_SPEC.md)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.