Proposal: EvalPort export for tasks/checklist/score — a portable interchange layer on top of the existing schema

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange format (TestCase/Grader/Result/ResultSet/GraderResult) for LLM eval data, plus zero-dependency Python (evalport-sdk on PyPI, import openeval) and TS SDKs. I read evaluation/score.py, evaluation/run_task.py, and CONTRIBUTING.md on main, and I think ResearchClawBench's task/checklist/score shape maps onto EvalPort cleanly enough to be worth a converter — flagging it here rather than opening a PR blind, since it'd live as a new top-level module and I'd rather confirm the shape is welcome first.

The mapping (verified against your actual field names, not the README prose):

ResearchClawBench	EvalPort
tasks/<Domain>_NNN/task_info.json["task"]	TestCase.input
task id ("Astronomy_000")	TestCase.id
task_info.json["data"][].description	TestCase.context (list of strings)
target_study/checklist.json[i] (type, content, path, keywords, weight)	one Grader per item — type="llm_judge" (matches score.py's structai.LLMAgent judge), params={"criteria": content, "keywords": keywords, "item_type": type, "image_path": path}, weight=item["weight"], description=content[:200]
_meta.json (run_id, task_id, agent_name, model, duration_seconds, timestamp)	ResultSet.run_id/suite_id, ResultSet.runner={"name": agent_name, "model": model}, ResultSet.started_at
report/report.md	Result.actual_output
_score.json["items"][i] (index, type, content, weight, score 0–100, reasoning)	one GraderResult per item — grader_id matching the Grader.id above, score=score/100 (clamped to EvalPort's required [0,1]), reason=reasoning, metadata={"raw_score_0_100": score}
_score.json["total_score"], total_weight	ResultSet.summary

One judgment call worth flagging explicitly: GraderResult.passed is a bare bool, but your rubric is a 0–100 scale where score.py's own docstring says "50 = matches paper." I'd map passed = raw_score >= 50, using your own stated semantics rather than inventing a threshold — but happy to make it configurable if that's the wrong call for how the leaderboard actually uses these scores.

Sketch (following the shape of the deepeval-openeval-adapter already in the EvalPort repo — real src/, real tests against the installed package, not a scaffold):

from openeval.types import TestCase, Grader, Result, GraderResult, ResultSet

def task_to_testcase(task_id: str, task_info: dict, checklist: list[dict]) -> TestCase:
    graders = [
        Grader(
            id=f"{task_id}_item_{i}",
            type="llm_judge",
            params={"criteria": item["content"], "keywords": item.get("keywords", []),
                    "item_type": item["type"], "image_path": item.get("path")},
            weight=item.get("weight", 1.0),
            description=item["content"][:200],
        )
        for i, item in enumerate(checklist)
    ]
    return TestCase(
        id=task_id,
        input=task_info["task"],
        graders=graders,
        context=[d.get("description", "") for d in task_info.get("data", [])],
        tags=[task_id.split("_")[0]],  # domain, e.g. "Astronomy"
    )

def score_to_resultset(meta: dict, score_data: dict, report_text: str) -> ResultSet:
    grader_results = [
        GraderResult(
            grader_id=f"{score_data['task_id']}_item_{item['index']}",
            type="llm_judge",
            score=max(0.0, min(1.0, item["score"] / 100.0)),
            passed=item["score"] >= 50,
            reason=item["reasoning"],
            metadata={"raw_score_0_100": item["score"]},
        )
        for item in score_data["items"]
    ]
    result = Result(
        test_case_id=score_data["task_id"],
        passed=score_data["total_score"] >= 50,
        grader_results=grader_results,
        actual_output=report_text,
        duration_ms=meta.get("duration_seconds", 0) * 1000,
        metadata={"total_score": score_data["total_score"], "total_weight": score_data["total_weight"]},
    )
    return ResultSet(
        version="1.0.0-rc.4",
        suite_id=score_data["task_id"],
        run_id=score_data["run_id"],
        started_at=meta["timestamp"],
        results=[result],
        runner={"name": meta.get("agent_name", "Unknown"), "model": meta.get("model", "")},
    )

If this looks worth doing, I'd send it as evaluation/openeval_export.py (or a standalone researchclawbench-openeval-adapter, matching the pattern in adapters/ — your call on which fits better given this repo focuses on the 40 base tasks and the actual submission path is the HF Space) with tests against real checklist.json/_score.json fixtures from tasks/Astronomy_000. Would rather confirm direction before writing it up as a PR — let me know if this is something you'd take, or if the leaderboard/HF pipeline makes a converter like this redundant on your end.

— Sahi, independent contributor (not affiliated with this project)