EvalPort <-> Together Evaluations interop: converters for input_data_file_path / EvaluationJob.results

[adhabnr-ux]

Hi maintainers,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one tool. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

I noticed src/together/types/evaluation.py already models a full evaluation workflow (EvaluationType.CLASSIFY/SCORE/COMPARE, ClassifyParameters/ScoreParameters/CompareParameters, all taking an input_data_file_path JSONL file, and EvaluationJob with results/status). That's a very natural place for a two-way bridge to a portable eval format — related precedent: openai/openai-python#3619 (openai/openai-python#3619, still open) is adding native to_openeval()/from_openeval() conversion for the same reason on OpenAI's SDK.

Two small converters would cover it:

# sketch, not a PR
def to_together_eval_file(suite: dict) -> list[dict]:
    """EvalPort suite.json test_cases -> rows for Together's input_data_file_path JSONL."""
    return [
        {"input": tc["input"], "reference": tc.get("expected_output")}
        for tc in suite["test_cases"]
    ]

def from_evaluation_job(job: EvaluationJob, *, suite_id: str) -> dict:
    """Together EvaluationJob.results -> an EvalPort ResultSet."""
    return {
        "suite_id": suite_id,
        "run_id": job.workflow_id,
        "results": job.results or {},
    }

EvalPort's suite/testcase shape: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json, and the ResultSet shape it would map to: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json

No pressure if this isn't a priority — just wanted to flag it given how closely evaluation.py already lines up with this. Happy to sketch a fuller PR if there's interest.