Proposal: ruby_llm-contract-openeval-adapter — export Dataset/Report to the EvalPort JSON format

[adhabnr-ux]

What this is

EvalPort is a small open JSON interchange spec for portable LLM eval data — two document types matter here: EvalSuite (test cases + graders) and ResultSet (per-test-case results + grader results + summary stats). Full field-level spec: spec/SPEC.md. There are 36 adapters already (DeepEval, Promptfoo, Ragas, Inspect AI, LangSmith, Braintrust, MLflow, OpenAI Evals, …) — all Python so far. ruby_llm-contract would be the first Ruby one, and it's a genuinely clean fit because the internal objects already line up almost 1:1 with EvalPort's shape:

ruby_llm-contract	EvalPort
RubyLLM::Contract::Eval::Dataset / Case (name, input, expected, expected_traits, evaluator, step_expectations)	EvalSuite.test_cases[]
RubyLLM::Contract::Eval::CaseResult (name, input, output, expected, score, passed?, details, duration_ms, cost, attempts)	ResultSet.results[].grader_results[]
RubyLLM::Contract::Eval::Report (dataset_name, results, score, pass_rate, passed, failed, skipped)	ResultSet + ResultSet.summary
RubyLLM::Contract::RakeTask::SuiteGate::Verdict (passed?, abort_reason, suite_cost) — the object your CI gate (suite_gate.rb) already produces	maps cleanly into ResultSet.metadata["openeval.ci_gate"]

In other words: Dataset/Case is already a native "test suite of test cases" object, and Report/CaseResult is already a native "run result" object with a pass/fail verdict — this isn't a bolt-on, it's a serialization of data you're already computing.

Proposal

A small standalone gem, ruby_llm-contract-openeval-adapter, with two pure functions and no runtime dependency on your gem beyond duck-typing the objects above:

# lib/ruby_llm/contract/openeval_adapter.rb
module RubyLLM
  module Contract
    module OpenEvalAdapter
      module_function

      # Dataset -> EvalSuite
      def suite_from_dataset(dataset, id:, graders: [{ "id" => "gr_case_match", "type" => "exact_match" }])
        {
          "$schema" => "https://evalport.org/schema/suite.json",
          "version" => "1.0.0",
          "id" => id,
          "name" => dataset.name,
          "graders" => graders,
          "test_cases" => dataset.cases.map { |c|
            {
              "id" => slug(c.name),
              "input" => c.input,
              "expected_output" => c.expected,
              "graders" => graders.map { |g| g["id"] },
              "metadata" => {
                "openeval.ruby_llm_contract" => {
                  "expected_traits" => c.expected_traits,
                  "has_evaluator" => !c.evaluator.nil?,
                  "step_expectations" => c.step_expectations
                }.compact
              }
            }
          }
        }
      end

      # Report (+ optional SuiteGate::Verdict) -> ResultSet
      def result_set_from_report(report, suite_id:, run_id:, started_at:, verdict: nil)
        {
          "$schema" => "https://evalport.org/schema/resultset.json",
          "version" => "1.0.0",
          "suite_id" => suite_id,
          "run_id" => run_id,
          "started_at" => started_at,
          "runner" => { "name" => "ruby_llm-contract", "version" => RubyLLM::Contract::VERSION },
          "results" => report.results.map { |cr|
            {
              "test_case_id" => slug(cr.name),
              "actual_output" => cr.output,
              "passed" => cr.passed?,
              "duration_ms" => cr.duration_ms,
              "grader_results" => [{
                "grader_id" => "gr_case_match",
                "type" => "exact_match",
                "score" => cr.score,
                "passed" => cr.passed?,
                "reason" => cr.label
              }],
              "metadata" => { "openeval.cost" => cr.cost, "openeval.attempts" => cr.attempts }.compact
            }
          },
          "summary" => {
            "total" => report.results.length,
            "passed" => report.passed,
            "failed" => report.failed,
            "skipped" => report.skipped,
            "pass_rate" => report.pass_rate_ratio,
            "avg_score" => report.score
          },
          "metadata" => verdict && {
            "openeval.ci_gate" => {
              "passed" => verdict.passed?,
              "abort_reason" => verdict.abort_reason,
              "suite_cost" => verdict.suite_cost
            }
          }
        }.compact
      end

      def slug(name) = name.to_s.downcase.gsub(/[^a-z0-9]+/, "_")
    end
  end
end

Everything project-specific (expected_traits, evaluator procs, step_expectations, cost/attempts) rides in metadata["openeval.*"], which the spec explicitly reserves for exactly this — nothing gets silently dropped, and nothing gets forced into a spec field it doesn't belong in.

Why file this as an issue rather than just opening a gem

Wanted your read on two judgment calls before building against a moving target:

1. test_case_id stability — Case#name is your uniqueness key (validate_unique_case_name!), so slugging it is a reasonable id, but it means a case rename breaks ResultSet continuity across runs. Worth a stable id: option on Dataset#add_case at some point, or is name-as-id acceptable for a first cut?
2. evaluator/proc-based cases — a Case with a Proc evaluator has no serializable "expected" value for the EvalSuite side. Current sketch just flags has_evaluator: true in metadata and otherwise no-ops it (the case still round-trips, just without a portable grading rule). Seems like the honest answer given evals-as-code isn't universal across the ecosystem, but flagging it in case you see a cleaner option given how evaluator is used elsewhere in the codebase.

If this looks reasonable I'll build it as its own gem (own gemspec, own CI, depending on ruby_llm-contract only for the object shapes it reads) and link it back here. No pressure on timeline — just wanted to check the mapping made sense to you before spending time on it, since you know the edges of Dataset/Report/SuiteGate far better than I do from the outside.