i'm using Braintrust to run judge regressions with replicated trials per fixture. in the UI i can group rows by input and sort/filter by agreement between replicates, which works fine at 20 fixtures, for instance. what i don't have is something i could track across runs or gate on, rather than something i look at.
i ran two experiments: 1 trial × 20 diffs gave me 93.75% criterion agreement. 5 trials × 20 diffs gave me 93.25%. near-identical mean, but the mean can't tell me whether the judge was stable or thrashing. i've implemented Fleiss' kappa and majority-vote accuracy myself for this.
the Scorer shape takes one output and returns one Score. kappa needs N outputs for the same input, so it's not expressible as a Scorer. trialCount added replication but i don't see anything downstream consuming it.
does this belong in autoevals as a separate offline utility over an array of Score outputs, or in the SDK's experiment-fetch/analysis layer, or is it out of scope and left to callers?
if it makes sense, happy to open a PR, either a separate package for consistency statistics or a function alongside Scorer.
i'm using Braintrust to run judge regressions with replicated trials per fixture. in the UI i can group rows by input and sort/filter by agreement between replicates, which works fine at 20 fixtures, for instance. what i don't have is something i could track across runs or gate on, rather than something i look at.
i ran two experiments: 1 trial × 20 diffs gave me 93.75% criterion agreement. 5 trials × 20 diffs gave me 93.25%. near-identical mean, but the mean can't tell me whether the judge was stable or thrashing. i've implemented Fleiss' kappa and majority-vote accuracy myself for this.
the Scorer shape takes one output and returns one Score. kappa needs N outputs for the same input, so it's not expressible as a Scorer. trialCount added replication but i don't see anything downstream consuming it.
does this belong in autoevals as a separate offline utility over an array of Score outputs, or in the SDK's experiment-fetch/analysis layer, or is it out of scope and left to callers?
if it makes sense, happy to open a PR, either a separate package for consistency statistics or a function alongside Scorer.