(Extends #10's relative metric into the actual scoring used by many optimization contests.)
Many optimization contests don't score on absolute value but on a relative per-case formula — e.g. for minimization score_i = round(K · best_i / your_i) summed over cases, where best_i is the best result across all competitors. The key properties: each case is worth the same maximum, and a fixed % improvement on an already-good case earns the same as on a weak one — so the plain absolute sum is a misleading optimization target.
You can't see competitors' global best_i, so it must be estimated from a maintained reference. Proposed:
- Persistent best-known-per-case store — updated automatically as you run configs/the suite over time (and optionally seeded from / merged with a user-provided best-known file). This is the reference
best_i.
- Relative-score estimate report — in the contest's units (
Σ round(K · best_i / your_i)), plus % of theoretical max and an estimated standing, computed against the maintained reference.
- Per-case relative contribution — rank cases by how many relative points they cost (
1 − best_i/your_i), which surfaces the cases worth optimizing — very different from the absolute-gap ranking, since absolute gaps over-weight large-magnitude cases.
This turns the suite runner (#10) from "absolute aggregate" into a faithful estimate of relative standing, which is what actually determines rank in relative-scored contests, and lets the optimizer (#9 / Optuna) target the right objective.
(Extends #10's relative metric into the actual scoring used by many optimization contests.)
Many optimization contests don't score on absolute value but on a relative per-case formula — e.g. for minimization
score_i = round(K · best_i / your_i)summed over cases, wherebest_iis the best result across all competitors. The key properties: each case is worth the same maximum, and a fixed % improvement on an already-good case earns the same as on a weak one — so the plain absolute sum is a misleading optimization target.You can't see competitors' global
best_i, so it must be estimated from a maintained reference. Proposed:best_i.Σ round(K · best_i / your_i)), plus % of theoretical max and an estimated standing, computed against the maintained reference.1 − best_i/your_i), which surfaces the cases worth optimizing — very different from the absolute-gap ranking, since absolute gaps over-weight large-magnitude cases.This turns the suite runner (#10) from "absolute aggregate" into a faithful estimate of relative standing, which is what actually determines rank in relative-scored contests, and lets the optimizer (#9 / Optuna) target the right objective.