Summary
A routing profile's optimize.latency weight is documented as a tuning knob that shifts candidate
selection toward low-latency providers (the dashboard editor exposes it as a "scoring weight" and the
default profile weights make it the most-weighted slot at latency: 0.55), but the routing
evaluator never scores latency as an independent objective. The value is normalized into the weight
sum and then leaks out only as the residual priorityWeight (an indirect, unintentional effect), while
real measured latency is folded into the health dimension instead. The selection step is a
deterministic argmax over the scored candidates, so with only latency-relevant evidence the winner
is chosen purely by configured declaration order, independent of any actual latency measurement.
Raising optimize.latency therefore does not make the router prefer faster providers; it mainly
increases the influence of the configured-candidate-priority term, which is not what an operator
setting this knob expects.
Why it is a real issue (file:line evidence)
src/routing/profile.ts:26-31 — DEFAULT_PROFILE_WEIGHTS = { latency: 0.55, health: 0.25, cost: 0.10, quota: 0.10 }.
Latency is the single most-weighted slot, yet it has no score term (see below) — the largest
weight is the knob doing something unrelated (configured priority).src/routing/profile.ts:504-521 — normalizeRoutingProfile reads raw.optimize, sums all four
weights and normalizes (each / safeSum), and stores optimize.latency as a normalized weight only.
It never reads a latency score.src/routing/evaluator.ts:397-448 — the candidate score is built only from
profile.optimize.health/quota/cost plus priorityWeight * configuredPriorityScore(...); there is
no latency score term and grep -c "latency" src/routing/evaluator.ts returns 0. The remaining
budget is const priorityWeight = Math.max(0, 1 - spentHealth - spentQuota - spentCost);
(evaluator.ts:404). Because the normalized weights sum to 1, priorityWeight === optimize.latency
only when health, quota, and cost each carry concrete evidence (each spent* equals its
weight); otherwise it is the residual budget and can exceed optimize.latency. So latency is not
ignored wholesale, but its only role is to become the weight of the configured-priority term
(configuredPriorityScore(index, total), evaluator.ts:183-185), never a latency term.src/routing/evaluator.ts:444-447 — selection is a deterministic argmax: if (evaluated.eligible && score.total > bestScore). No shuffle, no weighted-random, no latency re-selection; ties favor the earlier index (strict >). The dispatch path then routes the selected candidate directly (src/router.ts:552-564). So "ranked purely by configured order" is exactly what happens for the priority-only case.- Latency is captured only inside the health score:
- Real path:
src/routing/compatibility/assemble.ts:52-72 → policyCandidateHealthEvidence →
src/routing/health.ts:184-204, where the persisted duration_ms samples are folded into a p50
and carried as RouteHealthEvidence.recentLatencyMs (health.ts:323-325).
src/routing/health.ts:386-400 — latencyScore (from recentLatencyMs) is folded into the
healthScore composite and (on the routing side, evaluator.ts:407-410) scaled by the health
weight profile.optimize.health, not by profile.optimize.latency.- Net influence: under default weights, raw latency reaches the final score only as
optimize.health (0.25) × HEALTH.LATENCY_WEIGHT (0.20) = 0.05, while the "latency" knob's 0.55 goes
entirely to configured priority. The most-weighted knob has ~11× less real latency influence than
an operator would infer.
src/routing/trace.ts:133 — RouteScoreEvidence.components already declares a latent latency?: number slot, but the evaluator never fills it (no components.latency anywhere). The schema was
designed for a latency score component that the implementation omits.- Existing test pins the current behavior —
tests/routing-profile.test.ts:317-335
("dry-run evaluator: deterministic priority picks the earlier candidate") asserts
components: { configuredPriority: 1, health: 0.3, quota: 0.3, cost: 0.3 } (total 0.685) with
no latency component, and that the earlier candidate wins. This is not an accidental gap: the
behavior is intentional and test-locked, which strengthens the case that the knob is mis-documented
rather than merely unimplemented.
Reproduction / demonstration
Repro probe: repro/routing_latency_probe.mjs (a faithful self-contained mirror of the scoring block
evaluator.ts:397-448 + healthScore latency math health.ts:386-400, using the real constants
DEFAULT_PROFILE_WEIGHTS and HEALTH: { LATENCY_WEIGHT: 0.20, LATENCY_TARGET_MS: 60_000, ... }).
Run: node repro/routing_latency_probe.mjs.
=== Scenario A: optimize { latency:1, health:0, cost:0, quota:0 } ===
config order: [fast, slow] | fast p50=800ms, slow p50=40s
fast-provider: priorityWeight=1 total=1.000 (configuredPriority=1)
slow-provider: priorityWeight=1 total=0.500 (configuredPriority=0.5)
SELECTED: fast-provider (p50s 800,40000 never referenced in scoring)
=== Scenario A2: same weights, config order REVERSED [slow, fast] ===
slow-provider: priorityWeight=1 total=1.000
fast-provider: priorityWeight=1 total=0.500
SELECTED: slow-provider <- pure configured order, NOT latency
=== Scenario B (default weights): latency ~0.55, health ~0.25 ... ===
net latency influence = optimize.health * HEALTH.LATENCY_WEIGHT = 0.0500
vs priorityWeight 0.55 (the "latency" knob -> CONFIGURED PRIORITY)
=== Identity caveat (numeric) ===
optimize { latency:.4, health:.3, cost:.2, quota:.1 }, only health evidence -> priorityWeight=0.7
(NOT 0.4) -- identity priorityWeight===latency holds only when health/quota/cost all have evidence
Takeaways:
- With
optimize: { latency: 1, ... }, priorityWeight = 1 - 0 - 0 - 0 = 1 and the score reduces to
1 * priorityScore, an argmax on configured order — the faster provider is chosen only if listed
first, and reversing the config order picks the slower one. No measured latency is consulted.
- With
optimize: { latency: 0, health: 1, ... }, priorityWeight = 0 and only the latency-informed
healthScore term is used — this works, but only because latency is embedded in health, and it
costs the configured-priority signal entirely.
- The exact identity
priorityWeight === optimize.latency (from the original draft) is not the
general case: it equals 1 - spentHealth - spentQuota - spentCost; it matches optimize.latency
only when all three other dimensions carry concrete evidence.
Proposed direction
Either (a) implement a real latency score term in evaluator.ts scaled by profile.optimize.latency
(reusing the p50 latency already carried in RouteHealthEvidence.recentLatencyMs, and filling the
latent components.latency slot in trace.ts:133), keeping the health composite independent and
updating the pinned test at tests/routing-profile.test.ts:317-335; or (b) drop latency from the
optimize block and document that latency is already folded into the health dimension, so the
confusing knob cannot silently mutate priority weight. Existing tests should pin the chosen behavior.
Checks
References (web)
- opencodex Routing Profile Editor guide — documents the
optimize block with
"latency": 0.55 as the dominant weight in a "fast" profile example, with no note that latency is
not independently scored: https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/routing-profile-editor.md
- OpenRouter Provider Routing — latency is a first-class routing dimension
(sort: "latency" prioritizes lowest latency; preferred_max_latency filters by p50/p90/p99 over a
rolling window): https://openrouter.ai/docs/guides/routing/provider-selection
- LiteLLM Router — native
latency-based-routing strategy and lowest_latency_buffer, i.e. latency
as a real scoring criterion: https://docs.litellm.ai/docs/routing
- opencodex Model Routing guide — documents the resolver as order-based ("the first match wins"), with
no latency-aware selection: https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/model-routing.md
Summary
A routing profile's
optimize.latencyweight is documented as a tuning knob that shifts candidateselection toward low-latency providers (the dashboard editor exposes it as a "scoring weight" and the
default profile weights make it the most-weighted slot at
latency: 0.55), but the routingevaluator never scores latency as an independent objective. The value is normalized into the weight
sum and then leaks out only as the residual
priorityWeight(an indirect, unintentional effect), whilereal measured latency is folded into the health dimension instead. The selection step is a
deterministic argmax over the scored candidates, so with only latency-relevant evidence the winner
is chosen purely by configured declaration order, independent of any actual latency measurement.
Raising
optimize.latencytherefore does not make the router prefer faster providers; it mainlyincreases the influence of the configured-candidate-priority term, which is not what an operator
setting this knob expects.
Why it is a real issue (file:line evidence)
src/routing/profile.ts:26-31—DEFAULT_PROFILE_WEIGHTS = { latency: 0.55, health: 0.25, cost: 0.10, quota: 0.10 }.Latency is the single most-weighted slot, yet it has no score term (see below) — the largest
weight is the knob doing something unrelated (configured priority).
src/routing/profile.ts:504-521—normalizeRoutingProfilereadsraw.optimize, sums all fourweights and normalizes (each
/ safeSum), and storesoptimize.latencyas a normalized weight only.It never reads a latency score.
src/routing/evaluator.ts:397-448— the candidate score is built only fromprofile.optimize.health/quota/costpluspriorityWeight * configuredPriorityScore(...); there isno latency score term and
grep -c "latency" src/routing/evaluator.tsreturns0. The remainingbudget is
const priorityWeight = Math.max(0, 1 - spentHealth - spentQuota - spentCost);(
evaluator.ts:404). Because the normalized weights sum to 1,priorityWeight === optimize.latencyonly when
health,quota, andcosteach carry concrete evidence (eachspent*equals itsweight); otherwise it is the residual budget and can exceed
optimize.latency. Solatencyis notignored wholesale, but its only role is to become the weight of the configured-priority term
(
configuredPriorityScore(index, total),evaluator.ts:183-185), never a latency term.src/routing/evaluator.ts:444-447— selection is a deterministic argmax:if (evaluated.eligible && score.total > bestScore). No shuffle, no weighted-random, no latency re-selection; ties favor the earlier index (strict>). The dispatch path then routes the selected candidate directly (src/router.ts:552-564). So "ranked purely by configured order" is exactly what happens for the priority-only case.src/routing/compatibility/assemble.ts:52-72→policyCandidateHealthEvidence→src/routing/health.ts:184-204, where the persistedduration_mssamples are folded into a p50and carried as
RouteHealthEvidence.recentLatencyMs(health.ts:323-325).src/routing/health.ts:386-400—latencyScore(fromrecentLatencyMs) is folded into thehealthScorecomposite and (on the routing side,evaluator.ts:407-410) scaled by the healthweight
profile.optimize.health, not byprofile.optimize.latency.optimize.health (0.25) × HEALTH.LATENCY_WEIGHT (0.20) = 0.05, while the "latency" knob's 0.55 goesentirely to configured priority. The most-weighted knob has ~11× less real latency influence than
an operator would infer.
src/routing/trace.ts:133—RouteScoreEvidence.componentsalready declares a latentlatency?: numberslot, but the evaluator never fills it (nocomponents.latencyanywhere). The schema wasdesigned for a latency score component that the implementation omits.
tests/routing-profile.test.ts:317-335(
"dry-run evaluator: deterministic priority picks the earlier candidate") assertscomponents: { configuredPriority: 1, health: 0.3, quota: 0.3, cost: 0.3 }(total0.685) withno
latencycomponent, and that the earlier candidate wins. This is not an accidental gap: thebehavior is intentional and test-locked, which strengthens the case that the knob is mis-documented
rather than merely unimplemented.
Reproduction / demonstration
Repro probe:
repro/routing_latency_probe.mjs(a faithful self-contained mirror of the scoring blockevaluator.ts:397-448+healthScorelatency mathhealth.ts:386-400, using the real constantsDEFAULT_PROFILE_WEIGHTSandHEALTH: { LATENCY_WEIGHT: 0.20, LATENCY_TARGET_MS: 60_000, ... }).Run:
node repro/routing_latency_probe.mjs.Takeaways:
optimize: { latency: 1, ... },priorityWeight = 1 - 0 - 0 - 0 = 1and the score reduces to1 * priorityScore, an argmax on configured order — the faster provider is chosen only if listedfirst, and reversing the config order picks the slower one. No measured latency is consulted.
optimize: { latency: 0, health: 1, ... },priorityWeight = 0and only the latency-informedhealthScoreterm is used — this works, but only because latency is embedded in health, and itcosts the configured-priority signal entirely.
priorityWeight === optimize.latency(from the original draft) is not thegeneral case: it equals
1 - spentHealth - spentQuota - spentCost; it matchesoptimize.latencyonly when all three other dimensions carry concrete evidence.
Proposed direction
Either (a) implement a real latency score term in
evaluator.tsscaled byprofile.optimize.latency(reusing the p50 latency already carried in
RouteHealthEvidence.recentLatencyMs, and filling thelatent
components.latencyslot intrace.ts:133), keeping the health composite independent andupdating the pinned test at
tests/routing-profile.test.ts:317-335; or (b) droplatencyfrom theoptimizeblock and document that latency is already folded into the health dimension, so theconfusing knob cannot silently mutate priority weight. Existing tests should pin the chosen behavior.
Checks
src/routing/evaluator.ts:397-448confirms no latency score term,priorityWeight= residual,and argmax selection (
>, earlier-index tie-break).src/routing/profile.ts:26-31,504-521confirmslatency: 0.55default and thatlatencyparticipates only in weight normalization.
src/routing/health.ts:184-204,323-325,386-400confirms latency is only scored through thehealth dimension (scaled by
optimize.health, weight 0.20 in the composite).tests/routing-profile.test.ts:317-335pins the no-latency-component behavior (intentional).repro/routing_latency_probe.mjsreproduces the bias numerically (runnode ...).optimize.latencyknob (searched title scope).References (web)
optimizeblock with"latency": 0.55as the dominant weight in a"fast"profile example, with no note that latency isnot independently scored:
https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/routing-profile-editor.md(
sort: "latency"prioritizes lowest latency;preferred_max_latencyfilters by p50/p90/p99 over arolling window):
https://openrouter.ai/docs/guides/routing/provider-selectionlatency-based-routingstrategy andlowest_latency_buffer, i.e. latencyas a real scoring criterion:
https://docs.litellm.ai/docs/routingno latency-aware selection:
https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/model-routing.md