Skip to content

Cross-provider ranking policy beyond raw score #95

Description

@macanderson

SPEC.md §6.6 (F10, added in PR #87) states that score is provider-local and ordinal — the protocol defines no shared scale, so ranking a mixed set by raw score favours whichever provider scores most generously.

order_by_value still does exactly that. F10 permits it (some total order is required to place frames at all, and it is now documented as the host policy rather than a protocol guarantee), but "documented default" is a floor, not an answer. fold_to_edges is now public so a host can bring its own ranking, and nothing in the repo demonstrates one.

Worth providing at least one principled strategy:

  • per-provider quotas (each provider contributes its top-k, ranked internally where the ordering is meaningful)
  • round-robin interleave by within-provider rank
  • explicit host-configured trust weights per provider
  • a pluggable reranker hook

This matters most for the multi-provider fan-out that is the whole point of a host — with a single provider the question does not arise.


Definition of done

  • A ranking seam, not a second hard-coded function. A RankingStrategy trait in contextgraph-host/src/compose/ranking.rs that a host outside this crate can implement, re-exported from the crate root. Verified by an integration test (contextgraph-host/tests/cross_provider_ranking.rs) that uses only contextgraph_host::* — it compiles, so the surface is public and usable.
  • At least two concrete strategies, so the seam is proven to carry more than one. RoundRobinByRank (interleave by within-provider rank) and PerProviderQuota (each provider's top k per tier), neither needing configuration.
  • The strategy decides the evidence set, not just the placement. The ranking is applied before the budget pack in compose_for_prompt_with, so a policy changes which frames reach the prompt. Verified by starvation_shows_up_in_the_composed_prompt_under_a_real_budget.
  • A test that demonstrates the problem F10 describes. A generous-scoring provider and a conservative one over the same evidence, one budget: raw score cites only the generous provider, the interleave and the quota cite both.
  • A witness test that fails on main and passes with the change, checked the artisanal way (git stash the source, run the test, git stash pop), with both observations reported in the PR.
  • Determinism. Two runs over the same frames produce byte-identical output — no HashMap iteration anywhere in the ranking path, and every tie resolved to a total order. Verified by every_strategy_is_a_pure_function_of_the_set, a_composed_prompt_is_byte_identical_across_two_runs_of_the_same_strategy, and a twelve-provider case.
  • The default does not move. compose_for_prompt, order_by_value and fold_to_edges keep their signatures and their output; the pre-existing composition tests pass unchanged.
  • The budget bound survives every strategy. A strategy cannot select a set that exceeds global_budget, and the audit stays a total partition that explains every drop. Verified by a randomized loop over all strategies.
  • Every strategy degenerates correctly to one provider. With a single provider all three produce the same order as raw score.
  • An ADR (docs/adr/0015-cross-provider-ranking-strategies.md) recording why a seam rather than a replacement, which strategy is the default and why it stays, and how ties are broken.
  • Docs updated in the same PR: SPEC.md §6.6's "reference host, stated plainly" paragraph, docs/composing-frames-into-a-prompt.md, and CHANGELOG.md under [Unreleased]. The ADR is numbered 0015 (central allocation; 0012 and 0013 went to PRs ci(sdk): typecheck both typed SDKs, and catch a pin that drifted from its manifest #106 and feat(schema): name the schemas' identity on a branded, family-versioned URL #109).
  • Gate green: cargo fmt --check, cargo clippy -p contextgraph-host --all-targets -- -D warnings, cargo test -p contextgraph-host, and CI on the PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageAwaiting the triage agent: sizing + priority (SCR-005)

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions