Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ reimplementing them:
multi-channel score fusion (`weighted_convex_fuse` in
`reconstruct.py`) and the buyer-facing Rankings port
(`rankweave_client.py`) -- never invent a fused score or a theta.
Rankings disclose Cormack RRF channel contributions from owned
rank lists (ADR 0167).
- [TEPP](https://github.com/ContextualWisdomLab/TEPP)'s published wire
contract for calibrated measurement (`tepp_client.py`) -- never
reimplement TEPP's model here.
Expand Down
5 changes: 4 additions & 1 deletion ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,10 @@ flowchart LR
`rankweave_client.py`'s default transport raises
`RankWeaveNotAvailable`. `GET /api/rankings` then returns
`rankweave_not_available` and an empty ranking list. Hidden posts
are omitted from every channel. See ADR 0024.
are omitted from every channel. Accepted hits include
`channel_evidence` computed from owned temporal/lexical ranks
(Cormack weighted RRF contribution); RankWeave extra fields are
ignored and no theta is invented. See ADR 0024 and ADR 0167.

## Standards and citations

Expand Down
11 changes: 11 additions & 0 deletions CHANGELOG.d/2.12.17-rankweave-ranking-channel-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# 2.12.17 — RankWeave ranking channel evidence

Rankings now shows why a fused hit landed.

- `GET /api/rankings` attaches owned-channel evidence: 1-based
newest-first and title-overlap ranks plus Cormack RRF contribution
`weight / (η + rank)` with η = 60.
- A channel the post is missing from is omitted. RankWeave extra
fields are ignored. No fused score or theta is invented.
- The Rankings list discloses that evidence under each hit. Copy
states this is not a calibrated score. Click still opens the post.
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ All notable changes to this project are documented here. Format follows
environment, so local OIDC and synthetic-data workflows resolve the same
pinned dependencies as CI.

## [2.12.17] - 2026-08-24

### Added

- Rankings now names the newest-first and title-overlap ranks that
RankWeave fused, with each channel's Cormack RRF contribution.
Missing channels stay omitted. This is ranking evidence, not a
calibrated score. Click still opens that post.

## [2.12.16] - 2026-08-24

### Added
Expand Down
7 changes: 4 additions & 3 deletions backend/app/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -3123,11 +3123,12 @@ async def read_rankings(
account: CurrentAccount = Depends(get_current_account),
pool: asyncpg.Pool = Depends(get_pool),
) -> dict[str, Any]:
"""RankWeave fusion of ABAC-visible posts (ADR 0024).
"""RankWeave fusion of ABAC-visible posts (ADR 0024 / ADR 0167).

Hidden posts are omitted from every channel. Never invents a fused
score or a theta. Fail-closed when RankWeave is disabled or the
library is missing.
score or a theta. Channel evidence is computed from owned rank
lists. Fail-closed when RankWeave is disabled or the library is
missing.
"""
_require_post_read(account)
async with pool.acquire() as conn:
Expand Down
5 changes: 5 additions & 0 deletions docs/adr/0024-rankweave-fusion-fail-closed.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,11 @@ tables, and does not bind the demo IdP to production Keyverse.
4. After login, Rankings sits above Calendar. Unavailable copy is
**Rankings · RankWeave not available**. An accepted hit lists the
title; click opens that `source_post`.
5. Accepted hits also disclose owned-channel evidence (ADR 0167):
1-based `channel_rank` and Cormack contribution
`weight / (η + rank)` for each channel the post actually appears
in. Missing channels are omitted. RankWeave extra fields are
ignored. Copy states this is not a calibrated score.

## Consequences

Expand Down
67 changes: 67 additions & 0 deletions docs/adr/0167-rankweave-ranking-channel-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# ADR 0167 — Disclose RankWeave ranking channel evidence

**Decision status:** Accepted
**Date:** 2026-08-23

## Context

ADR 0024 already fuses ABAC-visible posts through RankWeave weighted
reciprocal-rank fusion (Cormack et al., 2009, η = 60; Samuel et al.,
2025, unequal weights). `GET /api/rankings` returned only `post_id`,
`post_title`, and 1-based `fused_rank`. A reader could open the hit
but could not see which owned channel ranked it, or how much that
rank contributed.

Event Lineage channel evidence (ADR 0124 on #387) explains
reconstructed parent→child edges from persisted convex-fusion
scores. Rankings is a different surface: two rank-only channels
(`temporal`, `lexical`) fused in-process at GET time. There is no
persisted ranking table and no TEPP theta. RankWeave extra fields
must not be trusted; a missing channel stays missing.

This ADR does not replace reconstruction fusion, leftover pairs,
TEPP receipts, Allen interval labels, or Event Lineage isolation
reasons.

## Decision

1. Compute ranking channel evidence from LineageWeave-owned rank
lists, never from RankWeave payload extras. For each fused hit
and each channel with a positive weight, take the 1-based rank
of that `post_id` in the ordered id list. Skip a channel the
post is absent from. Do not invent a rank.
2. Contribution is Cormack weighted RRF:
`weight / (η + rank)` with η = 60. Sort by contribution
descending, then `signal_code`. `rank` on the evidence row is
that 1-based evidence order. `channel_rank` is the 1-based
position in that channel.
3. Labels: `temporal` = **Newest first**, `lexical` = **Title
overlap**. The payload never includes a fused score or a theta.
4. Rankings lists the evidence under each accepted hit as an
accessible sibling list, not hover-only. Copy states that
RankWeave fused newest-first and title-overlap ranks, and that
this is not a calibrated score. Click still opens that post.
5. Unavailable RankWeave stays empty (`rankweave_not_available`).
Hidden posts remain omitted from every channel.

## Consequences

Buyers can see why a Rankings hit landed without treating RRF as
measurement. A later channel or weight change recomputes evidence
on the next GET; nothing is persisted. Event Lineage evidence
remains the reconstruct/convex path.

## References

Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal
rank fusion outperforms Condorcet and individual rank learning
methods. In *Proceedings of the 32nd international ACM SIGIR
conference on Research and development in information retrieval*
(pp. 758–759). ACM. https://doi.org/10.1145/1571941.1572114

Samuel, D., MacAvaney, S., Yates, A., Zhang, E., Zhang, S.,
Macdonald, C., & Ounis, I. (2025). *Weighted reciprocal rank fusion
for multi-channel retrieval* [Preprint].

ADR 0024 (RankWeave fusion fail-closed)
ADR 0124 (Event Lineage channel evidence; separate surface)
2 changes: 1 addition & 1 deletion docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ decision from them.
| Supporting document | Normative ADR |
|---|---|
| [`product-technical-gap-baseline.md`](../product-technical-gap-baseline.md) | Product/technical traceability projection across the ADR set; ADRs remain normative |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md) |
| [`lineage-bi-research-notes.md`](../lineage-bi-research-notes.md) | [0084](0084-lineage-research-grounding.md), [0062](0062-semantic-unit-embedding.md), [0064](0064-lineage-evidence-and-tree-assembly.md), [0024](0024-rankweave-fusion-fail-closed.md), [0167](0167-rankweave-ranking-channel-evidence.md) |
| [`PROV_O_IMPLEMENTATION.md`](../PROV_O_IMPLEMENTATION.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`PROV_O_IMPLEMENTATION_MATRIX.md`](../PROV_O_IMPLEMENTATION_MATRIX.md) | [0065](0065-prov-o-provenance-boundary.md) |
| [`ONTOLOGY_NAMESPACE_INVENTORY.md`](../doctoring/ONTOLOGY_NAMESPACE_INVENTORY.md) | [0157](0157-public-ontology-namespace-identity.md) |
Expand Down
2 changes: 1 addition & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "frontend",
"private": true,
"version": "2.12.16",
"version": "2.12.17",
"type": "module",
"scripts": {
"dev": "vite",
Expand Down
21 changes: 21 additions & 0 deletions frontend/src/App.css
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,27 @@
gap: 0.5rem;
}

.ranking-hit {
flex-direction: column;
align-items: stretch;
justify-content: flex-start;
}

.ranking-channel-evidence-copy {
margin: 0 0 0.75rem;
}

.ranking-channel-evidence {
list-style: none;
margin: 0;
padding: 0 0 0 0.5rem;
font-size: 0.85rem;
}

.ranking-channel-evidence li {
margin: 0.15rem 0;
}

.ticket-title {
flex: 1;
}
Expand Down
55 changes: 55 additions & 0 deletions frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,14 @@ describe("App, authenticated", () => {
post_id: string;
post_title: string;
fused_rank: number;
channel_evidence?: {
signal_code: string;
signal_label: string;
channel_rank: number;
weight: number;
contribution: number;
rank: number;
}[];
}[];
};
chatUnavailable?: boolean;
Expand Down Expand Up @@ -2592,11 +2600,47 @@ describe("App, authenticated", () => {
post_id: "post-1",
post_title: "Public post",
fused_rank: 1,
channel_evidence: [
{
signal_code: "lexical",
signal_label: "Title overlap",
channel_rank: 2,
weight: 0.75,
contribution: 0.75 / 62,
rank: 1,
},
{
signal_code: "temporal",
signal_label: "Newest first",
channel_rank: 2,
weight: 0.25,
contribution: 0.25 / 62,
rank: 2,
},
],
},
{
post_id: "post-2",
post_title: "Pricing renegotiation: revised quote sent",
fused_rank: 2,
channel_evidence: [
{
signal_code: "lexical",
signal_label: "Title overlap",
channel_rank: 1,
weight: 0.75,
contribution: 0.75 / 61,
rank: 1,
},
{
signal_code: "temporal",
signal_label: "Newest first",
channel_rank: 1,
weight: 0.25,
contribution: 0.25 / 61,
rank: 2,
},
],
},
],
},
Expand All @@ -2609,6 +2653,17 @@ describe("App, authenticated", () => {
expect(rankingButton).toHaveTextContent("Public post");
expect(rankingButton).toHaveTextContent("Rankings · rankweave");
expect(rankingButton).toHaveTextContent("rank 1");
expect(
screen.getByText(
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
),
).toBeInTheDocument();
expect(
screen.getByRole("list", { name: "Ranking evidence for Public post" }),
).toHaveTextContent("Title overlap rank 2, contribution 0.012097");
expect(
screen.getByRole("list", { name: "Ranking evidence for Public post" }),
).toHaveTextContent("Newest first rank 2, contribution 0.004032");
expect(screen.queryByRole("button", { name: /open ranking: private parent/i })).not.toBeInTheDocument();

await userEvent.click(rankingButton);
Expand Down
69 changes: 48 additions & 21 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -3058,6 +3058,10 @@ function AnalysisRunsPanel({
);
}

function formatRankingContribution(value: number): string {
return value.toFixed(6);
}
Comment thread
seonghobae marked this conversation as resolved.

function RankingsPanel({
accessToken,
onSelectPost,
Expand All @@ -3076,9 +3080,9 @@ function RankingsPanel({
}, [accessToken]);

return (
<section className="popup-section lineage-home" aria-label="Rankings">
<section className="popup-section lineage-home" aria-label={t("Rankings")}>
<div className="lineage-home-header">
<h2>Rankings</h2>
<h2>{t("Rankings")}</h2>
{ranking && (
<span className="post-badge">
{ranking.status === "accepted"
Expand All @@ -3088,30 +3092,53 @@ function RankingsPanel({
)}
</div>
{error && <p className="error">{error}</p>}
{ranking === null && !error && <p>Loading rankings...</p>}
{ranking === null && !error && <p>{t("Loading rankings...")}</p>}
{ranking && ranking.status === "unavailable" && (
<p className="popup-placeholder">Rankings · RankWeave not available</p>
<p className="popup-placeholder">{t("Rankings · RankWeave not available")}</p>
)}
{ranking && ranking.status === "accepted" && ranking.rankings.length === 0 && (
<p className="popup-placeholder">No fused rankings from RankWeave.</p>
<p className="popup-placeholder">{t("No fused rankings from RankWeave.")}</p>
)}
{ranking && ranking.rankings.length > 0 && (
<ul className="ticket-list" aria-label="Fused rankings">
{ranking.rankings.map((hit) => (
<li key={hit.post_id} className="ticket-list-item">
<button
className="post-list-item"
aria-label={`Open ranking: ${hit.post_title}`}
onClick={() => onSelectPost(hit.post_id)}
>
<span className="ticket-title">{hit.post_title}</span>
<span className="post-badge">Rankings · rankweave</span>
<span className="post-badge">rank {hit.fused_rank}</span>
</button>
</li>
))}
</ul>
)}
<>
<p className="ranking-channel-evidence-copy">
{t(
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
)}
</p>
<ul className="ticket-list" aria-label={t("Fused rankings")}>
{ranking.rankings.map((hit) => (
<li key={hit.post_id} className="ticket-list-item ranking-hit">
<button
className="post-list-item"
aria-label={tf("Open ranking: {title}", { title: hit.post_title })}
onClick={() => onSelectPost(hit.post_id)}
>
<span className="ticket-title">{hit.post_title}</span>
<span className="post-badge">{t("Rankings · rankweave")}</span>
<span className="post-badge">{tf("rank {rank}", { rank: String(hit.fused_rank) })}</span>
</button>
{(hit.channel_evidence ?? []).length > 0 ? (
<ul
className="ranking-channel-evidence"
aria-label={tf("Ranking evidence for {title}", { title: hit.post_title })}
>
{(hit.channel_evidence ?? []).map((item) => (
<li key={item.signal_code}>
{tf("{label} rank {rank}, contribution {contribution}", {
label: t(item.signal_label),
rank: String(item.channel_rank),
contribution: formatRankingContribution(item.contribution),
})}
Comment thread
seonghobae marked this conversation as resolved.
</li>
))}
</ul>
) : null}
</li>
))}
</ul>
</>
)}
</section>
);
}
Expand Down
10 changes: 10 additions & 0 deletions frontend/src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1037,10 +1037,20 @@ export function startAnalysisRun(
});
}

export interface RankingChannelEvidence {
signal_code: string;
signal_label: string;
channel_rank: number;
weight: number;
contribution: number;
rank: number;
}

export interface RankedPost {
post_id: string;
post_title: string;
fused_rank: number;
channel_evidence?: RankingChannelEvidence[];
}

export interface RankingList {
Expand Down
3 changes: 3 additions & 0 deletions frontend/src/i18n.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ describe("i18n", () => {
"Read leftover map rank {rank}, observed Y {observed}, and expected E {expected} after IRT main effects, then open this post.",
"Leftover map rank 0 means no leftover structure after IRT main effects. Read observed Y {observed} and expected E {expected}, then open this post.",
"Showing the first {shown} of {total} posts known at this cutoff.",
"Rankings",
"Title overlap",
"RankWeave fused newest-first and title-overlap ranks. This is not a calibrated score.",
] as const;

it("supports the five product locales", () => {
Expand Down
Loading