From 2ab96809c3741a9d19a8cbfb21102fd0c58bb3c4 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Sun, 23 Aug 2026 19:06:16 +0000 Subject: [PATCH 1/2] feat: measure leftover-map distances on two Gabriel axes (v2.12.18) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Jeon leftover interaction is a two-dimensional person–item map. Closest and farthest leftover_distance now use the two leading Gabriel axes; hidden SVD components after the second no longer change persisted distance. Rank-0 and rank-1 maps pad unused axes with zero. Missing cells stay out of the factorization. Independent of leftover-map coordinates (#481), coverage (#518), axis share (#519), comparison-strip reuse (#521), and residual disclosure (#502). Does not invent a leftover score or a theta. --- AGENTS.md | 7 +- ARCHITECTURE.md | 2 +- ...8-leftover-map-two-dimensional-distance.md | 8 + CHANGELOG.md | 11 ++ docs/adr/0048-persist-lsirm-leftover-pairs.md | 4 +- ...6-leftover-map-two-dimensional-distance.md | 56 +++++++ frontend/package.json | 2 +- lineageweave/leftover_pairs.py | 39 +++-- pyproject.toml | 2 +- tests/test_leftover_pairs.py | 141 +++++++++++++++++- uv.lock | 2 +- 11 files changed, 253 insertions(+), 21 deletions(-) create mode 100644 CHANGELOG.d/2.12.18-leftover-map-two-dimensional-distance.md create mode 100644 docs/adr/0166-leftover-map-two-dimensional-distance.md diff --git a/AGENTS.md b/AGENTS.md index 1728f9e61..c6b354f49 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -186,10 +186,11 @@ in the same spirit) -- never against real data, per the hard rule above. against a live local stack (`make up`) and self-skip without one -- see [README.md](README.md#local-product-stack-docker-compose). -Period leftover pairs (ADR 0017 / 0018) are computed in +Period leftover pairs (ADR 0048 / 0049 / 0166) are computed in `lineageweave/leftover_pairs.py` from the residual after a real -GRM/GPCM score, never invented. Missing cells stay out of the -Gabriel factorization. Closest and farthest post–criterion pairs +GRM/GPCM score, never invented. Distances are Euclidean on the +two-dimensional Gabriel leftover map; missing cells stay out of +the factorization. Closest and farthest post–criterion pairs persist to `report_leftover_pair` and sit above the member list so a click opens that post. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index d0280ff97..7798c3ac3 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -588,7 +588,7 @@ on those same fixed parameters (Kim, 2006 FIPC). After scoring, `information_polytomous` ranks the shared-bank items by Fisher information at the group's mean θ (Lord, 1980 max-info CAT). Rankings persist to `report_item_information`. After those IRT main effects, -residual SVD leftover pairs (Jeon et al., 2021; ADR 0017) persist to +residual SVD leftover pairs on two Gabriel axes (Jeon et al., 2021; ADR 0048 / 0166) persist to `report_leftover_pair`. Results persist to `report_period_score` / `report_member_score`. `GET /api/reports/{grouping}` lists the trend; diff --git a/CHANGELOG.d/2.12.18-leftover-map-two-dimensional-distance.md b/CHANGELOG.d/2.12.18-leftover-map-two-dimensional-distance.md new file mode 100644 index 000000000..ece15f9ba --- /dev/null +++ b/CHANGELOG.d/2.12.18-leftover-map-two-dimensional-distance.md @@ -0,0 +1,8 @@ +## 2.12.18 — Two-dimensional leftover-map distances + +- Measure closest and farthest leftover post–criterion distances on + the two-axis Jeon / Gabriel interaction map (ADR 0166). Hidden SVD + axes after the second no longer change `leftover_distance`. Rank-0 + and rank-1 maps still pad unused axes with zero. After `make seed`, + leftover pairs above the member list still open that post. Never + invent a leftover score or a theta. diff --git a/CHANGELOG.md b/CHANGELOG.md index c8ed1a099..33e93a0a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,17 @@ All notable changes to this project are documented here. Format follows environment, so local OIDC and synthetic-data workflows resolve the same pinned dependencies as CI. +## [2.12.18] - 2026-08-24 + +### Fixed + +- Closest and farthest leftover post–criterion distances are now + Euclidean on the two-axis Jeon / Gabriel interaction map (ADR 0166). + Hidden SVD axes after the second no longer change `leftover_distance`. + Rank-0 and rank-1 maps still pad unused axes with zero, so seed + leftover pairs above the member list still open that post. Never + invent a leftover score or a theta. + ## [2.12.6] - 2026-08-20 ### Added diff --git a/docs/adr/0048-persist-lsirm-leftover-pairs.md b/docs/adr/0048-persist-lsirm-leftover-pairs.md index 8e87383f0..6f548ed40 100644 --- a/docs/adr/0048-persist-lsirm-leftover-pairs.md +++ b/docs/adr/0048-persist-lsirm-leftover-pairs.md @@ -2,6 +2,7 @@ **Decision status:** Accepted **Date:** 2026-08-17 +**Amended by:** [ADR 0166](0166-leftover-map-two-dimensional-distance.md) (two leftover-map axes) ## Context @@ -30,7 +31,8 @@ and one `farthest` observed cell per period report in `report_leftover_pair` (3NF, two-or-more-word `snake_case`). The biplot lives in `lineageweave/leftover_pairs.py` so leftover -tests do not import `period_report` or `fast_mlsirm`. +tests do not import `period_report` or `fast_mlsirm`. Distances are +Euclidean on the two leftover-map axes (ADR 0166). Cascade the rows with `report_period_score`. A leftover post must also be a `report_member_score` row, and the leftover criterion diff --git a/docs/adr/0166-leftover-map-two-dimensional-distance.md b/docs/adr/0166-leftover-map-two-dimensional-distance.md new file mode 100644 index 000000000..f865c8d3a --- /dev/null +++ b/docs/adr/0166-leftover-map-two-dimensional-distance.md @@ -0,0 +1,56 @@ +# ADR 0166 — Measure leftover-map distances on two Gabriel axes + +**Decision status:** Accepted +**Date:** 2026-08-24 + +Amends [ADR 0048](0048-persist-lsirm-leftover-pairs.md). Independent of +leftover-map coordinates, leftover-map coverage, leftover-map axis +share, leftover residual-row disclosure, and leftover comparison-strip +reuse. + +## Context + +ADR 0048 persists closest and farthest leftover post–criterion pairs +after IRT main effects. Jeon et al. (2021, eq. 3) place the leftover +interaction `−γ‖ξ_p − ζ_i‖` on a two-dimensional person–item map. +Gabriel (1971) supplies those coordinates from a residual biplot. + +`leftover_pairs.py` previously measured Euclidean distance on every +kept SVD axis. A rank-3 residual therefore reported a leftover +distance that a buyer cannot read on the 2D interaction map, and +that would disagree with stored two-axis coordinates if those later +persist. Rank-1 seed fixtures still passed because unused axes were +absent, not because the estimator was two-dimensional. + +## Decision + +Closest and farthest leftover distances are Euclidean on **exactly +two** leftover-map axes. `_leftover_map_positions` may still return +the full Gabriel factorization; `_pad_map_axes` pads a rank-0 or +rank-1 map with zeros and truncates hidden axes after the second. +Missing cells stay out of the factorization. Rank-0 residuals still +emit a stable pair with distance zero. Do not invent a leftover +score. Do not invent a theta. + +This slice does not persist map coordinates or change leftover UI. +Those remain ADR 0048 / 0049, and persist-map tables stay on their +own PR stack. + +## Consequences + +`leftover_distance` on `report_leftover_pair` matches the 2D Jeon +map. After `make seed`, closest and farthest pairs above the member +list still open that post. A rank-3 synthetic residual proves the +stored distance equals the two-axis hypot and is not the full-rank +norm. + +## References + +Gabriel, K. R. (1971). The biplot graphic display of matrices with +application to principal component analysis. *Biometrika, 58*(3), +453–467. https://doi.org/10.1093/biomet/58.3.453 + +Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping +unobserved item–respondent interactions: A latent space item response +model with interaction map. *Psychometrika, 86*(2), 378–403. +https://doi.org/10.1007/s11336-021-09762-5 diff --git a/frontend/package.json b/frontend/package.json index e2e996bbe..938ac6bdd 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,7 +1,7 @@ { "name": "frontend", "private": true, - "version": "2.12.6", + "version": "2.12.18", "type": "module", "scripts": { "dev": "vite", diff --git a/lineageweave/leftover_pairs.py b/lineageweave/leftover_pairs.py index 353b1a6ef..7fa6e924d 100644 --- a/lineageweave/leftover_pairs.py +++ b/lineageweave/leftover_pairs.py @@ -1,9 +1,12 @@ -"""Jeon leftover post–criterion pairs after a main-effect IRT (ADR 0017). +"""Jeon leftover post–criterion pairs after a main-effect IRT (ADR 0048 / 0166). Does not import ``fast_mlsirm`` or ``period_report``. A Gabriel biplot of the residual ``R = Y − E[Y|θ, item]`` supplies person and item positions. Missing response cells are excluded from the factorization; -they are never treated as zero residuals. +they are never treated as zero residuals. Pair distances are Euclidean +on the two leftover-map axes (Jeon et al., 2021); unused axes pad with +zero rather than inventing a second component, and hidden SVD axes +after the second are dropped. """ from __future__ import annotations @@ -15,6 +18,7 @@ PAIR_KIND_CLOSEST = "closest" PAIR_KIND_FARTHEST = "farthest" _LEFTOVER_SINGULAR_FLOOR = 1e-12 +_LEFTOVER_MAP_AXES = 2 @dataclass(frozen=True) @@ -38,9 +42,10 @@ def leftover_pairs_from_residual( Jeon et al. (2021) leftover interaction is ``−γ‖ξ_p − ζ_i‖``. This estimator places persons and items from the residual after IRT main - effects (Gabriel, 1971). Only observed cells become pairs. A rank-0 - residual still emits a stable closest/farthest pair so seed is not - empty; it does not invent a leftover score. + effects (Gabriel, 1971). Only observed cells become pairs. Distances + use the two leftover-map axes; a rank-0 residual still emits a + stable closest/farthest pair so seed is not empty and does not + invent a leftover score. """ if matrix.shape != (len(post_ids), len(item_codes)): raise ValueError( @@ -72,13 +77,15 @@ def leftover_pairs_from_residual( if person_pos is not None and item_pos is not None: person_index = np.flatnonzero(keep_person) item_index = np.flatnonzero(keep_item) + person_xy = _pad_map_axes(person_pos) + item_xy = _pad_map_axes(item_pos) local_person = {int(person): local for local, person in enumerate(person_index)} local_item = {int(item): local for local, item in enumerate(item_index)} for person, item in observed: if person not in local_person or item not in local_item: continue distance = float( - np.linalg.norm(person_pos[local_person[person]] - item_pos[local_item[item]]) + np.linalg.norm(person_xy[local_person[person]] - item_xy[local_item[item]]) ) if not np.isfinite(distance): continue @@ -148,7 +155,12 @@ def _complete_case_positions( def _leftover_map_positions(filled: np.ndarray) -> tuple[np.ndarray, np.ndarray]: - """Gabriel biplot coordinates; rank-0 residuals collapse to the origin.""" + """Gabriel coordinates ordered by descending singular value. + + NumPy's SVD contract returns singular values largest-first, so filtering + by the numerical floor preserves a prefix and the first two columns remain + the two leading leftover-map axes. Rank-0 residuals collapse to the origin. + """ n_persons, n_items = filled.shape if n_persons == 0 or n_items == 0 or not np.any(np.abs(filled) > _LEFTOVER_SINGULAR_FLOOR): return ( @@ -157,12 +169,15 @@ def _leftover_map_positions(filled: np.ndarray) -> tuple[np.ndarray, np.ndarray] ) left, singular, right = np.linalg.svd(filled, full_matrices=False) keep = singular > _LEFTOVER_SINGULAR_FLOOR - if not np.any(keep): - return ( - np.zeros((n_persons, 1), dtype=np.float64), - np.zeros((n_items, 1), dtype=np.float64), - ) scale = np.sqrt(singular[keep]) person_pos = left[:, keep] * scale item_pos = right[keep, :].T * scale return person_pos, item_pos + + +def _pad_map_axes(positions: np.ndarray) -> np.ndarray: + """Pad or truncate Gabriel coordinates to two leftover-map axes.""" + padded = np.zeros((positions.shape[0], _LEFTOVER_MAP_AXES), dtype=np.float64) + width = min(_LEFTOVER_MAP_AXES, positions.shape[1]) + padded[:, :width] = positions[:, :width] + return padded diff --git a/pyproject.toml b/pyproject.toml index cb4be2916..9a3975230 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "lineageweave" -version = "2.12.6" +version = "2.12.18" description = "Reconstructs git-branch-style lineage DAGs from scattered short records using multi-channel score fusion and LLM adjudication." readme = "README.md" license = { text = "MIT" } diff --git a/tests/test_leftover_pairs.py b/tests/test_leftover_pairs.py index 72dea89b6..784809fa5 100644 --- a/tests/test_leftover_pairs.py +++ b/tests/test_leftover_pairs.py @@ -1,4 +1,4 @@ -"""Leftover post–criterion pairs after the main-effect IRT (ADR 0017). +"""Leftover post–criterion pairs after the main-effect IRT (ADR 0048 / 0166). Uses a constructed residual matrix so the closest and farthest pair are known without calling ``fit_polytomous``. Loads @@ -17,9 +17,12 @@ import pytest _LEFTOVER_PATH = Path(__file__).resolve().parents[1] / "lineageweave" / "leftover_pairs.py" +_LEFTOVER_SINGULAR_FLOOR = 1e-12 +_LEFTOVER_MAP_AXES = 2 def _load_leftover(): + """Load only the dependency-light leftover module under test.""" source = _LEFTOVER_PATH.read_text(encoding="utf-8") imported = [] for node in ast.parse(source).body: @@ -43,6 +46,22 @@ def _load_leftover(): leftover_pairs_from_residual = leftover.leftover_pairs_from_residual +def _gabriel_positions(filled: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Independent Gabriel coordinates used to prove leftover_distance axes.""" + left, singular, right = np.linalg.svd(filled, full_matrices=False) + keep = singular > _LEFTOVER_SINGULAR_FLOOR + scale = np.sqrt(singular[keep]) + return left[:, keep] * scale, right[keep, :].T * scale + + +def _pad_map_axes(positions: np.ndarray) -> np.ndarray: + """Independently pad or truncate coordinates to two map axes.""" + padded = np.zeros((positions.shape[0], _LEFTOVER_MAP_AXES), dtype=np.float64) + width = min(_LEFTOVER_MAP_AXES, positions.shape[1]) + padded[:, :width] = positions[:, :width] + return padded + + def test_leftover_residual_biplot_separates_aligned_and_opposed_cells() -> None: """A rank-1 leftover spike puts the aligned cell closest and the opposed cell farthest.""" post_ids = ["post-a", "post-b", "post-c"] @@ -70,6 +89,7 @@ def test_leftover_residual_biplot_separates_aligned_and_opposed_cells() -> None: def test_zero_residual_still_emits_stable_leftover_pairs() -> None: + """A rank-zero map retains deterministic closest and farthest rows.""" post_ids = ["alpha-post", "beta-post"] item_codes = ("item_one", "item_two") matrix = np.ones((2, 2), dtype=np.float64) @@ -111,8 +131,127 @@ def test_partial_observation_does_not_treat_missing_as_zero_residual() -> None: def test_leftover_is_empty_without_observed_cells() -> None: + """An entirely missing response matrix yields no invented pair.""" post_ids = ["post-empty"] item_codes = ("item_one",) matrix = np.array([[np.nan]], dtype=np.float64) expected = np.array([[0.0]], dtype=np.float64) assert leftover_pairs_from_residual(post_ids, item_codes, matrix, expected) == () + + +def test_rank_three_pair_distances_match_two_dimensional_gabriel_coords() -> None: + """Jeon leftover_distance is Euclidean on the 2D map, not the full SVD rank.""" + post_ids = ["post-a", "post-b", "post-c", "post-d"] + item_codes = ("item-a", "item-b", "item-c", "item-d") + matrix = np.array( + [ + [4.0, 1.0, 0.0, -1.0], + [0.0, 3.0, 1.0, -2.0], + [-2.0, 0.0, 2.0, 1.0], + [1.0, -1.0, 0.0, 4.0], + ], + dtype=np.float64, + ) + expected = np.zeros_like(matrix) + filled = matrix - float(np.mean(matrix)) + person_full, item_full = _gabriel_positions(filled) + assert person_full.shape[1] >= 3 + person_map = _pad_map_axes(person_full) + item_map = _pad_map_axes(item_full) + full_distances = np.linalg.norm(person_full[:, None, :] - item_full[None, :, :], axis=2) + map_distances = np.linalg.norm(person_map[:, None, :] - item_map[None, :, :], axis=2) + assert float(np.max(np.abs(full_distances - map_distances))) > 1e-6 + + pairs = leftover_pairs_from_residual(post_ids, item_codes, matrix, expected) + assert [pair.pair_kind for pair in pairs] == [PAIR_KIND_CLOSEST, PAIR_KIND_FARTHEST] + post_index = {post_id: index for index, post_id in enumerate(post_ids)} + item_index = {code: index for index, code in enumerate(item_codes)} + for pair in pairs: + person = post_index[pair.post_id] + item = item_index[pair.criterion_code] + assert pair.leftover_distance == pytest.approx(float(map_distances[person, item])) + assert pair.leftover_distance != pytest.approx( + float(full_distances[person, item]), abs=1e-9 + ) + + farthest_map = np.unravel_index(int(np.argmax(map_distances)), map_distances.shape) + farthest = pairs[1] + assert (post_index[farthest.post_id], item_index[farthest.criterion_code]) == farthest_map + + +def test_rejects_response_and_expectation_shape_mismatches() -> None: + """Scientific inputs must match their declared post and criterion axes.""" + with pytest.raises(ValueError, match="matrix shape"): + leftover_pairs_from_residual( + ["post-a"], + ("item-a",), + np.zeros((2, 1), dtype=np.float64), + np.zeros((2, 1), dtype=np.float64), + ) + with pytest.raises(ValueError, match="expected shape"): + leftover_pairs_from_residual( + ["post-a"], + ("item-a",), + np.zeros((1, 1), dtype=np.float64), + np.zeros((1, 2), dtype=np.float64), + ) + + +def test_sparse_residual_uses_only_observed_cells_for_fallback_distance() -> None: + """No complete rectangle still yields finite observed-cell distances.""" + matrix = np.array([[1.0, np.nan], [np.nan, -1.0]], dtype=np.float64) + pairs = leftover_pairs_from_residual( + ["post-a", "post-b"], + ("item-a", "item-b"), + matrix, + np.zeros_like(matrix), + ) + assert [(pair.post_id, pair.criterion_code) for pair in pairs] == [ + ("post-a", "item-a"), + ("post-b", "item-b"), + ] + assert [pair.leftover_distance for pair in pairs] == pytest.approx([1.0, 1.0]) + + +def test_nonfinite_map_distance_falls_back_to_centered_residual( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An unusable factorization coordinate cannot become persisted distance.""" + monkeypatch.setattr( + leftover, + "_complete_case_positions", + lambda *_args: ( + np.array([[np.inf]], dtype=np.float64), + np.array([[-np.inf]], dtype=np.float64), + ), + ) + pairs = leftover_pairs_from_residual( + ["post-a"], + ("item-a",), + np.array([[1.0]], dtype=np.float64), + np.array([[0.0]], dtype=np.float64), + ) + assert [pair.leftover_distance for pair in pairs] == [0.0, 0.0] + + +def test_empty_observation_mask_has_no_complete_case_axes() -> None: + """The complete-case helpers preserve an empty scientific boundary.""" + observed = np.zeros((1, 1), dtype=bool) + keep_person, keep_item = leftover._complete_case_masks(observed) + assert not keep_person.any() + assert not keep_item.any() + person_pos, item_pos = leftover._complete_case_positions( + np.zeros((1, 1), dtype=np.float64), + 0.0, + keep_person, + keep_item, + ) + assert person_pos is None + assert item_pos is None + + +def test_pad_map_axes_truncates_hidden_svd_components() -> None: + """Axes after the second leftover-map axis do not enter distance.""" + padded = leftover._pad_map_axes(np.array([[1.0, 2.0, 9.0]], dtype=np.float64)) + assert padded.shape == (1, 2) + assert padded[0].tolist() == pytest.approx([1.0, 2.0]) diff --git a/uv.lock b/uv.lock index 10bcf9ff1..112d46840 100644 --- a/uv.lock +++ b/uv.lock @@ -454,7 +454,7 @@ wheels = [ [[package]] name = "lineageweave" -version = "2.12.6" +version = "2.12.18" source = { editable = "." } dependencies = [ { name = "certifi" }, From c298a7008125abdc4952b6ebdd20b1ca23f49ad5 Mon Sep 17 00:00:00 2001 From: Seongho Bae Date: Mon, 24 Aug 2026 13:24:10 +0900 Subject: [PATCH 2/2] fix(frontend): wire ADR 0109 return-url capture and narrow admin token - The unauthenticated Log in handler now calls returnUrlFromLocation() then rememberOidcReturnUrl() before signinRedirect, per ADR 0109, so a shared /?post= link still opens that post after enterprise SSO. The previously unused oidcReturnUrl import is now load-bearing. - AdminPanel renders only when accessToken is a string; the OIDC access token is string | undefined before isAuthenticated narrowing. --- frontend/src/App.tsx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index 6fba0dd41..8765205c6 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -4610,7 +4610,8 @@ export default function App({ showLabPanels = false }: { showLabPanels?: boolean
- {destination === "admin" ? : null} + {destination === "admin" && accessToken ? : null}