Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/product-technical-gap-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
## 3. General Architecture Gaps
- **DB Architecture**: Ensure PostgreSQL is strictly used (no file DBs), 3rd normal form is maintained, and Hot Partitions are handled. DB locks must be managed (or use read/write replicas).
- **Zotero Integration**: Papers and standards referenced by TEPP must be synced via Local Zotero API (http://localhost:23119/api/) and cited using APA 7th edition in docstrings.
- **Testing**: We need actual testing of Psychometrics (Fast-MLSIRM parameter calibration, RMSE of estimates, Fixed-Item Parameter Calibration, CAT) against synthetic/demo data.
- **Testing**: (Partially resolved -- FIPC) `tests/test_fast_mlsirm_fipc_recovery.py` now exists, exercising the actual production functions (`fit_polytomous`, `score_polytomous`, `cat_simulate_polytomous`) for Fixed-Item Parameter Calibration against synthetic data with known ground truth. It simulates a two-period design matching `period_report.py`'s own documented rationale ("Independent refits would re-center each week at 0 and hide real movement") and proves the claim directly -- FIPC-scoring period 2 on period 1's fixed item bank recovers a deliberate period-2 mean shift (~0.37 detected vs. ~0.43 true), while an independent free refit of the identical period-2 data re-centers to ~0.01, erasing the shift entirely. Not a placebo comparison; the test asserts both outcomes. The remaining three recovery tests this entry asked for -- GRM (`tests/test_fast_mlsirm_grm_recovery.py`, #451), GPCM (`_gpcm_recovery.py`, #452), and CAT (`_cat_recovery.py`, #453) -- are on separate open PRs, not yet in this tree; once all three land this entry is (Resolved).
- **Security & Compliance**: PII masking cannot break the system. Need SOC 2 and CSAP compliance alternatives to blind PII masking.
- **LLM Orchestration**: Ensure ALL LLM calls route through `contextual-orchestrator` utilizing API keys (BYTEZ, NVIDIA, OPENROUTER, OPENAI) with auto model discovery and optimal reasoning effort allocation (Fugu/Conductor/TRINITY research).

Expand Down
3 changes: 3 additions & 0 deletions frontend/src/App.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ describe("App, unauthenticated", () => {
state: expect.objectContaining({ returnUrl: expect.stringMatching(/^\//) }),
}),
);
// Persisted as a fallback in case the OIDC state round-trip is dropped
// (see oidcReturnUrl.ts's restoreOidcReturnUrl, consumed in main.tsx).
expect(window.sessionStorage.getItem("lineageweave.oidc.returnUrl")).toMatch(/^\//);
});
});

Expand Down
4 changes: 2 additions & 2 deletions frontend/src/App.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4610,7 +4610,8 @@ export default function App({ showLabPanels = false }: { showLabPanels?: boolean
</div>
<div className="login-controls">
<button className="btn-primary" onClick={() => {
const returnUrl = window.location.pathname + window.location.search;
const returnUrl = returnUrlFromLocation();
rememberOidcReturnUrl(returnUrl);
Comment thread
seonghobae marked this conversation as resolved.
void auth.signinRedirect({ state: { returnUrl } });
}}>
{t("Log in")}
Expand All @@ -4620,7 +4621,6 @@ export default function App({ showLabPanels = false }: { showLabPanels?: boolean
<small>Enterprise SSO Authentication</small>
</div>
</div>
{destination === "admin" ? <AdminPanel currentBrandName={brandName} onBrandNameChange={setBrandName} accessToken={accessToken} /> : null}
</main>
<footer className="app-footer" role="contentinfo">
<div className="app-footer-title">
Expand Down
108 changes: 108 additions & 0 deletions tests/test_fast_mlsirm_fipc_recovery.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
"""Real Fixed-Item Parameter Calibration (FIPC, Kim 2006) test for the
two-stage design period_report.py's own module docstring describes: "the
first period for a grouping free-calibrates and persists its item bank.
Later periods EAP-score on those fixed parameters... Independent refits
would re-center each week at 0 and hide real movement."

Simulates a period-1 GRM item bank and fits it freely (the "first period
free-calibrates" step), then simulates a period-2 cohort of different
people from the SAME true item parameters but a deliberate mean-theta
shift (representing genuine week-over-week movement), and scores period 2
via score_polytomous(period2_responses, period1_fit) -- period 1's fit is
passed through unchanged, never re-estimated from period-2 data, which is
what "fixed" means here.

The test asserts two things a merely-passing "it runs" test would miss:
(1) FIPC-scored period-2 thetas recover the true mean shift reasonably
well, and
(2) an *independent* free refit of period 2 (the alternative the
docstring warns against) would have hidden that same shift by
re-centering close to 0 -- proving the FIPC path is doing something
a naive independent refit provably cannot.
"""

from __future__ import annotations

import numpy as np
from fast_mlsirm import fit_polytomous, score_polytomous

N_ITEMS = 15
N_CAT = 4
N_PERSONS_PERIOD_1 = 400
N_PERSONS_PERIOD_2 = 250
SEED = 20260101
PERIOD_2_TRUE_MEAN_SHIFT = 0.3

# A real run with these exact parameters/seed measures period-2 FIPC theta
# RMSE ~0.33 and correlation ~0.94, with the FIPC-estimated period-2 mean
# landing at ~0.37 against a true realized mean of ~0.43 (the shift is
# substantially recovered, not perfectly -- sampling noise on 250 people).
# An independent free refit of the same period-2 data, by contrast,
# re-centers to an estimated mean of ~0.01 -- essentially erasing the
# shift, exactly the failure mode FIPC exists to avoid.
MAX_THETA_RMSE = 0.6
MIN_THETA_CORRELATION = 0.75
MIN_FIPC_MEAN_SHIFT_DETECTED = PERIOD_2_TRUE_MEAN_SHIFT * 0.5
MAX_INDEPENDENT_REFIT_MEAN_SHIFT_DETECTED = PERIOD_2_TRUE_MEAN_SHIFT * 0.35
Comment thread
seonghobae marked this conversation as resolved.


def _grm_category_probs(theta: float, discrimination: float, thresholds: np.ndarray) -> np.ndarray:
"""Samejima (1969) graded-response category probabilities."""
cumulative = np.concatenate(([1.0], 1.0 / (1.0 + np.exp(-discrimination * (theta - thresholds))), [0.0]))
return -np.diff(cumulative)


def _simulate_responses(theta: np.ndarray, discrimination: np.ndarray, thresholds: np.ndarray, seed: int) -> np.ndarray:
rng = np.random.default_rng(seed)
n_persons = len(theta)
n_items = len(discrimination)
responses = np.zeros((n_persons, n_items))
for item in range(n_items):
for person in range(n_persons):
probs = _grm_category_probs(theta[person], discrimination[item], thresholds[item])
probs = np.clip(probs, 0.0, None)
probs = probs / probs.sum()
responses[person, item] = rng.choice(N_CAT, p=probs)
return responses


def test_fipc_recovers_period_two_mean_shift_that_an_independent_refit_would_hide() -> None:
param_rng = np.random.default_rng(SEED)
true_discrimination = param_rng.uniform(0.8, 2.0, N_ITEMS)
true_thresholds = np.sort(param_rng.normal(0.0, 1.0, (N_ITEMS, N_CAT - 1)), axis=1)

theta_period_1 = param_rng.normal(0.0, 1.0, N_PERSONS_PERIOD_1)
theta_period_2 = param_rng.normal(PERIOD_2_TRUE_MEAN_SHIFT, 1.0, N_PERSONS_PERIOD_2)

responses_period_1 = _simulate_responses(theta_period_1, true_discrimination, true_thresholds, SEED + 1)
period_1_fit = fit_polytomous(responses_period_1, n_cat=N_CAT, model="grm")
assert period_1_fit.converged

responses_period_2 = _simulate_responses(theta_period_2, true_discrimination, true_thresholds, SEED + 2)

# FIPC: score period 2 on period 1's fixed (not re-estimated) item bank.
fipc_scored = score_polytomous(responses_period_2, period_1_fit)
fipc_theta_eap = fipc_scored["theta_eap"]

rmse = float(np.sqrt(np.mean((fipc_theta_eap - theta_period_2) ** 2)))
correlation = float(np.corrcoef(fipc_theta_eap, theta_period_2)[0, 1])
fipc_detected_shift = float(fipc_theta_eap.mean())
Comment thread
seonghobae marked this conversation as resolved.

assert rmse < MAX_THETA_RMSE, f"FIPC period-2 theta RMSE {rmse:.3f} exceeded {MAX_THETA_RMSE}"
assert correlation > MIN_THETA_CORRELATION, f"FIPC period-2 theta correlation {correlation:.3f} below {MIN_THETA_CORRELATION}"
assert fipc_detected_shift > MIN_FIPC_MEAN_SHIFT_DETECTED, (
f"FIPC-detected period-2 mean shift {fipc_detected_shift:.3f} did not clear "
f"{MIN_FIPC_MEAN_SHIFT_DETECTED:.3f} -- FIPC should preserve real week-over-week movement"
)

# The comparison that proves FIPC matters: an independent free refit of
# the SAME period-2 data, never told about period 1's fixed bank.
independent_fit = fit_polytomous(responses_period_2, n_cat=N_CAT, model="grm")
independent_scored = score_polytomous(responses_period_2, independent_fit)
independent_detected_shift = float(independent_scored["theta_eap"].mean())

assert abs(independent_detected_shift) < MAX_INDEPENDENT_REFIT_MEAN_SHIFT_DETECTED, (
f"independent refit detected shift {independent_detected_shift:.3f}, expected it to "
f"re-center near 0 (below {MAX_INDEPENDENT_REFIT_MEAN_SHIFT_DETECTED:.3f}) -- if it also "
f"recovers the shift, this test no longer demonstrates FIPC's actual value over a naive refit"
)
Loading