diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0b12c6c2c..c7f17f666 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -15,17 +15,15 @@ repos in the ecosystem: trajectories, uncertainty-quantified estimates) is [TEPP](https://github.com/ContextualWisdomLab/TEPP)'s job. -This is why the org-wide rule that mathematical/psychometrics computation -layers must be Rust with GPU + CPU multithreading does not apply to this -repo: LineageWeave does no such computation. Its heaviest per-request work -is fusing a handful of `[0, 1]` channel scores over a bounded candidate -window (`reconstruct.DEFAULT_CANDIDATE_WINDOW`, default 50) -- a scheduling -and orchestration problem, not a numerical-estimation one. If a future -version added real statistical inference (e.g. estimating thread-assignment -uncertainty), that layer would move into TEPP rather than being built here, -consistent with the dependency direction the ecosystem's own architecture -docs already establish (`psychometrics-commons`'s TRD explicitly forbids a -downstream product from reimplementing a measurement engine's model). +ADR 0208 fixes the end state: LineageWeave retains wire validation, +authorization, provenance persistence, and UI projection only. The current +Python IRT/report, residual-map, similarity, graph-ranking, and fusion paths +are explicitly inventoried migration debt rather than evidence that this +repository owns their mathematics. They move by construct to TEPP, +fast-mlsirm, or RankWeave after versioned Rust CPU/GPU owner contracts pass +recovery/equivalence checks; affected product paths fail closed during each +cutover rather than substituting a local estimate. See +`docs/doctoring/python-mathematical-compute-boundary-audit.md`. ## Data flow diff --git a/docs/adr/0208-externalize-local-mathematical-compute.md b/docs/adr/0208-externalize-local-mathematical-compute.md new file mode 100644 index 000000000..42a5a0591 --- /dev/null +++ b/docs/adr/0208-externalize-local-mathematical-compute.md @@ -0,0 +1,116 @@ +# ADR 0208 — Externalize local mathematical computation + +**Decision status:** Accepted +**Date:** 2026-08-25 +**Amends:** ADR 0003, ADR 0024, ADR 0064, ADR 0084, ADR 0132, ADR 0145, +ADR 0148, ADR 0167, ADR 0168, ADR 0182, ADR 0185, ADR 0200, ADR 0201, and +ADR 0205 + +## Context + +LineageWeave's product boundary says that it reconstructs, authorizes, +persists, and presents evidence but does not own calibrated estimation. The +current exact head nevertheless contains Python implementations of IRT report +fitting and scoring, expected-information channel weights, residual SVD and +Gabriel coordinates, embedding cosine, graph random-walk ranking, score +normalization, and RRF contribution arithmetic. Calling a Rust-backed Python +package does not remove the local arithmetic that prepares, transforms, or +interprets its numerical result. + +The ecosystem product boundaries are already sufficient: + +- TEPP's approved PRD owns multilingual temporal and relational measurement, + shared-latent topic identity, trajectories, uncertainty, and event lineage. +- fast-mlsirm's PRD owns reusable IRT/LSIRM estimation, prediction, + diagnostics, recovery, multilevel and multiple-membership computation. +- RankWeave owns retrieval fusion, ranking, evaluation, comparison, and + policy selection. Its calculation core must itself move behind a Rust + CPU/GPU implementation before LineageWeave treats a new result as governed + numerical evidence. + +LineageWeave has no standalone canonical PRD file on this exact head. Until +one lands, `ARCHITECTURE.md` and the accepted ADR set are the product baseline; +this absence remains a product-documentation gap, not permission to infer a +different responsibility. + +## Decision + +1. **No new local numerical model.** LineageWeave adds no Python + mathematical, statistical, psychometric, ranking, fusion, optimization, + matrix-factorization, graph-centrality, or similarity implementation. +2. **Owner by construct.** TEPP owns temporal/topic/event/trajectory + measurement. fast-mlsirm owns psychometric estimation, item information, + expected responses, residual interaction maps, uncertainty, recovery, and + multilevel/multiple-membership post importance. RankWeave owns retrieval + fusion, ranking metrics, contribution evidence, comparisons, and policy + selection. A construct is not moved merely to obtain a preferred language. +3. **Rust execution contract.** New or migrated owner computation executes in + the owner's Rust core with GPU acceleration when supported and a + deterministic multithreaded CPU path. Python may be a generated binding or + transport adapter only; it may not reproduce a formula. +4. **Consumer-only LineageWeave.** This repository retains request/envelope + validation, ABAC filtering, immutable input/output digests, run and model + versions, knowledge cutoff, provenance persistence, and UI projection. + Missing, malformed, non-converged, mixed-snapshot, or unsupported results + fail closed. It never repairs, normalizes, estimates, or substitutes a + numerical result. +5. **No big-bang rewrite.** Existing local computation is frozen as named + migration debt in + `docs/doctoring/python-mathematical-compute-boundary-audit.md`. Each owner + contract lands and proves recovery/equivalence before the corresponding + LineageWeave implementation is deleted. Existing behavior is not relabeled + as compliant while it remains local. +6. **Independent TEPP anchor.** Event-Lineage channel-weight activation keeps + ADR 0205's exact TEPP anchor requirement. fast-mlsirm may estimate weights + conditional on that accepted independent anchor; it does not manufacture + the criterion. +7. **No heuristic exception.** Candidate windows, score floors, token overlap, + string similarity, or equal weights are not promoted to measurement. + Operational bounds may remain only as disclosed resource limits and may + not determine a scientific score or ground truth. + +## Stacked delivery order + +1. Owner PRs publish versioned request/result schemas, model identity, + convergence/uncertainty evidence, input digest, and deterministic recovery + tests: TEPP first, fast-mlsirm second, RankWeave third. +2. A LineageWeave contract-only PR adds strict clients and provenance tables; + no UI activates from an unpersisted envelope. +3. A shadow-validation PR compares owner outputs with frozen synthetic + fixtures and records aggregate, non-identifying evidence. +4. Separate deletion PRs remove `channel_weight_estimation.py`, numerical + portions of `period_report.py` and `leftover_pairs.py`, local cosine/RWR, + and local ranking contribution/normalization code after their owner path is + accepted. +5. The final PR removes NumPy/fast-mlsirm/RankWeave calculation imports from + LineageWeave, updates architecture/PRD/ADRs, and makes the transition guard + require an empty debt inventory. + +## Consequences + +The Dashboard may show TEPP topics and fast-mlsirm importance only from exact, +persisted owner artifacts. It can explain the source posts, memberships, +levels, time window, model version, uncertainty, and provenance, but cannot +recalculate or rank them locally. During migration, affected capabilities +remain explicitly legacy or unavailable rather than presenting local results +as Rust/GPU-backed. + +## References (APA 7th) + +Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel item +response model using Gibbs sampling. *Psychometrika, 66*(2), 271–288. +https://doi.org/10.1007/BF02294839 + +Gabriel, K. R. (1971). The biplot graphic display of matrices with application +to principal component analysis. *Biometrika, 58*(3), 453–467. +https://doi.org/10.1093/biomet/58.3.453 + +Jeon, M., Jin, I. H., Schweinberger, M., & Baugh, S. (2021). Mapping unobserved +item-respondent interactions: A latent space item response model with +interaction map. *Psychometrika, 86*(2), 378–403. +https://doi.org/10.1007/s11336-021-09762-5 + +Roberts, M. E., Stewart, B. M., & Tingley, D. (2019). stm: An R package for +structural topic models. *Journal of Statistical Software, 91*(2), 1–40. +https://doi.org/10.18637/jss.v091.i02 + diff --git a/docs/adr/0210-temporal-topic-context-influence-dashboard.md b/docs/adr/0210-temporal-topic-context-influence-dashboard.md new file mode 100644 index 000000000..e9be55d48 --- /dev/null +++ b/docs/adr/0210-temporal-topic-context-influence-dashboard.md @@ -0,0 +1,236 @@ +# ADR 0210: TEPP temporal topics and fast-mlsirm context influence + +- Status: Accepted +- Implementation maturity: producer-contract required; consumer projection not yet shipped +- Date: 2026-08-25 +- Depends on: ADR 0132 (TEPP topic-lineage boundary), ADR 0206 (operations Dashboard) +- Upstream authorities: TEPP ADR 0012; fast-mlsirm ADR 0002 and ADR 0007 + +## Context + +The operations Dashboard must show how topics evolve through Event Lineage and +which posts materially influence a topic's fitted state at business-unit, +process-unit (PU), team, and person levels. A lexical cluster, one topic model +per time bin, raw topic proportion, engagement count, or hand-authored weighted +sum cannot answer that question. Those approaches lose stable topic identity, +ignore multiple membership, understate dependence, or silently redefine +"important". + +TEPP's approved PRD and ADR 0012 already own Temporal Relational Shared-Latent +Topic Measurement (TRSL-TM): global topic identities, event time, explicit +document relations, weighted cross-classified memberships, posterior +uncertainty, and topic activity over time. LineageWeave therefore consumes a +TEPP artifact; it does not fit or label topics locally. + +For this surface, **important post** has one exact statistical meaning: +case-deletion influence on a fitted topic-by-context parameter. For topic +`k`, context dimension `l`, and post `d`, fast-mlsirm reports + +\[ +D_{dkl}=(\hat\psi_{kl,-d}-\hat\psi_{kl})^\top +I_{kl}(\hat\psi)(\hat\psi_{kl,-d}-\hat\psi_{kl}), +\] + +where `I` is the same fitted model's observed-information block and +`psi[-d]` is the estimate after removing that post's complete observation. +This is a multilevel case-deletion diagnostic, not business value, causal +impact, author performance, or an outlier-removal instruction (Shi & Chen, +2008). It is selected because it is defined by the fitted likelihood and +observed information, so no arbitrary cross-level weights or score constants +are introduced. + +## Product requirements (PRD) + +1. The Dashboard presents TEPP topics on one event-time axis with stable topic + identity and explicit active, dormant, and reactivated states. Topic + birth/split/merge/retirement appears only when the TEPP artifact explicitly + supplies that lineage event. +2. Selecting a topic shows separate business-unit, PU, team, and person views. + A post may belong to more than one context in the same dimension and to + contexts in several dimensions. The UI never flattens those assignments + into a single owner. +3. Each level lists posts by fast-mlsirm case-deletion influence `D[d,k,l]`, + with exact value, uncertainty/diagnostic status, source event time, topic + state, membership provenance, and a link to the authorized source post. + No score threshold is applied. Equal values remain ties; deterministic + source time and post identity order only stabilize rendering and do not + break the statistical tie. +4. Copy names the estimand as **model influence**. It must not say business + importance, performance, causality, risk, or priority unless a separately + validated outcome model establishes that construct. +5. Pending, failed, non-converged, unidentified, incomplete-membership, + CPU/GPU-parity-failed, or contract-mismatched runs render an actionable + unavailable state. LineageWeave never fills them with keyword search, + engagement counts, RankWeave output, default weights, or a local estimate. +6. All rows are authorization-filtered before topic/context aggregation. A + hidden source post contributes neither a displayed rank nor an exact value + that could disclose its influence. +7. The topic view is a Dashboard section, not a new external-information + board. It reuses the existing GNB destination and post-detail navigation. + +## Technical requirements (TRD) + +### TEPP producer contract + +The accepted TEPP result schema must include: + +- immutable model-run, source-snapshot SHA-256, knowledge cutoff, model/schema + version, event clock, and posterior-draw identity; +- global topic identity and activity interval, plus explicit lineage event and + provenance when present; +- per-post posterior logistic-normal topic coordinates or plausible values, + not a hard topic label derived from a threshold; +- Event Lineage/document-relation edges admitted by the TEPP run; +- versioned, time-valid business-unit, PU, team, and person membership edges + with source-derived weights and evidence. A missing weight is unavailable; + equal membership is never invented. + +LineageWeave verifies the exact snapshot and cutoff before persisting a 3NF +projection. It does not inspect TEPP's private tables or reinterpret posterior +coordinates. + +TEPP protected main currently exposes `tepp.trsl_topic_lineage.v1`, a +digest-bound CPU-`f64` artifact containing fitted forward sequence edges and +aggregate counts. That is real producer progress, but it does not contain the +per-post posterior coordinates/plausible values or dimension-qualified +membership evidence required by this decision. LineageWeave must reject that +schema for the context-influence surface rather than reconstruct the omitted +inputs from its association-strength field. + +### fast-mlsirm producer contract + +fast-mlsirm owns a versioned `topic_context_influence` estimand over TEPP +posterior plausible values. It jointly retains topic, event time, and the four +dimension-qualified multiple-membership designs. Rust owns likelihood, +gradients, observed information, deletion refits, posterior-draw combination, +and influence arithmetic. The CPU `f64` path is the numerical reference; +GPU execution is a Rust device path and must pass identification-aware parity. +Python may validate and marshal only. + +The result envelope contains the exact TEPP run/snapshot/cutoff, fast-mlsirm +version and code revision, estimand/schema version, backend/precision, +convergence and identification diagnostics, posterior-draw coverage, context +membership fingerprint, post/topic/context identities, `D[d,k,l]`, and its +uncertainty evidence. A result for one context dimension cannot be copied to +another dimension. + +### LineageWeave consumer and persistence + +Use normalized objects such as `topic_model_run`, `topic_definition`, +`topic_activity_interval`, `topic_lineage_relation`, `topic_post_coordinate`, +`topic_context_membership`, `topic_influence_run`, and +`topic_post_context_influence`. Large result tables are partitioned by tenant +and modeled-period identity rather than one global time partition. Foreign +keys bind every influence row to the exact TEPP and fast-mlsirm artifacts. + +The API returns only persisted accepted rows after ABAC. It returns exact +ties, producer diagnostics, and provenance rather than computing or +renormalizing scores. The frontend renders an exact-value table alongside the +temporal topic view, uses text/pattern as well as color for topic state, and +supports keyboard, touch, reduced motion, narrow viewports, and screen readers. + +```mermaid +sequenceDiagram + participant Source as Authorized source snapshot + participant TEPP + participant MLS as fast-mlsirm Rust CPU/GPU + participant LW as LineageWeave projection + participant UI as Dashboard + Source->>TEPP: snapshot + cutoff + Event Lineage + memberships + TEPP-->>MLS: versioned posterior topic plausible values + MLS->>MLS: multilevel multiple-membership fit + MLS->>MLS: post deletion refits + observed-information D + MLS-->>LW: accepted topic_context_influence envelope + LW->>LW: exact contract, cutoff, digest, diagnostics, ABAC validation + LW-->>UI: temporal topics + level-specific tied influence rows +``` + +## Verification and acceptance + +The feature is not release-ready until all of the following are protected-main +evidence rather than a local or contract-only claim: + +1. TEPP simulation recovers known global topic identity, temporal prevalence, + relation effects, dormancy/reactivation, and cross-classified membership + effects with reported bias, RMSE, interval coverage, and posterior-draw + diagnostics; relation-aware splits prove no future leakage. +2. fast-mlsirm simulation recovers known context effects and ranks known + injected influential posts by the declared deletion estimand. Tests include + nested, crossed, weighted multiple-membership, time-varying membership, + sparse/unbalanced levels, missing observations, exact ties, and masked or + jointly influential cases. Correlation alone is not acceptance evidence. +3. Rust CPU worker-count determinism and CPU/GPU parity pass on the same + estimand. A GPU test proves actual device execution; fallback is explicit. +4. Contract tests reject wrong snapshot/cutoff/model/schema, missing posterior + draws, invented membership weights, non-convergence, unidentified + information blocks, non-finite influence, mixed producer runs, and partial + result sets. +5. Integration tests prove 3NF foreign-key integrity, idempotent replay, hot- + partition distribution, pre-aggregation ABAC, and no hidden-post leakage. +6. Storybook and browser screenshots cover populated, ties, dormant/reactivated, + multiple-membership, unavailable, narrow, dark, reduced-motion, keyboard, + and touch scenes. The exact-value table remains usable without the chart. +7. Public docstring, production line/branch, interaction, design-token, i18n, + and edge-case coverage remain 100% under repository gates. + +## Alternatives considered + +1. **LineageWeave fits a local dynamic topic model.** Rejected because TEPP + owns the temporal/relational posterior and measurement contract. +2. **Rank by posterior topic share, recency, engagement, or a weighted sum.** + Rejected because it ignores contextual influence or invents a construct and + weights. RankWeave may present an independently authorized retrieval rank, + but it is not this measurement. +3. **Use fast-mlsirm's current crossed binary kernel unchanged.** Rejected + because thresholding TEPP posterior coordinates into binary responses + discards uncertainty and changes the estimand. The producer must expose the + versioned topic-context influence contract above. +4. **Call the diagnostic business impact.** Rejected. Statistical influence + measures sensitivity of fitted topic/context parameters, not causal or + economic value. + +## Consequences + +The user receives a precise, reproducible answer to “which posts shape this +topic at this organizational level?” without arbitrary weights. Activation +depends on two upstream protected contracts and full recovery evidence; until +then the Dashboard truthfully shows why the result is unavailable rather than +inventing a ranking. + +## References (APA 7th) + +American Educational Research Association, American Psychological +Association, & National Council on Measurement in Education. (2014). +*Standards for educational and psychological testing*. American Educational +Research Association. + +Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. In *Proceedings +of the 23rd International Conference on Machine Learning* (pp. 113–120). +Association for Computing Machinery. https://doi.org/10.1145/1143844.1143859 + +Browne, W. J., Goldstein, H., & Rasbash, J. (2001). Multiple membership +multiple classification (MMMC) models. *Statistical Modelling, 1*(2), +103–124. https://doi.org/10.1177/1471082X0100100202 + +Fox, J.-P., & Glas, C. A. W. (2001). Bayesian estimation of a multilevel IRT +model using Gibbs sampling. *Psychometrika, 66*(2), 271–288. +https://doi.org/10.1007/BF02294839 + +Jin, I. H., Jeon, M., Schweinberger, M., Yun, J., & Lin, L. (2022). +Multilevel network item response modelling for discovering differences +between innovation and regular school systems in Korea. *Journal of the Royal +Statistical Society: Series C (Applied Statistics), 71*(5), 1225–1244. +https://doi.org/10.1111/rssc.12569 + +Molenaar, D., & Jeon, M. (2026). Regularized joint maximum likelihood +estimation of latent space item response models. *Psychometrika, 91*(1), +335–359. https://doi.org/10.1017/psy.2025.10068 + +Shi, L., & Chen, G. (2008). Case deletion diagnostics in multilevel models. +*Journal of Multivariate Analysis, 99*(9), 1860–1877. +https://doi.org/10.1016/j.jmva.2008.01.023 + +Zhang, D. C., & Lauw, H. (2022). Dynamic topic models for temporal document +networks. In *Proceedings of the 39th International Conference on Machine +Learning* (pp. 26281–26292). PMLR. +https://proceedings.mlr.press/v162/zhang22n.html diff --git a/docs/adr/README.md b/docs/adr/README.md index 6aca500f1..39c7c8bd3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -18,6 +18,8 @@ decision from them. | [`storybook-inventory.md`](../storybook-inventory.md) | [0118](0118-uiux-standard-guide-v3-design-overhaul.md), [0184](0184-ontology-provenance-explorer.md) | | [`POSTGRESQL_CONCURRENCY_REFERENCES.md`](../doctoring/POSTGRESQL_CONCURRENCY_REFERENCES.md) | [0204](0204-analysis-run-short-transaction-delivery.md) | | Evidence operations Dashboard (`/`) | [0206](0206-evidence-operations-dashboard.md) | +| [`temporal-topic-context-influence-research.md`](../temporal-topic-context-influence-research.md) | [0210](0210-temporal-topic-context-influence-dashboard.md) | +| [`python-mathematical-compute-boundary-audit.md`](../doctoring/python-mathematical-compute-boundary-audit.md) | [0208](0208-externalize-local-mathematical-compute.md) | [0011](0011-prov-o-standard-relations.md) and [0065](0065-prov-o-provenance-boundary.md) cite the dated W3C PROV-O and PROV-DM Recommendations (https://www.w3.org/TR/2013/REC-prov-o-20130430/ and https://www.w3.org/TR/2013/REC-prov-dm-20130430/). diff --git a/docs/doctoring/python-mathematical-compute-boundary-audit.md b/docs/doctoring/python-mathematical-compute-boundary-audit.md new file mode 100644 index 000000000..f1dbeee83 --- /dev/null +++ b/docs/doctoring/python-mathematical-compute-boundary-audit.md @@ -0,0 +1,64 @@ +# Python mathematical-compute boundary audit + +**Exact-head audit date:** 2026-08-25 +**Normative decision:** [ADR 0208](../adr/0208-externalize-local-mathematical-compute.md) + +This inventory names migration debt; it is not evidence that the current +Python paths satisfy the Rust/GPU requirement. + +## Product-boundary sources read + +- LineageWeave `ARCHITECTURE.md` and accepted ADRs 0003, 0132, 0145, + 0200, 0201, and 0205. This exact head has no standalone canonical PRD. +- TEPP `docs/product/prd-v0.4-approved.md`, whose approved TRSL-TM scope + owns temporal, relational, multilingual, topic, event, and trajectory + measurement. +- fast-mlsirm `docs/PRD.md`, whose reusable library scope owns + multilevel/contextual/longitudinal psychometric estimation, diagnostics, + recovery, and versioned artifacts rather than hosted product storage. +- RankWeave `README.md` and `ARCHITECTURE.md`. Its exact head has no PRD; + those files define the current fusion/ranking/evaluation responsibility. + A canonical RankWeave PRD is required before expanding that contract. + +| Current LineageWeave path | Local computation | Owner | Consumer replacement | Principal callers / tests | +|---|---|---|---|---| +| `lineageweave/channel_weight_estimation.py` | dichotomization, synthetic simulation, MLS2PLM input construction, expected item information and normalization | fast-mlsirm, conditional on TEPP anchor | versioned anchored-weight artifact; strict digest/convergence validation | estimation scripts, seed/server/rebuild paths; `tests/test_channel_weight_estimation.py`, estimator-script tests | +| `lineageweave/period_report.py` | response matrix, GRM/GPCM fit/FIPC/EAP, likelihood, category expectation, information ordering | fast-mlsirm | period-measurement artifact with item bank, scores, uncertainty, diagnostics | report ingestion and demo seed; period-report and report API tests | +| `lineageweave/leftover_pairs.py` | residual matrix, complete-case selection, SVD/Gabriel coordinates, distances, reconstruction, axis shares | fast-mlsirm | residual-interaction artifact with observed/expected identity and coverage | `period_report.py`, report ingestion/seed; `tests/test_leftover_pairs.py`, report tests | +| `lineageweave/embedding_client.py` and `backend/app/post_chat_ingestion.py` | cosine similarity, vector norms, maximum semantic score | RankWeave retrieval-score contract | ranked evidence envelope over ABAC-visible semantic units | reconstruction text channel and Global Ask retrieval; embedding/post-chat tests | +| `lineageweave/knowledge_graph.py` | random walk with restart, convergence delta, adaptive relevance cutoff | RankWeave graph-ranking contract | ranked-node artifact with contribution and convergence evidence | related-person/entity API paths; knowledge-graph tests | +| `lineageweave/reconstruct.py` | channel-weight renormalization, candidate-score fusion and minimum-score decision | RankWeave fusion; TEPP supplies independent lineage criterion | accepted edge-ranking artifact; LineageWeave persists selected edge and channel provenance | lineage rebuild/start/seed/server; reconstruct, persistence, API tests | +| `lineageweave/rankweave_client.py` | channel construction, token overlap, RRF weights and contribution arithmetic | RankWeave | strict ranking artifact exposing owner-computed contributions | `/api/rankings`, frontend Rankings; `tests/test_rankweave_client.py` and frontend tests | + +`lineageweave/post_evaluation.py` imports fast-mlsirm only for its published +judge contract and `to_irt_row` projection. It performs no fitted numerical +estimation, but remains in the transition guard because any direct owner-package +import must be reviewed before LineageWeave's final wire-only state. + +Validation-only uses of `math.isfinite` and database aggregation are not model +ownership and remain. Date ordering, counts, pagination, authorization, schema +validation, and presentation formatting also remain LineageWeave concerns. + +## Required owner contracts + +- **TEPP:** temporal-relational topic identity and Event-Lineage criterion + artifacts, with snapshot/cutoff, posterior uncertainty, evidence status, + lineage transitions, and deterministic Rust CPU/GPU execution evidence. +- **fast-mlsirm:** anchored channel information; GRM/GPCM fit, score and item + information; Gabriel residual interaction map; and topic-conditional + multiple-membership multilevel importance for business unit, PU, team, and + person, with recovery/RMSE and coverage evidence. +- **RankWeave:** Rust-backed similarity, graph ranking, fusion, contribution, + evaluation, and policy-selection artifacts. Its present Python calculation + core is the correct product owner but not the final execution architecture. + +## Persistence and UI blast radius + +Owner envelopes require normalized run/artifact tables keyed by analysis run, +owner contract version, model version, source snapshot SHA-256, knowledge +cutoff, and authorization scope. Topic, membership, level-specific importance, +uncertainty, and source-post evidence occupy separate child rows; arrays or +labels do not replace foreign keys. Dashboard and post detail endpoints read +only accepted persisted rows and preserve source-post ABAC. Storybook covers +accepted, pending, failed, stale-digest, non-converged, hidden-evidence, and +multiple-membership cases before UI activation. diff --git a/docs/temporal-topic-context-influence-research.md b/docs/temporal-topic-context-influence-research.md new file mode 100644 index 000000000..05a80c68a --- /dev/null +++ b/docs/temporal-topic-context-influence-research.md @@ -0,0 +1,72 @@ +# Temporal topic and context-influence evidence brief + +## Research question + +How can LineageWeave present time-aware, lineage-aware topics and identify the +posts that influence topic estimates at business-unit, PU, team, and person +levels without keyword rules or arbitrary weights? + +## Search and source selection + +- Concepts: dynamic topic models, temporal document networks, multilevel IRT, + multiple-membership multiple-classification, latent-space IRT, and + multilevel case-deletion diagnostics. +- Priority: peer-reviewed primary papers, official proceedings, accepted + author manuscripts, TEPP's approved PRD/ADR, and fast-mlsirm's normative + ADR/research register. +- Excluded as authorities: review-only pages, vendor summaries, lexical topic + matching, engagement ranking, and methods that do not preserve posterior, + time, relation, or membership identity. + +## Findings + +### Temporal topic identity and document relations + +Blei and Lafferty (2006) establish state-space topic evolution rather than +independent time-bin models. Zhang and Lauw (2022) jointly model temporal +document topics and network structure; this directly supports consuming +explicit Event Lineage as relational evidence rather than matching topic +labels after fitting. TEPP PRD v0.4 and ADR 0012 combine those concerns in the +TRSL-TM producer boundary, including global topic identity, posterior +coordinates, multiple clocks, relations, and cross-classified membership. + +The evidence does not establish that every relation is causal or that a +reactivated topic is newly born. Those states and lineage events must arrive +from a versioned TEPP result. + +### Multilevel and multiple-membership measurement + +Fox and Glas (2001) show why latent rather than observed scores should be +modeled jointly with cluster effects and measurement error. Browne, Goldstein, +and Rasbash (2001) define crossed and weighted multiple-membership structures. +Jin et al. (2022) demonstrate a multilevel network item-response model that +can expose differences missed by conventional multilevel models. These papers +support distinct business-unit, PU, team, and person dimensions with explicit +time-valid membership; they do not support inferring equal weights when the +source has none. + +### “Important post” estimand + +Shi and Chen (2008) define case-deletion diagnostics at multiple levels for +fixed and random parameters. ADR 0210 therefore gives importance the bounded +name **model influence** and defines it as observed-information-scaled change +in the topic-by-context estimate after deleting the complete post +observation. This answers sensitivity of the fitted model, not business value +or causality. Molenaar and Jeon (2026) support recovery-tested regularized JML +for latent-space IRT, but do not by themselves validate this product-specific +construct; the fast-mlsirm producer must implement and recover the exact +versioned influence estimand before LineageWeave activates it. + +## Architecture consequence + +LineageWeave is a strict consumer. TEPP owns temporal/relational topic +posterior arithmetic. fast-mlsirm owns multilevel multiple-membership fitting, +observed information, deletion refits, posterior-draw combination, and CPU/GPU +parity in Rust. LineageWeave persists exact accepted artifacts, applies ABAC, +and renders tied exact values; it adds no threshold, fallback score, or local +numerical formula. + +## Primary sources (APA 7th) + +See [ADR 0210](adr/0210-temporal-topic-context-influence-dashboard.md#references-apa-7th) +for the full APA 7 bibliography and exact architecture mapping. diff --git a/tests/test_math_boundary_inventory.py b/tests/test_math_boundary_inventory.py new file mode 100644 index 000000000..d4e07eb8c --- /dev/null +++ b/tests/test_math_boundary_inventory.py @@ -0,0 +1,47 @@ +"""Freeze known Python numerical ownership while ADR 0208 moves it upstream.""" + +from __future__ import annotations + +import ast +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +NUMERICAL_OWNER_MODULES = {"fast_mlsirm", "numpy", "rankweave", "scipy", "sklearn"} +KNOWN_LOCAL_NUMERICAL_FILES = { + "lineageweave/channel_weight_estimation.py", + "lineageweave/leftover_pairs.py", + "lineageweave/period_report.py", + "lineageweave/post_evaluation.py", + "lineageweave/rankweave_client.py", + "lineageweave/reconstruct.py", +} + + +def _numerical_import_files() -> set[str]: + """Return production Python files importing a numerical owner package.""" + + found: set[str] = set() + for base in (ROOT / "lineageweave", ROOT / "backend" / "app"): + for path in base.rglob("*.py"): + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + imports = { + alias.name.split(".", 1)[0] + for node in ast.walk(tree) + if isinstance(node, ast.Import) + for alias in node.names + } + imports.update( + node.module.split(".", 1)[0] + for node in ast.walk(tree) + if isinstance(node, ast.ImportFrom) and node.module + ) + if imports & NUMERICAL_OWNER_MODULES: + found.add(path.relative_to(ROOT).as_posix()) + return found + + +def test_no_new_local_numerical_owner_imports() -> None: + """Require an ADR 0208 inventory update before local numerical scope grows.""" + + assert _numerical_import_files() == KNOWN_LOCAL_NUMERICAL_FILES