feat(dml): DMLDiD survey-design and cluster support on both lanes (DML survey PR)

[igerber]

Summary

• DMLDiD gains full survey-design and cluster support on BOTH lanes: survey_design= on fit() (pweight-only full-design TSL — weights/strata/PSU/FPC) and cluster= on the constructor (coarser-than-unit clustering).
• Declared designs enter the ESTIMATOR (CS precedent): Hájek-weighted p̂/λ̂/θ̂, weighted Ĝ₂λ (a weights= extension of the _dr_scores RCS slope family, bit-identical when absent), sample_weight into both nuisance learners (user learner objects must accept it by keyword — up-front TypeError otherwise), and weighted influence-function payloads.
• Cross-fitting becomes PSU-COHESIVE when the design's PSU is strictly coarser than the sampling unit (clustered-DML prescription; assign_folds(cluster_ids=)), with an explicit Chang I_kz^c fold-composition guard on the RCS lane, an effective-fold-count reduction to n_psu (warned; effective_n_folds provenance on results/summary/to_dict/repr) when 2 ≤ n_psu < n_folds, and a stratified-fold fallback only for single-PSU designs (whose clustered variance is NaN either way).
• Per-cell SEs are PSU-gated exactly like CallawaySantAnna (_cluster_robust_se_from_per_gt_if; NaN propagates as the deliberate unidentified-variance signal on a RETAINED cell); analytical inference uses df = n_PSU − n_strata t-statistics; the survey multiplier bootstrap and <2-PSU NaN contracts activate via the inherited CS mixins.
• Aggregation kits now carry the producer's bootstrap label, so post-fit replays brand the <2-PSU warning with the fitting estimator instead of hardcoded "CallawaySantAnna" (_BOOTSTRAP_LABEL ClassVar → plain annotation across its five hosts).
• Bare cluster= keeps the moment kernels unweighted (folds/variance/df_inference only); zero_weight_mass joins the DMLDiD skip vocabulary; universal-base reference cells gate on the weighted cohort mass; aggregate('total') fails closed on declared-survey fits.
• Replicate-weight designs fail closed (tracked as a TODO.md Actionable row). Downstream consumer surfaces synced (reporting helpers, MMM scale hint, practitioner template, Business/Diagnostic report narratives, variance-conventions CI-pinned pair, survey-theory rosters, all four guides, choosing_estimator/decision-tree/survey-roadmap/migration/v4-design, api RST, doc-deps mappings).

Methodology references (required if estimator / math changes)

• Method name(s): Chang (2020) DML DiD (Cases 1–2) under complex survey designs — a documented LIBRARY EXTENSION (Chang assumes i.i.d. sampling; clustering/weighting are not discussed in the paper). Design-based variance via Binder (1983) TSL on influence functions (the library's shared survey machinery); clustered cross-fitting per the DoubleML clustered-DML prescription.
• Paper / source link(s): Chang, N.-C. (2020), The Econometrics Journal 23(2), 177–191; Binder (1983); Korn & Graubard (1990) survey df; docs/methodology/REGISTRY.md "DMLDiD" survey Notes; docs/methodology/survey-theory.md.
• Any intentional deviations from the source (and why): the entire survey lane is REGISTRY-noted as a library extension — Hájek-weighted moments, the weighted-Ĝ₂λ plug-in (Theorem 2's coverage claim does not carry over on the weighted-λ RCS lane), PSU-cohesive folds replacing the D/D×T fold-stratification MECHANISM (Chang's fold-composition REQUIREMENT preserved by an explicit guard), and df = n_PSU − n_strata t-inference. No external oracle exists (DoubleML has no survey support; R did:: is survey-naive) — validation is the library's survey invariant battery plus direct compute_survey_if_variance kernel cross-checks.

Validation

• Tests added/updated: tests/test_survey_dml.py (new, 62 tests: invariant battery — uniform-weights ≡ unweighted, scale invariance, FPC-shrinks-SE; per-cell SE == compute_survey_if_variance; cluster-wiring branches; fold cohesion/reduction/boundary pins; spy-learner weight-threading; zero_weight_mass; survey bootstrap + replay-label + legacy-kit fallback; rejections; df threading/rendering; aggregation masses incl. a >2⁵³ cohort-key regression), tests/test_dr_scores.py (weighted-slope family), tests/test_dml_did.py (cluster mutation-roster case), tests/test_guides.py (fit-signature pin), tests/test_variance_conventions.py (inventory rows, CI-pinned with the doc twin).
• Backtest / simulation / notebook evidence (if applicable): N/A — no notebook changes; existing DMLDiD/CS suites pass unchanged (no-survey fits bit-identical).

Security / privacy

• Confirm no secrets/PII in this PR: Yes

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 74718bf0b2b43d2b3813736d0d78704d6746cd20

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings.

Executive Summary

• Affects Chang Case 1 panel and Case 2 repeated-cross-section DMLDiD.
• Weighted moments, augmented IFs, PSU-cohesive folds, and survey-df inference match the documented library extension.
• NaN inference, control selection, fold composition, aggregation, and bootstrap paths are handled consistently.
• Remaining limitations are fail-closed and tracked.

Methodology

• P3 — Documented deviation. Chang assumes i.i.d. sampling; survey weighting, clustered folds, Binder TSL variance, and finite-df inference are explicitly labeled as a library extension in docs/methodology/REGISTRY.md:L3028-L3092. The implementation follows that documented contract in diff_diff/dml_did.py:L1312-L1404 and diff_diff/dml_did.py:L1695-L1793.
  • Impact: Informational; Chang’s Theorem 2 coverage is correctly not claimed for weighted RCS.
  • Concrete fix: None required.

No undocumented methodology mismatch, missing assumption guard, or incorrect SE formula found.

Code Quality

No findings. New inference uses safe_inference() consistently, and cluster is covered by the shared parameter-introspection lifecycle.

Performance

No findings. Survey variance and bootstrap reuse existing shared IF/PSU machinery.

Maintainability

No findings. Survey resolution, aggregation, bootstrap, and variance logic delegate to established shared helpers.

Tech Debt

• P3 — Tracked: Replicate-weight designs fail closed and are recorded in TODO.md:L41.
• P3 — Tracked: Survey/RC totals remain fail-closed and are recorded in DEFERRED.md:L46.
• P3 — Tracked: Bootstrap survey-df metadata provenance is recorded in TODO.md:L40.

These do not block approval.

Security

No findings. No secrets were introduced, and foreign learner representations/errors are sanitized before persistence.

Documentation/Tests

No findings. Coverage includes both estimator lanes, weight invariance, FPC, direct survey-variance checks, zero-weight cells, PSU folds, bootstrap, df propagation, and aggregation. Static syntax validation passed; runtime tests could not be executed because the review environment lacks pytest/numpy.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: f0c5f8b7ae7986d1ad149d3f6c89434d8ce961fa

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings. One P2 provenance issue remains.

Executive Summary

• Affects Chang Case 1 panel and Case 2 repeated-cross-section DMLDiD.
• Weighted moments, PSU-cohesive folds, Binder TSL variance, and finite-df inference match the documented library extension.
• NaN inference, zero-weight cells, control selection, aggregation, and bootstrap paths are handled consistently.
• Survey weight metadata is incorrectly recomputed from normalized rather than raw weights.
• Runtime tests were unavailable; all 19 changed Python files parsed successfully.

Methodology

• Severity: P3 — Documented deviation
  • Impact: Survey weighting, clustered cross-fitting, weighted (G_{2\lambda}), and survey-df inference extend Chang’s i.i.d. theory. These deviations are explicitly documented in docs/methodology/REGISTRY.md:L3028-L3092.
  • Concrete fix: None required.

No undocumented methodology mismatch, missing assumption check, or incorrect variance/SE formula found.

Code Quality

• Severity: P2 — Raw survey metadata is overwritten with normalized weights
  diff_diff/dml_did.py:L1950-L1956
  • Impact: compute_survey_metadata() expects original pre-normalization weights, but receives resolved_survey_unit.weights. Consequently, survey_metadata.sum_weights and weight_range report normalized values; on RCS fits, sum_weights generally becomes the row count. Estimates and inference are unaffected, but serialized provenance is incorrect.
  • Concrete fix: Retain the raw survey weights, collapse them to one value per unit for panel fits, and pass that raw unit-level vector to compute_survey_metadata(). Add panel and RCS tests using non-unit-scale weights that assert sum_weights and weight_range.

Performance

No findings.

Maintainability

No findings. The implementation appropriately reuses shared survey, aggregation, bootstrap, and inference helpers.

Tech Debt

• Severity: P3 — Tracked: Replicate-weight DMLDiD designs fail closed and are recorded in TODO.md:L41.
• Severity: P3 — Tracked: Bootstrap survey-df provenance is recorded in TODO.md:L40.
• Severity: P3 — Tracked: Survey/RCS totals remain fail-closed in DEFERRED.md:L46.
  • Impact: Informational; these limitations do not produce silent incorrect estimates.
  • Concrete fix: None required for this PR.

Security

No findings. No secrets or unsafe persistence of user learner objects were identified.

Documentation/Tests

No additional findings. Coverage includes both estimator lanes, weight invariance, direct survey-variance checks, zero-weight cells, clustered folds, bootstrap behavior, df propagation, and aggregation masses.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: eafd6270b56a809f30a2286eb37c0b08eb7c91d8

---

Overall Assessment

✅ Looks good — no unmitigated P0 or P1 findings.

Executive Summary

• Affects Chang Case 1 panel and Case 2 repeated-cross-section DMLDiD.
• Survey weighting, PSU-cohesive folds, Binder TSL variance, and finite-df inference match the documented library extension.
• Zero-weight cells, control selection, NaN inference, aggregation, and bootstrap paths are handled consistently.
• The prior raw survey-metadata provenance issue is resolved for panel, RCS, and injected-cluster designs.
• Runtime tests were unavailable because pytest/numeric dependencies are not installed; all 19 changed Python files parsed successfully.

Methodology

• Severity: P3 — Documented deviation
  • Impact: Hájek-weighted moments, weighted (G_{2\lambda}), clustered cross-fitting, Binder variance, and survey-df inference extend Chang’s i.i.d. theory.
  • Concrete fix: None required. These deviations are explicitly documented in docs/methodology/REGISTRY.md:L3028-L3092.

No undocumented methodology mismatch, missing assumption check, or incorrect variance/SE formula found.

Code Quality

No findings. The prior normalized-weight metadata issue is fixed in diff_diff/dml_did.py:L1878-L1891 and diff_diff/dml_did.py:L1966-L1983, with regression tests at tests/test_survey_dml.py:L478-L505.

Performance

No findings.

Maintainability

No findings. The implementation reuses shared survey, aggregation, bootstrap, and safe-inference helpers.

Tech Debt

• Severity: P3 — Tracked limitations
  • Impact: Replicate-weight designs, bootstrap replay df provenance, and survey/RCS totals remain unavailable or provenance-limited.
  • Concrete fix: None required for this PR; tracked in TODO.md:L41-L42 and DEFERRED.md:L46.

Security

No findings. Foreign learner objects and their potentially sensitive error representations are not persisted.

Documentation/Tests

No findings. Coverage includes both estimator lanes, raw-weight metadata, weighting invariants, direct variance-kernel checks, PSU folds, zero-weight cells, bootstrap behavior, finite-df inference, and aggregation masses.