diff --git a/CHANGELOG.md b/CHANGELOG.md index fb7128241..01e812558 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -83,6 +83,46 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 expression), pointing at `full_model_window=True` as the mroi route. ### Changed +- **Family-wide `anticipation` domain validation ([M-144], landing at 4.0 — + the locked ladder's next release, with no warn-then-error window per the + M-096/M-142 precedent that validation tightenings ship immediately; + retires the TODO "library-wide anticipation domain validation" row).** All + nine anticipation-taking estimators (`CallawaySantAnna`, `SunAbraham`, + `ImputationDiD`, `TwoStageDiD`, `StackedDiD`, `ContinuousDiD`, + `EfficientDiD`, `WooldridgeDiD`, `SpilloverDiD`) now validate + `anticipation` at construction via the shared + `utils.validate_anticipation` (non-negative integer; `bool` rejected; + `set_params` transactional) AND re-check it on the fit path — the uniform + direct-mutation defense, in the assignment form that also normalizes + numpy scalars to built-in `int`. Previously seven of the nine accepted + anything, and an out-of-domain window silently changed the ESTIMAND: + measured, `CallawaySantAnna(anticipation=-1)` moved the overall ATT by + −85% and flipped its sign under `control_group="not_yet_treated"`; + `anticipation=True` fit bit-identically to `1` (a silent one-period + window); `SunAbraham(anticipation=1.5)` returned `att=nan` without + raising. `0` stays legal. The one break of previously-CORRECT code: + whole-valued floats (`anticipation=1.0`, `np.float64(1.0)`) previously + fit bit-identically to their integer value on + CS/SunAbraham/ImputationDiD/TwoStageDiD/EfficientDiD/WooldridgeDiD and + now raise — use the int (StackedDiD already crashed on floats via an + incidental `range()` TypeError, now a clear constructor `ValueError`; + ContinuousDiD's float behavior was fixture-dependent). Also visible: + accepted numpy integers are retyped — the public `anticipation` + attribute and `get_params()["anticipation"]` are now always built-in + `int`, not a numpy scalar; `WooldridgeDiD`'s message text changed to the + shared wording, its `None`/str raw `TypeError` became `ValueError`, and + its constructor error ordering moved (bad + `bootstrap_weights`/`vcov_type`/`df_convention` now report before a bad + `anticipation`); `SpilloverDiD`'s raise moved from fit to construction + (the fit-time re-check is retained, ordered before the ref-period + arithmetic), and its negative-int message dropped the `(type ...)` + suffix (shared text). `EfficientDiD.hausman_pretest` normalizes its own + `anticipation` argument (an unsigned numpy scalar previously wrapped its + event-time arithmetic and silently degraded the pretest to an all-NaN + inconclusive result). The deprecated `StaggeredTripleDifference` stays + construction-permissive by design (fit-validated via the shared engine). + Both LLM guides note the domain; policy suite: + `tests/test_anticipation_policy.py`. - **DiagnosticReport's event-study-gated checks now consume the post-fit `results.aggregate('event_study')` surface** (the 3.9 M-020 family; retires the TODO "diagnostic_report ES-gated checks" row): on a modern diff --git a/TODO.md b/TODO.md index 70cad87a4..81cbad4b5 100644 --- a/TODO.md +++ b/TODO.md @@ -24,7 +24,6 @@ Related tracking surfaces: | Post-fit `aggregate()` for the staggered DDD container: `StaggeredTripleDiffResults` carries no `AggregationMixin`, which is why the phase-3(b) merge had to carry fit-time `aggregate=`/`balance_e=` onto the surviving `TripleDifference` (rows M-140/M-141) as the ONE documented exception to the section-6 aggregate-postfit program. Porting the container onto the M-122 aggregation contract retires both rows; note the bootstrapped-fit recompute levels will need draw retention or a fail-closed relay, the same problem tracked for CS/EfficientDiD/ImputationDiD. Until it lands, the DDD docs deliberately keep teaching the fit-time kwarg (the canonical route there) | `diff_diff/staggered_triple_diff_results.py`, `diff_diff/aggregation.py`, `docs/api/triple_diff.rst`, `docs/tutorials/08_triple_diff.ipynb` | 3(b) | Heavy | Medium | | Staggered-DDD power support: `simulate_power`/`simulate_mde`/`simulate_sample_size` now REJECT a staggered-configured `TripleDifference` (both registered DDD generators emit 2x2x2 data and fit with `(group, partition, post)`, so a staggered config would be simulated under the wrong design). Support needs a staggered DDD DGP profile plus fit-kwargs builder, and a decision on whether the mode is selected by profile or by the estimator's own config | `diff_diff/power.py` | 3(b) | Mid | Low | | Bootstrap-`seed` provenance on multiplier-bootstrap results containers: neither `StaggeredTripleDiffResults` nor `CallawaySantAnnaResults` carries the `seed` that generated its bootstrap SEs / p-values / sup-t bands, so a serialized result cannot report the random configuration behind its inference. NOT a 3(b) regression - `seed` reaches the engine and `get_params()` correctly (same seed reproduces the SE bit-exactly, a different seed moves it), the gap is results-object observability only, it predates the merge, and both containers inherit it from the shared `CallawaySantAnnaBootstrapMixin`. Add `seed` (and consider `n_bootstrap`/`bootstrap_weights`/`cband`) to BOTH containers plus `to_dict()`, with seeded and unseeded pins; sequence it with the M-014 container unification rather than schema-changing one container mid-merge. Precedent for exposing it: `ContinuousDiDResults`, `EfficientDiDResults`, `SyntheticDiDResults` already do | `diff_diff/staggered_triple_diff_results.py`, `diff_diff/staggered_results.py` | 3(b) | Quick | Low | -| Library-wide `anticipation` domain validation: `TripleDifference` now rejects non-integral / negative / `bool` windows at construction (phase 3(b)) because the value feeds BOTH the base-period rule and the not-yet-treated threshold, so `anticipation=-1` silently makes the universal base period `g` (already treated) and admits cohorts treated at the evaluation period as clean controls. Only `spillover.py` and `wooldridge.py` validate it today (and neither rejects `bool`, which coerces to a silent one-period window); `CallawaySantAnna`, `SunAbraham`, `ImputationDiD`, `TwoStageDiD`, `StackedDiD`, `ContinuousDiD`, `EfficientDiD` and the deprecated `StaggeredTripleDifference` do not. The shared validator now EXISTS - `utils.validate_anticipation`, adopted by `TripleDifference.__init__` and by the staggered engine (so `StaggeredTripleDifference` fails closed at fit too); aligning the remaining seven estimators is a matter of calling it from each constructor | `diff_diff/staggered.py`, `diff_diff/sun_abraham.py`, `diff_diff/imputation.py`, `diff_diff/two_stage.py`, `diff_diff/stacked_did.py`, `diff_diff/continuous_did.py`, `diff_diff/efficient_did.py`, `diff_diff/spillover.py`, `diff_diff/wooldridge.py` | 3(b) | Mid | Medium | | `ContinuousDiD.pscore_trim` still validates `0.0 <= x < 0.5`, i.e. it admits `0`, while `TripleDifference` tightened to `0 < x < 0.5` in phase 3(b) (row M-142) on the grounds that `trim=0` disables the `np.clip(pscore, trim, 1-trim)` overlap guard keeping the `1/(1-p)` weights finite. The same argument applies to ContinuousDiD; aligning it was out of scope for a DDD merge and is recorded in the REGISTRY staggered-mode Note rather than left as silent drift. `TripleDifference` additionally gained a TYPE guard in 3(b) (reject bool/non-real-scalar/non-finite BEFORE the range comparison) because a bare `0 < x < 0.5` raises an incidental `TypeError` on `None`/str/complex/list, an ambiguous-truth error on a multi-element array, and silently ACCEPTS a 1-element array as the parameter; `ContinuousDiD`'s `np.isfinite(self.pscore_trim) and ...` has the same hole. Aligning both is one change - promote the guard to a shared `utils.validate_pscore_trim(value, *, allow_zero)` alongside `validate_n_bootstrap` rather than copying it | `diff_diff/continuous_did.py`, `diff_diff/utils.py` | 3(b) | Quick | Low | | Staggered-mode cluster-robust ANALYTICAL SEs: `cluster=` raises in `TripleDifference`'s staggered mode (and is accepted-then-ignored on the deprecated class), so clustered inference there is bootstrap-only. Implementing a clustered analytical path for the GMM-combined influence function would let the raise become a real lane | `diff_diff/_staggered_triple_diff_engine.py` | 3(b) | Heavy | Low | | diagnostic_report admission for `EventStudyResults` surfaces (the TWFE event-study mode + `aggregate('event_study')` containers): DiagnosticReport/BusinessReport now REJECT the surface explicitly (Phase 3(a); previously a silent zero-check report / all-null headline) and practitioner_next_steps serves the generic fall-through - admission needs source-aware routing (the type-name-keyed `_APPLICABILITY`/`_HANDLERS` registries cannot discriminate the unified container's producers) and a scalar-vs-per-period headline design; MPD-native results received {parallel_trends, pretrends_power, sensitivity, bacon, design_effect} | `diff_diff/diagnostic_report.py`, `diff_diff/business_report.py`, `diff_diff/practitioner.py` | 3(a) | Mid | Medium | diff --git a/diff_diff/_staggered_triple_diff_engine.py b/diff_diff/_staggered_triple_diff_engine.py index f8707a466..3fa1235c8 100644 --- a/diff_diff/_staggered_triple_diff_engine.py +++ b/diff_diff/_staggered_triple_diff_engine.py @@ -64,10 +64,13 @@ class _StaggeredTripleDiffEngineMixin: supply the constructor attributes and the CS aggregation/bootstrap mixins this core calls. The annotations below exist because mypy type-checks this class independently of its hosts (`attr-defined` is not disabled) - they are - declarations, never assignments. + declarations, never assignments — with ONE exception: `anticipation` is + re-assigned by the core's fit-time re-validation (the mutation-defense + re-check normalizes it to a Python int; see `_fit_staggered_core`). """ - # Constructor attributes read from the host class. + # Constructor attributes read from the host class. (`anticipation` is + # additionally RE-ASSIGNED at fit — the validate-and-normalize re-check.) estimation_method: str control_group: str alpha: float @@ -178,8 +181,10 @@ def _fit_staggered_core( # the deprecated StaggeredTripleDifference (whose 3.x API SHAPE is # frozen through removal - that freeze was never a licence to emit # silently-biased numbers) and direct attribute mutation on either - # class, which bypasses __init__ and set_params alike. - validate_anticipation(self.anticipation) + # class, which bypasses __init__ and set_params alike. The + # assignment form also normalizes a numpy scalar to a Python int + # before any `g - 1 - anticipation` arithmetic can overflow. + self.anticipation = validate_anticipation(self.anticipation) from diff_diff.survey import ( _resolve_survey_for_fit, _validate_unit_constant_survey, diff --git a/diff_diff/continuous_did.py b/diff_diff/continuous_did.py index 2521db8cc..21610f1c8 100644 --- a/diff_diff/continuous_did.py +++ b/diff_diff/continuous_did.py @@ -45,7 +45,7 @@ build_unit_first_row_index, compute_survey_vcov, ) -from diff_diff.utils import safe_inference, validate_n_bootstrap +from diff_diff.utils import safe_inference, validate_anticipation, validate_n_bootstrap if TYPE_CHECKING: from diff_diff.survey import ResolvedSurveyDesign, SurveyDesign @@ -182,7 +182,8 @@ class ContinuousDiD(_ContinuousDiDAggregationMixin, BaseEstimator): ``P(D=d_L) > 0``) and no never-treated units present. Single-cohort only (multi-cohort and ``covariates=`` raise ``NotImplementedError``). anticipation : int, default=0 - Number of periods of treatment anticipation. + Number of periods of treatment anticipation. Must be a + non-negative integer; ``bool`` is rejected. base_period : str, default="varying" ``"varying"`` or ``"universal"``. alpha : float, default=0.05 @@ -310,12 +311,18 @@ def __init__( self._validate_constrained_params() def _validate_constrained_params(self) -> None: - """Validate control_group, base_period, and estimation_method values.""" + """Validate control_group, base_period, and estimation_method values. + + Also validates ``anticipation`` and re-assigns it as a normalized + Python ``int`` — idempotent on an already-normalized value, so a + re-run never changes fitted config. + """ if self.control_group not in self._VALID_CONTROL_GROUPS: raise ValueError( f"Invalid control_group: '{self.control_group}'. " f"Must be one of {self._VALID_CONTROL_GROUPS}." ) + self.anticipation = validate_anticipation(self.anticipation) if self.base_period not in self._VALID_BASE_PERIODS: raise ValueError( f"Invalid base_period: '{self.base_period}'. " @@ -442,6 +449,14 @@ def fit( else: aggregate = None + # Fit-time re-check: __init__ and set_params validate eagerly, so + # this only catches DIRECT attribute mutation (est.anticipation = ...) + # — an out-of-domain value silently changes the ESTIMAND. The + # assignment also re-normalizes a mutated numpy scalar to int. Placed + # AFTER the deprecation shim so a caller who both mutated and passed + # a deprecated argument still sees the FutureWarning before the raise. + self.anticipation = validate_anticipation(self.anticipation) + # 1. Validate & prepare _VALID_AGGREGATES = (None, "dose", "eventstudy") if aggregate not in _VALID_AGGREGATES: diff --git a/diff_diff/efficient_did.py b/diff_diff/efficient_did.py index 8aab3f720..79e60c590 100644 --- a/diff_diff/efficient_did.py +++ b/diff_diff/efficient_did.py @@ -64,7 +64,7 @@ compute_omega_star_nocov, enumerate_valid_triples, ) -from diff_diff.utils import safe_inference, validate_n_bootstrap +from diff_diff.utils import safe_inference, validate_anticipation, validate_n_bootstrap # Re-export for convenience __all__ = ["EfficientDiD", "EfficientDiDResults", "EDiDBootstrapResults"] @@ -290,7 +290,8 @@ class EfficientDiD(EfficientDiDBootstrapMixin, _EfficientAggregationMixin, BaseE Random seed for reproducibility. anticipation : int, default 0 Number of anticipation periods (shifts the effective treatment - boundary forward by this amount). When combined with + boundary forward by this amount). Must be a non-negative + integer; ``bool`` is rejected. When combined with ``control_group="last_cohort"``, also trims the pseudo-control period set at ``t >= last_g - anticipation`` (see REGISTRY.md). sieve_k_max : int or None @@ -375,7 +376,12 @@ def __init__( self._validate_params() def _validate_params(self) -> None: - """Validate constrained parameters.""" + """Validate constrained parameters. + + Also validates ``anticipation`` and re-assigns it as a normalized + Python ``int`` — idempotent on an already-normalized value, so the + fit-time re-run never changes fitted config. + """ if self.pt_assumption not in ("all", "post"): raise ValueError(f"pt_assumption must be 'all' or 'post', got '{self.pt_assumption}'") if self.control_group not in ("never_treated", "last_cohort"): @@ -383,6 +389,7 @@ def _validate_params(self) -> None: f"control_group must be 'never_treated' or 'last_cohort', " f"got '{self.control_group}'" ) + self.anticipation = validate_anticipation(self.anticipation) valid_weights = ("rademacher", "mammen", "webb") if self.bootstrap_weights not in valid_weights: raise ValueError( @@ -1544,7 +1551,8 @@ def hausman_pretest( cluster : str, optional Cluster column for cluster-robust covariance. anticipation : int - Anticipation periods. + Anticipation periods. Must be a non-negative integer; ``bool`` + is rejected. control_group : str ``"never_treated"`` or ``"last_cohort"``. alpha : float @@ -1557,6 +1565,12 @@ def hausman_pretest( ------- HausmanPretestResult """ + # The classmethod uses `anticipation` in its OWN event-time + # arithmetic (`e < -ant` below), not just forwarding to the two + # constructed estimators — validate and normalize it here so an + # unsigned numpy scalar cannot wrap the comparison. + anticipation = validate_anticipation(anticipation) + # Fit under both assumptions (analytical SEs only, no bootstrap) common_kwargs = dict( cluster=cluster, diff --git a/diff_diff/guides/llms-full.txt b/diff_diff/guides/llms-full.txt index 3bf811e78..0bfc36cbb 100644 --- a/diff_diff/guides/llms-full.txt +++ b/diff_diff/guides/llms-full.txt @@ -206,7 +206,7 @@ Callaway-Sant'Anna (2021) estimator for staggered DiD with heterogeneous treatme ```python CallawaySantAnna( control_group: str = "never_treated", # "never_treated" or "not_yet_treated" - anticipation: int = 0, # Anticipation periods + anticipation: int = 0, # Anticipation periods (non-negative integer; bool rejected) estimation_method: str = "dr", # "dr", "ipw", or "reg" alpha: float = 0.05, cluster: str | None = None, # Cluster col; activates CR1 on the IF via synthesized SurveyDesign(psu=col). None → per-unit IF. @@ -383,7 +383,7 @@ Sun-Abraham (2021) interaction-weighted estimator for staggered DiD. ```python SunAbraham( control_group: str = "never_treated", # "never_treated" or "not_yet_treated" - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, cluster: str | None = None, # Defaults to unit-level clustering (dropped on explicit vcov_type='hc2' / 'classical') n_bootstrap: int = 0, # 0 = analytical cluster-robust SEs @@ -431,7 +431,7 @@ Borusyak-Jaravel-Spiess (2024) imputation DiD estimator. Efficient estimator pro ```python ImputationDiD( - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, cluster: str | None = None, # Defaults to unit-level clustering vcov_type: str = "hc1", # {"hc1"} only — IF-based variance per Borusyak et al. (2024) Theorem 3 @@ -484,7 +484,7 @@ Gardner (2022) two-stage DiD estimator. Point estimates match ImputationDiD; use ```python TwoStageDiD( - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected alpha: float = 0.05, cluster: str | None = None, n_bootstrap: int = 0, @@ -540,7 +540,7 @@ SpilloverDiD( conley_lag_cutoff: int | None = None, cluster: str | None = None, alpha: float = 0.05, - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected event_study: bool = False, # Wave C: per-event-time × ring decomposition (Butts Table 2) horizon_max: int | None = None, # Bin event-times outside [-H,+H] into endpoint pools (event-study mode); H>=1 or None — H=0 rejected (use event_study=False for aggregate spec) rank_deficient_action: str = "warn", @@ -709,6 +709,8 @@ TripleDifference( **Alias:** `DDD` +Staggered mode (`fit(first_treat=...)`) adds constructor params not shown here — `anticipation` (non-negative integer; `bool` rejected), `control_group`, `base_period`, `n_bootstrap` (non-default values of these FOUR are rejected in 2x2x2 mode; the defaults are fine), plus `bootstrap_weights`/`seed`/`cband` (accepted but inert in 2x2x2 — unreachable without `n_bootstrap > 0`, per the M-013 ledger note). + **fit() parameters:** ```python @@ -743,7 +745,7 @@ ContinuousDiD( num_knots: int = 0, # Interior knots dvals: np.ndarray | None = None, # Custom dose evaluation grid control_group: str = "never_treated", # "never_treated", "not_yet_treated", or "lowest_dose" (Remark 3.1, P(D=0)=0) - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected base_period: str = "varying", # "varying" or "universal" alpha: float = 0.05, n_bootstrap: int = 0, @@ -1047,7 +1049,7 @@ StackedDiD( clean_control: str | None = None, # DEPRECATED alias for control_group= (M-043; FutureWarning, removed in 4.0) cluster: str = "unit", # "unit" or "unit_subexp" alpha: float = 0.05, - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected rank_deficient_action: str = "warn", vcov_type: str = "hc1", # {"hc1","hc2_bm"}; classical/hc2 rejected (intrinsically clustered), conley deferred. survey_design=... requires hc1 balance: str = "none", # {"none","entropy"}; "entropy" = CBWSDID covariate balancing (Ustyuzhanin 2026), requires fit(covariates=[...]) + weighting="aggregate", no survey_design @@ -1099,7 +1101,7 @@ EfficientDiD( n_bootstrap: int = 0, # Multiplier bootstrap iterations bootstrap_weights: str = "rademacher", # "rademacher", "mammen", or "webb" seed: int | None = None, - anticipation: int = 0, + anticipation: int = 0, # non-negative integer; bool rejected omega_ridge: float = 1e-6, # Ridge for the Omega* inversion behind the efficient weights (PT-All's overidentified moments make sample Omega* numerically singular); 0 = legacy exact-inverse/pseudoinverse path. See the Omega* ridge Note in the methodology registry. ) ``` @@ -1352,7 +1354,7 @@ StaggeredTripleDifference( estimation_method: str = "dr", # "dr", "ipw", or "reg" control_group: str = "notyettreated", # "nevertreated" or "notyettreated" alpha: float = 0.05, - anticipation: int = 0, + anticipation: int = 0, # validated at fit(); construction stays permissive (deprecated class, frozen shape) base_period: str = "varying", # "varying" or "universal" n_bootstrap: int = 0, bootstrap_weights: str = "rademacher", @@ -1405,7 +1407,7 @@ Wooldridge (2023, 2025) Extended Two-Way Fixed Effects (ETWFE) estimator. OLS pa WooldridgeDiD( method: str = "ols", # "ols", "logit", or "poisson" control_group: str = "not_yet_treated", # "not_yet_treated" or "never_treated" - anticipation: int = 0, # Number of anticipation periods + anticipation: int = 0, # Number of anticipation periods (non-negative integer; bool rejected) demean_covariates: bool = True, # Demean covariates within cohort*period cells alpha: float = 0.05, cluster: str | None = None, diff --git a/diff_diff/guides/llms-practitioner.txt b/diff_diff/guides/llms-practitioner.txt index 4015f8874..fb404b346 100644 --- a/diff_diff/guides/llms-practitioner.txt +++ b/diff_diff/guides/llms-practitioner.txt @@ -114,7 +114,8 @@ Variants for staggered designs: ### No-Anticipation Treatment does not affect outcomes before it is implemented. Violated when units adjust behavior in anticipation of future treatment. Set `anticipation=k` -to allow k periods of anticipation. +to allow k periods of anticipation (`k` a non-negative integer; `bool` is +rejected). --- diff --git a/diff_diff/imputation.py b/diff_diff/imputation.py index a1db4999f..6815c32ee 100644 --- a/diff_diff/imputation.py +++ b/diff_diff/imputation.py @@ -48,6 +48,7 @@ ) from diff_diff.utils import ( safe_inference, + validate_anticipation, validate_df_convention, validate_n_bootstrap, ) @@ -82,6 +83,7 @@ class ImputationDiD(ImputationDiDBootstrapMixin, _ImputationAggregationMixin, Ba ---------- anticipation : int, default=0 Number of periods before treatment where effects may occur. + Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. cluster : str, optional @@ -234,7 +236,7 @@ def __init__( self._validate_leave_one_out(leave_one_out) validate_df_convention(df_convention) - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.alpha = alpha self.cluster = cluster self.vcov_type = vcov_type @@ -352,6 +354,10 @@ def fit( # attribute mutation (est.vcov_type = ...). self._validate_vcov_type(self.vcov_type) self._validate_leave_one_out(self.leave_one_out) + # Same direct-mutation defense for the anticipation window (an + # out-of-domain value silently changes the ESTIMAND); the assignment + # also re-normalizes a mutated numpy scalar to a Python int. + self.anticipation = validate_anticipation(self.anticipation) # Validate inputs required_cols = [outcome, unit, time, first_treat] diff --git a/diff_diff/spillover.py b/diff_diff/spillover.py index e9b13f744..506d4c654 100644 --- a/diff_diff/spillover.py +++ b/diff_diff/spillover.py @@ -42,7 +42,7 @@ from diff_diff.linalg import _rank_guarded_inv, solve_ols from diff_diff.results import SpilloverDiDResults from diff_diff.two_stage import _compute_gmm_corrected_meat, _LSMRUnconvergedError -from diff_diff.utils import _iterative_fe_solve, safe_inference +from diff_diff.utils import _iterative_fe_solve, safe_inference, validate_anticipation if TYPE_CHECKING: from diff_diff.survey import SurveyDesign @@ -1695,7 +1695,8 @@ class SpilloverDiD(BaseEstimator): Number of pre-treatment periods where effects may occur. Treatment and ring-membership clocks both shift by ``-anticipation`` so the stage-1 untreated-and-unexposed subsample correctly excludes - anticipation rows. + anticipation rows. Must be a non-negative integer; ``bool`` is + rejected. event_study : bool, default=False If ``True``, emit per-event-time × ring coefficients (Butts Table 2 staggered specification). The result's ``spillover_effects`` @@ -1768,7 +1769,7 @@ def __init__( self.conley_lag_cutoff = conley_lag_cutoff self.cluster = cluster self.alpha = alpha - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.event_study = event_study self.horizon_max = horizon_max self.rank_deficient_action = rank_deficient_action @@ -2186,19 +2187,15 @@ def fit( # injection); the gate cannot live up here because at this point # the user-supplied `cluster=` has not yet been injected into # the survey design as the effective PSU. - # Validate `anticipation` up front: must be a non-negative integer. - # Accepting fractional or negative values would silently shift - # treatment timing and ring exposure beyond what the estimator's - # identification contract supports. Validated BEFORE the - # event_study / horizon_max checks because the ref_period - # compatibility check below computes `-1 - self.anticipation` and - # would otherwise raise a raw TypeError on non-numeric input - # (PR #456 R2 fix). - if not isinstance(self.anticipation, (int, np.integer)) or self.anticipation < 0: - raise ValueError( - f"anticipation must be a non-negative integer; got " - f"{self.anticipation!r} (type {type(self.anticipation).__name__})." - ) + # In-fit `anticipation` re-check: __init__ and set_params now + # validate eagerly via the shared helper, so this re-assignment only + # catches DIRECT attribute mutation (est.anticipation = ...) — and + # normalizes a mutated numpy scalar to a Python int. It MUST stay + # ordered BEFORE the event_study / horizon_max checks because the + # ref_period compatibility check below computes + # `-1 - self.anticipation` and would otherwise raise a raw TypeError + # on non-numeric input (PR #456 R2 fix). + self.anticipation = validate_anticipation(self.anticipation) # Wave C: event-study path is now supported. Validate horizon_max # up front (fail-fast before any stage-1 work). if self.horizon_max is not None: diff --git a/diff_diff/stacked_did.py b/diff_diff/stacked_did.py index afd897a22..4a9f51aa5 100644 --- a/diff_diff/stacked_did.py +++ b/diff_diff/stacked_did.py @@ -33,7 +33,12 @@ from diff_diff.balancing import BalanceError, entropy_balance from diff_diff.linalg import effective_cluster_count, solve_ols from diff_diff.stacked_did_results import StackedDiDResults # noqa: F401 (re-export) -from diff_diff.utils import resolve_tail_df, safe_inference, validate_df_convention +from diff_diff.utils import ( + resolve_tail_df, + safe_inference, + validate_anticipation, + validate_df_convention, +) __all__ = [ "StackedDiD", @@ -77,7 +82,8 @@ class StackedDiD(BaseEstimator): alpha : float, default=0.05 Significance level for confidence intervals. anticipation : int, default=0 - Number of anticipation periods. When anticipation > 0: + Number of anticipation periods. Must be a non-negative integer; + ``bool`` is rejected. When anticipation > 0: - Reference period shifts from e=-1 to e=-1-anticipation - Post-treatment includes anticipation periods (e >= -anticipation) - Event window expands by anticipation pre-periods @@ -279,7 +285,7 @@ def __init__( self.control_group = control_group self.cluster = cluster self.alpha = alpha - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.rank_deficient_action = rank_deficient_action self.vcov_type = vcov_type self.balance = balance @@ -424,6 +430,14 @@ def fit( else: aggregate = None + # Fit-time re-check: __init__ and set_params validate eagerly, so + # this only catches DIRECT attribute mutation (est.anticipation = ...) + # — an out-of-domain value silently changes the ESTIMAND. The + # assignment also re-normalizes a mutated numpy scalar to int. Placed + # AFTER the deprecation shim so a caller who both mutated and passed + # a deprecated argument still sees the FutureWarning before the raise. + self.anticipation = validate_anticipation(self.anticipation) + # ---- Validate inputs ---- if aggregate in ("group", "all"): raise ValueError( diff --git a/diff_diff/staggered.py b/diff_diff/staggered.py index bc9ffbb68..0636e5319 100644 --- a/diff_diff/staggered.py +++ b/diff_diff/staggered.py @@ -38,7 +38,12 @@ CallawaySantAnnaResults, GroupTimeEffect, ) -from diff_diff.utils import safe_inference, safe_inference_batch, validate_n_bootstrap +from diff_diff.utils import ( + safe_inference, + safe_inference_batch, + validate_anticipation, + validate_n_bootstrap, +) if TYPE_CHECKING: from diff_diff.survey import SurveyDesign @@ -315,7 +320,8 @@ class CallawaySantAnna( anticipation : int, default=0 Number of periods before treatment where effects may occur. Set to > 0 if treatment effects can begin before the official - treatment date. + treatment date. Must be a non-negative integer; ``bool`` is + rejected. estimation_method : str, default="dr" Estimation method: - "dr": Doubly robust (recommended) @@ -608,7 +614,7 @@ def __init__( self._validate_vcov_type(vcov_type) self.control_group = control_group - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.estimation_method = estimation_method self.alpha = alpha self.cluster = cluster @@ -1952,6 +1958,10 @@ def fit( # second layer only catches DIRECT attribute mutation # (est.vcov_type = ...) before it propagates to Results metadata. self._validate_vcov_type(self.vcov_type) + # Same direct-mutation defense for the anticipation window (an + # out-of-domain value silently changes the ESTIMAND); the assignment + # also re-normalizes a mutated numpy scalar to a Python int. + self.anticipation = validate_anticipation(self.anticipation) # --- allow_unbalanced_panel routing (RC-on-panel = R's allow_unbalanced_panel) --- # Detect an unbalanced panel (some units unobserved in some periods). diff --git a/diff_diff/staggered_triple_diff.py b/diff_diff/staggered_triple_diff.py index 9f2aafaf5..977baafc6 100644 --- a/diff_diff/staggered_triple_diff.py +++ b/diff_diff/staggered_triple_diff.py @@ -85,7 +85,9 @@ class StaggeredTripleDifference( alpha : float, default=0.05 Significance level. anticipation : int, default=0 - Number of anticipation periods. + Number of anticipation periods. Must be a non-negative integer; + ``bool`` is rejected - validated at ``fit()`` (construction stays + permissive on this deprecated class). base_period : str, default="varying" Base period selection: "varying" (consecutive comparisons) or "universal" (always vs g-1-anticipation). diff --git a/diff_diff/sun_abraham.py b/diff_diff/sun_abraham.py index dd473153e..7e7ff5b92 100644 --- a/diff_diff/sun_abraham.py +++ b/diff_diff/sun_abraham.py @@ -42,6 +42,7 @@ resolve_tail_df, safe_inference, snap_absorbed_regressors, + validate_anticipation, validate_df_convention, validate_n_bootstrap, ) @@ -530,6 +531,7 @@ class SunAbraham(BaseEstimator): - "not_yet_treated": Use never-treated and not-yet-treated units anticipation : int, default=0 Number of periods before treatment where effects may occur. + Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. cluster : str, optional @@ -727,7 +729,7 @@ def __init__( ) self.control_group = control_group - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.alpha = alpha self.cluster = cluster validate_n_bootstrap(n_bootstrap) @@ -797,6 +799,12 @@ def fit( ValueError If required columns are missing or data validation fails. """ + # Fit-time re-check: __init__ and set_params validate eagerly, so + # this only catches DIRECT attribute mutation (est.anticipation = ...) + # — an out-of-domain value silently changes the ESTIMAND. The + # assignment also re-normalizes a mutated numpy scalar to int. + self.anticipation = validate_anticipation(self.anticipation) + # Validate inputs required_cols = [outcome, unit, time, first_treat] if covariates: diff --git a/diff_diff/triple_diff.py b/diff_diff/triple_diff.py index 41cd5524a..67e48f2b9 100644 --- a/diff_diff/triple_diff.py +++ b/diff_diff/triple_diff.py @@ -477,7 +477,8 @@ class TripleDifference( deprecated ``StaggeredTripleDifference`` and die with it at 4.0. anticipation : int, default=0 Number of periods before the enabling period in which units may - already respond. Must be a non-negative integer. Shifts each cohort's + already respond. Must be a non-negative integer; ``bool`` is + rejected. Shifts each cohort's base period earlier and excludes cohorts entering treatment within the window from the comparison group. base_period : str, default="varying" @@ -636,9 +637,9 @@ def __init__( # threshold to max(t, base) - 1, admitting cohorts treated at the # evaluation period as "clean" controls. Neither condition is # observable in the output. `bool` is rejected too - `True` would - # otherwise coerce to a silent one-period window. (The sibling - # estimators taking `anticipation` are not yet uniformly validated; - # aligning the family is tracked in TODO.md.) + # otherwise coerce to a silent one-period window. (The whole family + # now validates uniformly via the shared helper; see the family-wide + # adoption note, ledger row M-144, in REGISTRY.md.) validate_anticipation(anticipation) if base_period not in ("varying", "universal"): raise ValueError(f"base_period must be 'varying' or 'universal', got '{base_period}'") @@ -708,7 +709,8 @@ def __init__( # Staggered-mode config (M-013). Inert in 2x2x2 mode, and fit() rejects # any non-default value there rather than letting it pass unused. self.control_group = control_group - self.anticipation = anticipation + # Already validated above (early-validation ordering); normalize only. + self.anticipation = int(anticipation) self.base_period = base_period self.n_bootstrap = n_bootstrap self.bootstrap_weights = bootstrap_weights diff --git a/diff_diff/two_stage.py b/diff_diff/two_stage.py index b60e179cf..b2c5a836f 100644 --- a/diff_diff/two_stage.py +++ b/diff_diff/two_stage.py @@ -51,7 +51,7 @@ TwoStageBootstrapResults, # noqa: F401 TwoStageDiDResults, ) # noqa: F401 (re-export) -from diff_diff.utils import safe_inference, validate_n_bootstrap +from diff_diff.utils import safe_inference, validate_anticipation, validate_n_bootstrap if TYPE_CHECKING: # Forward reference for the Wave E.1 survey-design path. Imported under @@ -1237,6 +1237,7 @@ class TwoStageDiD(TwoStageDiDBootstrapMixin, _TwoStageAggregationMixin, BaseEsti ---------- anticipation : int, default=0 Number of periods before treatment where effects may occur. + Must be a non-negative integer; ``bool`` is rejected. alpha : float, default=0.05 Significance level for confidence intervals. cluster : str, optional @@ -1339,7 +1340,7 @@ def __init__( ) self._validate_vcov_type(vcov_type) - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.alpha = alpha self.cluster = cluster self.vcov_type = vcov_type @@ -1447,6 +1448,10 @@ def fit( # (BaseEstimator probe re-init), so this only catches DIRECT # attribute mutation (est.vcov_type = ...). self._validate_vcov_type(self.vcov_type) + # Same direct-mutation defense for the anticipation window (an + # out-of-domain value silently changes the ESTIMAND); the assignment + # also re-normalizes a mutated numpy scalar to a Python int. + self.anticipation = validate_anticipation(self.anticipation) # ---- Data validation ---- required_cols = [outcome, unit, time, first_treat] diff --git a/diff_diff/utils.py b/diff_diff/utils.py index ea7ae4d4d..85400303e 100644 --- a/diff_diff/utils.py +++ b/diff_diff/utils.py @@ -550,8 +550,14 @@ def staggered_ddd_ctor_offenders(estimator: Any) -> List[str]: ] -def validate_anticipation(anticipation: Any) -> None: - """Raise ValueError unless ``anticipation`` is a non-negative integer. +def validate_anticipation(anticipation: Any) -> int: + """Validate ``anticipation`` and return it as a normalized Python ``int``. + + Raises ``ValueError`` unless the value is a non-negative integer; on + success returns ``int(anticipation)`` so numpy integers are normalized at + the assignment site (``self.anticipation = validate_anticipation(...)``) + before any ``-1 - anticipation``-style arithmetic can overflow on an + unsigned numpy scalar. An out-of-domain anticipation window does not fail loudly on its own — it silently changes the ESTIMAND. On the staggered DDD engine the value feeds @@ -565,9 +571,14 @@ def validate_anticipation(anticipation: Any) -> None: one-period window. Numpy integers are accepted, matching :func:`validate_n_bootstrap`, whose shape this follows. - Adopted by ``TripleDifference`` and the shared staggered engine (so the - deprecated ``StaggeredTripleDifference`` fails closed too). The remaining - estimators taking ``anticipation`` are tracked for alignment in TODO.md. + Adopted at ``__init__`` by all nine anticipation-taking estimators + (CallawaySantAnna, SunAbraham, ImputationDiD, TwoStageDiD, StackedDiD, + ContinuousDiD, EfficientDiD, WooldridgeDiD, SpilloverDiD) plus + ``TripleDifference``, and RE-CHECKED on the fit path on all of them — + the uniform direct-mutation defense: manual re-assignments on seven, + EfficientDiD via its ``_validate_params`` re-run, SpilloverDiD via its + in-fit check, TripleDifference and the deprecated + ``StaggeredTripleDifference`` via the shared staggered engine. """ if isinstance(anticipation, bool) or not isinstance(anticipation, (int, np.integer)): raise ValueError( @@ -576,6 +587,7 @@ def validate_anticipation(anticipation: Any) -> None: ) if anticipation < 0: raise ValueError(f"anticipation must be a non-negative integer; got {anticipation!r}.") + return int(anticipation) def resolve_tail_df( diff --git a/diff_diff/wooldridge.py b/diff_diff/wooldridge.py index cf65f4142..c2fdc3483 100644 --- a/diff_diff/wooldridge.py +++ b/diff_diff/wooldridge.py @@ -36,6 +36,7 @@ resolve_tail_df, safe_inference, snap_absorbed_regressors, + validate_anticipation, validate_df_convention, validate_n_bootstrap, within_transform, @@ -878,7 +879,8 @@ class WooldridgeDiD(BaseEstimator): comparison (tracked in ``TODO.md``). anticipation : int Number of periods before treatment onset to include as treatment cells - (anticipation effects). 0 means no anticipation. + (anticipation effects). 0 means no anticipation. Must be a + non-negative integer; ``bool`` is rejected. demean_covariates : bool If True (jwdid default), ``xtvar`` covariates are demeaned within each cohort×period cell before entering the regression. Set to False to @@ -995,7 +997,6 @@ def __init__( self._validate_constructor_args( method=method, control_group=control_group, - anticipation=anticipation, bootstrap_weights=bootstrap_weights, vcov_type=vcov_type, cohort_trends=cohort_trends, @@ -1004,7 +1005,7 @@ def __init__( self.method = method self.control_group = control_group - self.anticipation = anticipation + self.anticipation = validate_anticipation(anticipation) self.demean_covariates = demean_covariates self.alpha = alpha self.cluster = cluster @@ -1037,7 +1038,6 @@ def _validate_constructor_args( *, method: str, control_group: str, - anticipation: int, bootstrap_weights: str, vcov_type: str, cohort_trends: bool = False, @@ -1055,8 +1055,6 @@ def _validate_constructor_args( raise ValueError( f"control_group must be one of {_VALID_CONTROL_GROUPS}, got {control_group!r}" ) - if anticipation < 0: - raise ValueError(f"anticipation must be >= 0, got {anticipation}") if bootstrap_weights not in _VALID_BOOTSTRAP_WEIGHTS: raise ValueError( f"bootstrap_weights must be one of {_VALID_BOOTSTRAP_WEIGHTS}, " @@ -1145,6 +1143,16 @@ def fit( require_arg("WooldridgeDiD.fit", "first_treat", first_treat) # Body-local name; the public parameter is first_treat (M-032). cohort = first_treat + + # Fit-time re-check: __init__ and set_params validate eagerly, so + # this only catches DIRECT attribute mutation (est.anticipation = ...) + # — an out-of-domain value silently changes the ESTIMAND. The + # assignment also re-normalizes a mutated numpy scalar to int. Placed + # AFTER the cohort→first_treat rename shim so a caller who both + # mutated and passed the deprecated kwarg still sees the + # FutureWarning before the raise. + self.anticipation = validate_anticipation(self.anticipation) + df = data.copy() df = _warn_and_fill_nan_cohort(df, cohort, stacklevel=2) diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 70b225b8d..e52c7fb5e 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1065,6 +1065,7 @@ The multiplier bootstrap uses random weights w_i with E[w]=0 and Var(w)=1: - [ ] Multiplier bootstrap preserves panel structure - [x] Repeated cross-sections (`panel=False`) for non-panel surveys (Phase 7b) +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## ChaisemartinDHaultfoeuille @@ -1441,6 +1442,7 @@ labels.* - **Note:** The R-style convention of coding never-treated units as `first_treat=inf` is still accepted and normalized to `first_treat=0` internally, but the estimator now emits a `UserWarning` reporting the row count so the silent recategorization is surfaced (axis-E silent coercion under the Phase 2 audit). Only `+inf` is recoded (matching the R convention). Any **negative** `first_treat` value (including `-inf`) raises `ValueError` with the row count, since such units would otherwise silently fall out of both the treated (`g > 0`) and never-treated (`g == 0`) masks. Pass `0` directly for never-treated units to avoid the warning. - **Note:** Rows where `first_treat=0` (never-treated) carry a nonzero `dose` are silently zeroed for internal consistency (never-treated cells must have `D=0` in the dose response). The estimator now emits a `UserWarning` with the affected row count before the zeroing, so unintended nonzero doses on never-treated rows are no longer absorbed without a signal (axis-E silent coercion). +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## EfficientDiD @@ -1634,6 +1636,7 @@ where `q_{g,e} = pi_g / sum_{g' in G_{trt,e}} pi_{g'}`. - **Note:** `set_params(vcov_type=bad)` raises immediately on EVERY estimator: since the shared `BaseEstimator` mixin (`diff_diff/_base.py`, v4 2(c)-i), `set_params` validates transactionally by constructor probe re-init, so it enforces exactly `__init__`'s validation, eagerly, library-wide. The former split — EfficientDiD eager vs `ImputationDiD`/`TripleDifference`/`CallawaySantAnna` (and six more: SunAbraham, StackedDiD, StaggeredTripleDifference, SpilloverDiD, TROP, PreTrendsPower) accepting constructor-rejected values until `fit()` — is retired; the fit-time re-validation layers remain as a second check against DIRECT attribute mutation (`est.vcov_type = ...`), which no setter can see. - **Note (post-fit aggregate() - rows M-023/M-120):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work) in favor of post-fit `EfficientDiDResults.aggregate(type, balance_e=)` - a LAZY RECOMPUTING KIT (the CallawaySantAnna class, not a StackedDiD/dCDH view relay): `fit()` computes nothing extra, the results object retains an `AggregationKit`, and `aggregate('event_study'/'group', balance_e=)` re-runs the extracted `_EfficientAggregationMixin` aggregators on a throwaway host while `aggregate('simple')` relays the stored overall row bit-exact. (a) RETAINED BUFFERS (memory contract; phrased as maxima - optional design fields stay None when unsupplied): the per-(g,t) EIF dict, O(n_units x n_gt), the dominant payload - retained on EVERY fit regardless of `store_eif`, which since 3.9 governs only the public `influence_functions` field; `unit_cohorts` (cohort labels), `unit_level_weights`, factorized cluster codes - O(n_units) each; on ordinary (TSL) survey fits the unit-level `ResolvedSurveyDesign` adds `weights` plus, where supplied, `strata`/`psu`/`fpc` (factorized int codes / float values, never raw labels) - up to four O(n_units) arrays; on replicate designs it adds the O(n_units x n_replicates) replicate matrix plus, where supplied, `replicate_strata`/`replicate_rscales` (O(n_replicates)); per-row dict SNAPSHOTS of `group_time_effects` plus copies of the `groups`/`time_periods` lists and the scalar `pt_assumption`/`n_treated+n_control` provenance (aggregate() recomputes exclusively from these private snapshots, never from the mutable public result fields - a user edit of the public rows cannot mix altered point estimates with the retained EIF variance); scalars `n_units`, `cohort_fractions`, and the POST-OVERALL `df_survey` snapshot (captured after the overall inference and before the ES/group gates: the group pass can degenerate the working df to None on replicate designs with `n_valid <= 1`, and every fit-time aggregation seeds from the post-overall value, so recompute replays the exact seed). The data-minimization guarantee is scoped to unit identifiers - no unit-label container is retained. (b) `balance_e` uses the ANCHOR-HORIZON rule (keep cohorts with a finite effect at `e == balance_e`, then retain all their horizons) - the SAME rule CallawaySantAnna uses, divergent only from ImputationDiD/TwoStageDiD's balanced-window rule; an anchor no cohort reaches warns and yields a legal zero-row container. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall row verbatim (percentile se/p/CI beside the finite safe_inference t) with a NaN df column, while the RECOMPUTE levels (ES/group) fail closed - per-horizon draws are not retained (exact-replay wiring is a TODO row). The prior uniform-conservatism BY-DECISION rule was superseded 2026-08-05 with the M-027 per-level convergence; its rationale - no level publishes analytical-provenance fields beside percentile inference - is honored by the relay's NaN df column. The fit-time bootstrap override still clears the group rows' `df_used` key. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the primary ground is the absent joint event-study covariance (container `vcov=None`, all-NaN per-row df - the scalar `df_survey` channel is the container's only df provenance; the per-row hole is the tracked M-092-completion TODO row). Reference semantics are regime-dependent: under `pt_assumption="all"` there is NO reference row (universal first-period baseline; e=-1 is a genuine estimate); under `"post"` the per-cohort baseline cell is materialized as a mechanical zero anchor at `e = -1 - anticipation` whenever it is not the panel's first period, and the MEMBERSHIP-GATED `reference_period` property (the SunAbraham rule - never synthesized when the anchor cell was not estimated) marks it `is_reference` in the container and corrects `plot_event_study`'s inferred reference (previously the `-1` fallback) on PT-Post `anticipation>0` fits. (e) 'simple' relay conventions: `target="att"`, `n = n_treated_units + n_control_units` with `n_kind="units"` (DISJOINT by construction - `last_cohort` trimming reassigns before the counts, so a true total exists, unlike StackedDiD's overlapping sets), `df` = the post-overall snapshot (provenance-exact where `survey_metadata.df_survey` can diverge in the degenerate replicate state); 'group' relay: `n_kind="cells"`, `weight=None` (equal within-cohort weights, no cross-cohort mass), per-row `df_used` array captured at each row's `safe_inference` call (exact by construction; a stated divergence from CS's conservative-min scalar broadcast); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose `survey_metadata` copy carries the snapshot `df_survey`. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## SunAbraham @@ -1823,6 +1826,7 @@ where weights ŵ_{g,e} = n_{g,e} / Σ_g n_{g,e} (sample share of cohort g at eve - [x] R comparison: Event study effects match perfectly (correlation 1.0) - [x] Survey design support (Phase 3): weighted within-transform, survey weights in LinearRegression with TSL vcov; bootstrap+survey supported (Phase 6) via Rao-Wu rescaled bootstrap. Replicate weights supported via estimator-level refit (see Replicate Weight Variance section); replicate+bootstrap rejected. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## ImputationDiD @@ -1979,6 +1983,7 @@ where `W_it(h) = 1[K_it = h]` are lead indicators, estimated on `Omega_0` only. - **Note (post-fit aggregate() - rows M-021/M-118):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `imputation_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-070) in favor of post-fit `ImputationDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit (not an EIF-payload kit: ES/group aggregation is a target-specific Theorem-3 recompute - each `balance_e` re-masks which treated observations enter every horizon and re-solves the untreated projection - so no compact influence payload can replace the frame). (a) RETAINED BUFFERS (memory contract): the kit's bookkeeping holds REFERENCES to the SAME per-fit objects `_fit_data` already retains for `pretrend_test()` - the working panel copy (all user columns plus `_tau_hat`/`_rel_time`/`_never_treated`), the Omega masks, `unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`, the resolved survey design, and `survey_weights` - ZERO marginal memory, and pickles are unchanged via memoization (`_estimator_ref` already ships these objects); plus value SNAPSHOTS for isolation (a `treatment_groups` copy, config scalars, a `dataclasses.replace` copy of `survey_metadata`, `overall_att`, `n_treated_obs`) and TWO df-provenance scalars (`survey_df_seed`, what the analytical aggregators received; `survey_df_final`, what the stored overall inference received). Each `aggregate()` call runs on a fresh throwaway host with a call-local projection cache (the fit-local factorizations are unpicklable and never retained). (b) `balance_e` uses the BALANCED-WINDOW rule: a cohort is retained iff its observed relative-time set - checked against the FULL panel via `_build_cohort_rel_times()` - covers the contiguous window `[-balance_e, max_h]`; the SAME rule TwoStageDiD uses, divergent from CS/EfficientDiD's anchor-horizon rule. A window no cohort satisfies warns and yields the reference-marker-only dict (a legal near-empty container). (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (the per-target psi machinery makes exact replay tractable - a TODO row); the prior uniform fail-closed rule was superseded 2026-08-05 with the M-027 per-level convergence. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is REJECTED BY DESIGN (both terminal TypeErrors state it): the surface carries no joint event-study covariance - per-horizon conservative SEs only (container `vcov=None`; the scalar `df_survey` channel is its only df provenance, the per-row hole being the tracked M-092-completion TODO row). (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (the treated/control UNIT sets overlap - a treated unit with pre-periods counts in both - so the CS/EDiD disjoint-units convention cannot apply; |Omega_1| is the population the ATT averages over, of which only finite-tau-hat observations enter the average - `n` reports the raw count, so on partially unidentified fits `n` exceeds the averaged support) and `df = survey_df_final`; 'group' rows carry per-row `df_used` captured at each row's `safe_inference` (the replicate override rewrites it, the bootstrap override clears it, the all-NaN cohort branch writes no key - consumers read via `.get`); 'event_study' rides the shared `_from_relative_dict` builder via a carrier whose metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replicate_override_aggregates` with a LEVEL-MATCHED stack: `compute_replicate_refit_variance` validates replicates JOINTLY (all-finite rows), so `aggregate(L)` reproduces `fit(aggregate=L)` exactly, a `fit(aggregate='all')` surface is NOT the equivalence target when a replicate NaNs on exactly one family's targets, and - the documented migration delta - moving a replicate fit from `fit(aggregate=)` to plain fit + post-fit `aggregate()` can change the public OVERALL row's se/CI/df on such degenerate designs (each surface self-consistent; pinned in the contract tests). `pretrends=True` + replicate: post-fit `aggregate('event_study')` raises the same NotImplementedError the fit-time gate raises (per-replicate lead refits unimplemented); 'group'/'simple' still work. Recompute re-emits the fit-time warnings (LSMR, Prop-5, empty-window) with fit-tuned stacklevels - post-fit attribution lands on a library frame, an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits) and `heterogeneity` checks run without the deprecated fit-time kwarg; the re-emitted recompute warnings are captured and re-published on the consuming report section (record-and-republish), and derivation failures (bootstrap, missing kit, the replicate gate above) surface as explicit per-check skip reasons. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## TwoStageDiD @@ -2068,6 +2073,7 @@ Our implementation uses multiplier bootstrap on the GMM influence function: clus - **Note (post-fit aggregate() - rows M-022/M-119):** `fit(aggregate=, balance_e=)` is deprecated (3.9; removed 4.0; joint FutureWarning, warn-and-still-work; the `two_stage_did` wrapper forwards the shared sentinel so plain wrapper calls never fire the aggregate warning; since 3.9 the wrapper itself warns per M-071) in favor of post-fit `TwoStageDiDResults.aggregate(type, balance_e=)` - a PANEL-BACKED lazy recompute kit: each level is a fresh Stage-2 OLS + joint Gardner-GMM sandwich on a level-specific design, so no compact influence payload exists. (a) RETAINED BUFFERS (memory contract - the FIRST panel retention on TwoStageDiD results, a deliberate break from the CS/EDiD identifier-minimization guarantee, with a `store_kit` opt-out tracked in DEFERRED.md): a COLUMN-SUBSET COPY of the working frame - `unit`/`time`/`outcome`/`first_treat` + covariates + the cluster column (deduplicated: `cluster=` may legally name a core column) + `_never_treated`/`_rel_time`/`_y_tilde` - O(n_obs) on every results object and pickle; the Stage-1 FE model (`unit_fe`/`time_fe`/`grand_mean`/`delta_hat`/`kept_cov_mask`), the Omega masks, the full-domain `keep_mask`, the Wave-E.3-GATED `score_pad_mask`/`cluster_ids_full` values fit actually passed (None unless the always-treated pad was active), `survey_weights`, and the resolved survey design - on replicate designs that adds the O(n_obs x R) replicate matrix; plus value snapshots (`treatment_groups` copy, `ref_period`, `overall_att`, `n_treated_obs`, a `dataclasses.replace` copy of `survey_metadata`) and TWO df scalars (`survey_df_stage2`, the recompute seed; `survey_df_final`, what the stored overall inference received). (b) `balance_e` uses the BALANCED-WINDOW rule (`[-balance_e, max_h]` coverage against the full panel - the ImputationDiD rule, divergent from CS/EfficientDiD's anchor-horizon rule); zero qualifying cohorts warns and yields the reference-row-only dict with `vcov=None`. (c) BOOTSTRAP fits: 'simple' RELAYS the stored overall quintet verbatim (finite safe_inference t included) with a NaN df column, while the RECOMPUTE levels fail closed (per-level GMM scores are function-locals; replay is a TODO row) - the prior uniform rule superseded 2026-08-05 with the M-027 per-level convergence; a fit whose bootstrap FAILED (`bootstrap_results=None`, analytical inference retained) aggregates normally. (d) CONTAINER ADMISSION into `compute_honest_did`/`compute_pretrends_power` is DEFERRED, not by-design (both terminal TypeErrors state it): analytical surfaces DO carry the real joint Gardner-GMM covariance (M-092), but the pre-period coefficients are stage-1 residual MEANS - the reference horizon is dropped from the no-intercept Stage-2 design and the zero anchor row is appended mechanically - not contrasts against the advertised reference, while HonestDiD's Delta^RM/Delta^SD arithmetic hard-codes the `delta_0 = 0` normalization into its boundary/bridge constraints; admission awaits a normalization derivation (either re-estimating Stage 2 with the reference horizon in the design or deriving the residual-to-reference mapping) - the DEFERRED.md paper-gated row. (e) RELAY CONVENTIONS: 'simple' relays the stored overall quintet bit-exact with `n = n_treated_obs`, `n_kind="obs"` (overlapping unit sets - the StackedDiD carve-out class; the pre-filter |Omega_1| count, while the ATT's Stage-2 support excludes rows whose `y_tilde` is non-finite - on such degenerate fits `n` exceeds the averaged support) and `df = survey_df_final` (on replicate fits that value came from the `[overall]`-only joint stack - snapshotted, never re-derived); 'group' relays a SCALAR df broadcast (deliberate divergence from ImputationDiD's per-row `df_used`: `_stage2_group` passes one immutable `survey_df` to every row's `safe_inference`, so the scalar is provenance-exact by construction and the moved method stays verbatim); 'event_study' reproduces the M-092 container contract exactly - analytical fits thread the recomputed joint vcov + `vcov_index` + the finite-and->0 df scalar through the carrier, replicate fits thread `vcov=None`/`index=None` with the REPLAYED level-matched df, and the carrier's metadata is a copy-on-use of the KIT's fit-final metadata copy. REPLICATE-WEIGHT fits replay the extracted `_replay_replicate_inference` with a LEVEL-MATCHED stack (the ImputationDiD semantics: `aggregate(L)` reproduces `fit(aggregate=L)`; `fit(aggregate='all')` is not the equivalence target on degenerate designs; the OVERALL-row migration delta on such designs is documented and pinned). Recompute re-emits fit-time warnings with fit-tuned stacklevels - an accepted verbatim-move trade-off. `DiagnosticReport` now derives this container internally on plain fits (raw `event_study_effects` absent), so its `parallel_trends` (`pretrends=True` fits; the recomputed joint vcov drives the joint-Wald path) and `heterogeneity` checks run without the deprecated fit-time kwarg; re-emitted recompute warnings are captured and re-published on the consuming report section, and derivation failures surface as explicit per-check skip reasons. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## StackedDiD @@ -2215,6 +2221,7 @@ The pooled estimator is `DID^{CBWSDID}_e = Σ_a (N^D_a/N^D_Ω)(Δ̄^D_{a,e} − - Missing pre-treatment row, or covariate absent / `balance`↔`covariates` mismatch: `ValueError` at `fit()`. - **Ragged / unbalanced event windows** (a unit not observed at every event time in a sub-experiment): **fail-closed `ValueError`** — `balance="entropy"` requires balanced windows. The paper assumes balanced event windows; off them the unit-count corrector and the observation-count `aggregate` Q diverge (the count-convention is unresolved, deferred). `balance="none"` continues to support unbalanced panels via observation-count Q. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## WooldridgeDiD (ETWFE) @@ -2401,6 +2408,7 @@ Consolidated list of substantive deviations from the W2025 paper and from R `etw 6. **Anticipation + aggregation**: `aggregate(type="simple", weights="cell")` uses `t >= g` as the post-treatment threshold regardless of `anticipation`. Anticipation-window leads are estimated as placebos but excluded from `overall_att`. See § Edge cases Note. 7. **Response-scale ATT vs R `etwfe` log-link coefficients** (Poisson + logit): diff-diff's `WooldridgeDiD(method="poisson" | "logit")` returns ATT on the response scale (counterfactual mean difference per paper W2023 ASF / APE framework); R `etwfe(family="poisson" | "logit")` returns the cell-level log-link / log-odds coefficient. Numerical cell-level R-parity for nonlinear paths requires either `emfx()`-based APE extraction on the R side or link-function inversion with baseline-mean adjustment; deferred (DEFERRED.md row, added in PR-B). See `tests/test_methodology_wooldridge.py::TestWooldridgeParityRPoisson` / `TestWooldridgeParityRLogit` for the current surface-test scope. +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense); this superseded the local `>= 0` check, CHANGING the error message text (now the shared `anticipation must be a non-negative integer ...` wording) and turning the `None`/str raw `TypeError` into a `ValueError` — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## LPDiD @@ -3194,8 +3202,9 @@ shared verbatim. - **Note (`anticipation` domain validated from birth):** `anticipation` is one of the seven staggered-only constructor params introduced on `TripleDifference` in 3.9, and it is validated as a non-negative integer (`bool` rejected) from that - birth - so this is a new param's input contract, not a tightening of an existing - one, and it needs no lifecycle row. The guard is load-bearing rather than + birth - TripleDifference's own param needed no lifecycle row (a from-birth + contract); the FAMILY-WIDE tightening of the pre-existing siblings is rowed as + [M-144] (see the family-wide adoption paragraph below). The guard is load-bearing rather than cosmetic: the value feeds BOTH the base-period rule and the comparison-cohort threshold, so `anticipation=-1` would make the universal base period `g` - an ALREADY-TREATED period - and relax the not-yet-treated threshold to @@ -3211,10 +3220,25 @@ shared verbatim. signature contract is untouched) and FIT raises, because this is an identification guard rather than an API change. The engine-level call also covers direct attribute mutation, which bypasses `__init__` and `set_params` - alike. **Sibling divergence, recorded rather than silently tolerated:** among - the remaining estimators taking `anticipation`, only `spillover.py` and - `wooldridge.py` validate it (and neither rejects `bool`); aligning the family - behind the shared helper is tracked as a `TODO.md` row. + alike. **Family-wide adoption (ledger row [M-144]):** all nine + anticipation-taking estimators (CallawaySantAnna, SunAbraham, ImputationDiD, + TwoStageDiD, StackedDiD, ContinuousDiD, EfficientDiD, WooldridgeDiD, + SpilloverDiD) now call the shared `utils.validate_anticipation` at `__init__` + (transactional `set_params` inherits via the BaseEstimator probe re-init) AND + re-check it on the fit path via the assignment form `self.anticipation = + validate_anticipation(self.anticipation)` — the uniform direct-mutation + defense, which also normalizes numpy scalars to Python `int` before any + `g - 1 - anticipation` arithmetic can overflow on an unsigned scalar. + SpilloverDiD's re-check is the in-fit check, kept ordered before the + ref-period arithmetic (the PR #456 R2 guarantee); EfficientDiD's rides its + `_validate_params` fit-time re-run; the staggered engine covers + TripleDifference and the deprecated sibling. WooldridgeDiD's weaker local + `>= 0` check is superseded (its message text changed, and `None`/str now + raise `ValueError` instead of a raw `TypeError`). The deprecated + `StaggeredTripleDifference` stays construction-permissive by design (pin: + test_v4_merge_ddd.py::test_deprecated_sibling_also_fails_closed_on_anticipation) + — a recorded choice, not a gap. Policy suite: + `tests/test_anticipation_policy.py` (roster-guarded). - **Note (degenerate enabling cohorts are reported, not hidden):** a positive `first_treat` cohort whose units are ALL `partition == 0` cannot identify `ATT(g,t)` — the DDD contrast needs the eligible-treated cell — so it @@ -5518,6 +5542,7 @@ Degrees of freedom for the t-distribution lookup use `ResolvedSurveyDesign.df_su **Implementation:** `diff_diff/spillover.py`. Public class `SpilloverDiD`; result class `SpilloverDiDResults(DiDResults)` at `diff_diff/results.py`. Tests at `tests/test_spillover.py`; DGP factories `tests/_dgp_utils.py::generate_butts_nonstaggered_dgp` / `generate_butts_staggered_dgp` (satisfy Butts Assumptions 1/3/5/7 by construction). +- **Note:** `anticipation` is validated at construction (non-negative integer; `bool` rejected) via the shared `utils.validate_anticipation`, and re-checked on the fit path (direct-mutation defense) — see the family-wide adoption note (ledger row [M-144]) in the TripleDifference staggered-mode section. --- ## ConleySpatialHAC diff --git a/docs/migration-4.0.md b/docs/migration-4.0.md index bd0094a1e..6b397b832 100644 --- a/docs/migration-4.0.md +++ b/docs/migration-4.0.md @@ -216,8 +216,18 @@ each one with its target. ## Remaining 4.0 changes Smaller items that do not fit the families above — two inert `SyntheticDiD` constructor -parameters, the `covariates=` constructor-to-`fit()` move, a retired transition warning, and the -Bacon roster re-homing. See the appendix. +parameters, the `covariates=` constructor-to-`fit()` move, a retired transition warning, the +Bacon roster re-homing, and the family-wide `anticipation` validation below. The appendix lists +the ledger-derived removals/flips; [M-144] is a behavior tightening with no removal/deprecation +fields and appears here only. + +- `anticipation` is validated across the family ([M-144], landing at 4.0): whole-valued floats + that previously fit identically to their integer now raise — pass the `int`; bool and + negative/non-integer values raise too. Accepted numpy integers are normalized, so the public + `anticipation` attribute and `get_params()["anticipation"]` are now always built-in `int` + (previously a numpy scalar survived). WooldridgeDiD's error message text changed to the shared + wording, and its constructor now reports a bad `bootstrap_weights`/`vcov_type`/`df_convention` + before a bad `anticipation` (the ordering flipped). One pending decision: the `DIFF_DIFF_SOLVE_OLS_FASTPATH` environment default has a go/no-go due at 4.0 that has not been made. If it lands on, it is a numerics change and will be documented diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 0a8bbea0c..7fecf683d 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -1733,3 +1733,16 @@ rows: test_ref: tests/test_v4_merge_cic.py code_refs: [diff_diff/changes_in_changes_results.py, diff_diff/changes_in_changes.py, diff_diff/practitioner.py, docs/methodology/REGISTRY.md] notes: "Results-side mirror of [M-015] under section 8 rule 9: the field already held exactly the tag the new method= param sets ('cic'/'qdid'), so leaving it named 'estimator' would ship a param and its echoing field under different names and keep emitting 'estimator' from to_dict() forever. WooldridgeDiDResults.method is the in-repo template (a lowercase method tag selected by a method= constructor param, mirrored into summary()/to_dict()/__repr__). It also resolves a pre-existing cross-class collision: AggregationResult.estimator holds a CLASS NAME while this field holds a method tag. Shim is deprecated_field_property (read-only BY DESIGN - setattr must fail rather than write a shadow attribute the renamed field cannot see) plus a __setstate__ key migration; to_dict() emits BOTH keys through 3.9 per the [M-094] twin and spec section 5, so test_to_dict_keys stays green untouched and the old key drops at 4.0. __repr__'s user-visible label flips to method=. ENFORCEMENT NOTE: adding this row forces 'estimator' into test_naming_guard's _PATTERN_TOKENS (test_predicate_binds_to_ledger_tokens requires every live param/field rename token to satisfy _pattern_hit), which arms Duty A repo-wide - hence the four SURFACE_ALLOWLIST entries for the independent same-named surfaces (AggregationResult.estimator and the estimator= instance param on the three power entry points). 'estimator' is also added to _AMBIGUOUS_TOKENS so Duty C matches through the attr/quoted lanes only: the kwarg lane's remaining hits are the construction site (which becomes method= here) and validate_covariate_names(..., estimator=...), an unrelated parameter. REGISTRY.md is in code_refs because this PR's own REGISTRY subsection names the renamed field and would otherwise be an uncovered Duty C hit in the same diff that writes it. SCOPE OF THE WINDOW (decision, recorded as a REGISTRY Note): the shim covers the READ path, not the CONSTRUCT path - ChangesInChangesResults(..., estimator=...) raises TypeError from 3.9 rather than warning until 4.0, matching [M-094] and [M-114], neither of which preserved a deprecated constructor keyword. The container has one library call site; a constructor shim would add a deprecated surface with no row of its own." + - id: M-144 + kind: behavior + group: policy-anticipation + old: "diff_diff:CallawaySantAnna[anticipation]" + new: null + introduced_in: "4.0" + deprecated_in: null + removed_in: null + status: done + phase: 5 + test_ref: tests/test_anticipation_policy.py + code_refs: [diff_diff/utils.py, diff_diff/staggered.py, diff_diff/sun_abraham.py, diff_diff/imputation.py, diff_diff/two_stage.py, diff_diff/stacked_did.py, diff_diff/continuous_did.py, diff_diff/efficient_did.py, diff_diff/spillover.py, diff_diff/wooldridge.py, diff_diff/triple_diff.py, diff_diff/staggered_triple_diff.py, diff_diff/_staggered_triple_diff_engine.py, diff_diff/guides/llms-full.txt, diff_diff/guides/llms-practitioner.txt, docs/methodology/REGISTRY.md] + notes: "Family-wide anticipation domain validation: nine estimators (CS, SA, ImputationDiD, TwoStageDiD, StackedDiD, ContinuousDiD, EfficientDiD, WooldridgeDiD, SpilloverDiD) adopt the shared utils.validate_anticipation at __init__ plus a uniform fit-path mutation re-check in the assignment form (the validator now RETURNS the normalized Python int, adopted via assignment at every call site - constructor AND fit path - so numpy scalars, np.uint64 included, are normalized before any g-1-anticipation arithmetic can overflow). Prior state: seven constructors unvalidated; spillover validated only at the fit path with a bool hole; wooldridge validated >= 0 only, with bool + raw-TypeError-on-None/str holes. Not cosmetic - the value feeds the base-period rule and the NYT threshold: CS anticipation=-1 moved overall_att 2.18 -> 0.34 on the measurement fixture and flipped its sign under control_group='not_yet_treated'; True fit bit-identically to 1 (a silent one-period window); SunAbraham(anticipation=1.5) returned att=NaN without raising. Behavior delta: accepted -> loud ValueError; whole-valued floats that previously fit identically to their int now raise on CS/SA/Imputation/TwoStage/EfficientDiD/Wooldridge; numpy-integer inputs previously survived into the public attribute and get_params() as numpy scalars and are now retyped to built-in int; Wooldridge's message text changed and its constructor error ORDERING moved (the anticipation raise now fires at the assignment, after the bootstrap_weights/vcov_type/df_convention checks); Spillover's raise moved to construction with the fit-path re-check retained. No in-repo caller passed out-of-domain values. introduced_in 4.0: shipped post-3.9.0-cut, so the locked ladder's next release (4.0, tests/test_naming_guard.py _NEXT_RELEASE) is the first release carrying it; rowed per the M-096/M-142 shape precedent; status done is terminal so the row gates nothing further. Deprecated StaggeredTripleDifference stays construction-permissive by design (frozen 3.x shape; the shared engine validates at fit)." diff --git a/docs/v4-design.md b/docs/v4-design.md index 817937c4f..6edbb4e1f 100644 --- a/docs/v4-design.md +++ b/docs/v4-design.md @@ -826,7 +826,7 @@ above; anything only one PR cares about stays in that PR's plan.** | 2: contract foundations | 3.9 | (a) results base + unified event-study representation [M-092] + to_dict completion + the Diagnostic marker base on the diagnostic result roster [M-091] (section 3.5); (b) `aggregate()` + fit(aggregate=) shims [M-020..M-027] [M-139] (M-020's shim already shipped; M-139 is the HAD workflow twin, a pre-cut amendment); (c) param renames [M-030..M-047] [M-084] [M-086..M-089] + their results-field mirrors [M-094] [M-095] (section 8 rule 9) + the public-function completeness sweep [M-097..M-113] (section 8 rule 10) + the dCDH results mirror [M-114] + the fourth `robust` site [M-115] + the 2(c)-ii missed-rename amendments [M-136..M-138] (LPDiD `level` value; the two post-dummy diagnostics params) + BaseEstimator mixin + ContinuousDiD covariates move; (d) alias introduction [M-062] (the Spillover introduction is cancelled [M-063]) + the alias-diet `__getattr__` warning shim [M-135] + wrapper deprecations [M-070..M-077] + the two inference-surface policies: `n_bootstrap` semantic unification [M-081] and the wild-cluster-bootstrap roster guard [M-096]; shipped insertions (all done): the aggregate contract [M-122], the ETWFE reference-period family [M-123] [M-124] [M-125], and the variance-consolidation program [M-126] [M-127] | | 3: merges | 3.9 | (a) TWFE event-study mode [M-010] + EventStudy warn [M-060] + the fit `time`->`post` rename [M-082] (gates: section 4.1's equivalence/divergence/pooled-parity test triple) (shipped: tests/test_v4_merge_mpd.py; consumer ports incl. HonestDiD/PreTrendsPower calendar routes); (b) TripleDifference facade [M-013] + the SDDD alias [M-064] (shipped: tests/test_v4_merge_ddd.py; the engine relocation into the private shared mixin, the keyword-only staggered fit params, and the pscore_trim tightening [M-142]. The fit-time aggregate=/balance_e= carve-out rows are scheduled REMOVALS and so are cited in the phase-5 cell, not here - their only lifecycle version is removed_in 4.0); (c) CiC method= [M-015] + its results-field mirror [M-143] (shipped: tests/test_v4_merge_cic.py; method= is keyword-only and lowercase-only, the QDiD CLASS is deprecated while the METHOD is not, and ChangesInChangesResults.estimator -> .method carries a dual-key to_dict() window through 3.9) | | 4: release + soak | 3.9 cut | Migration guide written (skeleton: section 10); maintainer cuts 3.9; maint/3.8 rule active | -| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080]; merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; docs/llms.txt/README refresh | +| 5: enforcement | 4.0 | Removals [M-010..M-015, M-020..M-027, M-139, M-030, M-032..M-047 old names, M-060, M-061, M-064, M-070..M-077, M-084, M-086..M-089, M-001..M-003, M-117, M-118, M-119, M-120, M-140, M-141, M-143] + the alias diet [M-132]..[M-134] + the amendment's old names [M-094] [M-095] [M-097..M-115] [M-136..M-138] (incl. their consumer migrations and the `clean_control` serialized reporting key); M-031's old `time` name persists as the merged class's calendar column, so it is deliberately absent from the removal roster (its 4.0 enforcement is the M-085 behavior entry below); property window: [M-016] property-flips at 4.0 (removal at 5.0); storage flips [M-050..M-058]; default policies [M-004..M-006, M-128..M-131, M-080]; merged-class behavior enforcements [M-083] [M-085]; warning retirement [M-007]; fastpath go/no-go [M-008]; diagnostic-family docs/roster reorganization [M-090]; sentinel retirement [M-093]; the family-wide anticipation validation [M-144] (behavior tightening landing at 4.0 - shipped post-3.9-cut, terminal `done`, no removal/deprecation fields); docs/llms.txt/README refresh | | 6: front door | 4.1 | `event_study(data, outcome, unit, time, first_treat, estimator=...)` comparison entry point over the staggered family (sketch only; specified in its own plan) | Citation semantic for the table: a cell may cite a row whose current `phase` @@ -1086,14 +1086,15 @@ forever - a removed symbol resurrecting is a test failure. class/function rows and alias rows also assert `__all__` membership consistent with their status (stale `import *` entries fail). The shipped row ids are a - committed snapshot in the enforcement test (121 as of 2(b) PR-4's - HAD workflow-aggregate row: Phase 1 + the diagnostic-family + committed snapshot in the enforcement test (126 as of the family-wide + anticipation validation row: Phase 1 + the diagnostic-family amendment + the M-092/M-093 results-contract rows + the M-094..M-096 amendment rows + the M-097..M-115 completeness sweep + M-117..M-120/M-122 + the ETWFE reference-period pair M-123/M-124 + M-125 + M-126 + M-127..M-131 + the alias-diet family M-132..M-135 + the 2(c)-ii amendments - M-136..M-138 + M-139; + M-136..M-138 + M-139 + the DDD-merge rows M-140..M-142 + the CiC + results-field mirror M-143 + the anticipation policy row M-144; the snapshot extends by a new id range in the same diff that appends rows): ids are never deleted or reused, and the test fails if any snapshot id disappears. diff --git a/tests/test_anticipation_policy.py b/tests/test_anticipation_policy.py new file mode 100644 index 000000000..ecebe0be9 --- /dev/null +++ b/tests/test_anticipation_policy.py @@ -0,0 +1,317 @@ +"""Family-wide ``anticipation`` domain policy (ledger row M-144). + +The contract: every anticipation-taking estimator validates the param via +the shared ``diff_diff.utils.validate_anticipation`` at ``__init__`` +(``set_params`` inherits transactionally via the BaseEstimator probe +re-init) AND re-checks it on the fit path — the uniform direct-mutation +defense, using the ASSIGNMENT form ``self.anticipation = +validate_anticipation(self.anticipation)`` so a mutated numpy scalar is +normalized to a Python ``int`` before any ``g - 1 - anticipation``-style +arithmetic can overflow on an unsigned scalar. + +Roster notes: + +- SpilloverDiD's fit-path pins live in ``tests/test_spillover.py`` (its + keyword-only XOR ``treatment``/``first_treat`` interface doesn't fit + this suite's shared invocation); the other adopters are pinned HERE in + the mutation lanes. +- The deprecated ``StaggeredTripleDifference`` stays + construction-permissive BY DESIGN (frozen 3.x API shape; the shared + staggered engine validates at fit) — pinned by + test_v4_merge_ddd.py::test_deprecated_sibling_also_fails_closed_on_anticipation, + and deliberately NOT in this roster. + +Message pins match the FULL text via ``re.escape`` (repo convention), +except numpy-scalar values whose repr is NEP-51-dependent — those use the +shared prefix, matching the existing test_v4_merge_ddd pin. +""" + +import inspect +import re +import warnings + +import numpy as np +import pandas as pd +import pytest + +import diff_diff +from diff_diff import ( + CallawaySantAnna, + ContinuousDiD, + EfficientDiD, + ImputationDiD, + SpilloverDiD, + StackedDiD, + StaggeredTripleDifference, + SunAbraham, + TripleDifference, + TwoStageDiD, + WooldridgeDiD, + generate_staggered_data, +) +from diff_diff._base import BaseEstimator + +# =========================================================================== +# Constants +# =========================================================================== + +ANTICIPATION_MSG_PREFIX = "anticipation must be a non-negative integer" + +# The nine sweep adopters + TripleDifference (validated since birth). +VALIDATED_CLASSES = [ + CallawaySantAnna, + SunAbraham, + ImputationDiD, + TwoStageDiD, + StackedDiD, + ContinuousDiD, + EfficientDiD, + WooldridgeDiD, + SpilloverDiD, + TripleDifference, +] + +_CTOR_KWARGS = {SpilloverDiD: {"rings": [0.0, 100.0]}} + +# The full nine-value set from test_v4_merge_ddd.py's staggered-DDD pin. +BAD_VALUES = [-1, -5, 1.5, 0.5, "1", None, True, False, np.float64(2.0)] + + +def _make(cls, **overrides): + return cls(**{**_CTOR_KWARGS.get(cls, {}), **overrides}) + + +def _expected_message(value): + """The validator's full message for ``value``, per its branch rule. + + ONLY non-bool negative ints take the NEGATIVE branch (no type suffix); + everything else — bools included, because the ``isinstance(..., bool)`` + guard precedes the integer check — takes the TYPE branch. + """ + if isinstance(value, int) and not isinstance(value, bool) and value < 0: + return f"{ANTICIPATION_MSG_PREFIX}; got {value!r}." + return f"{ANTICIPATION_MSG_PREFIX}; got {value!r} (type {type(value).__name__})." + + +def _match_for(value): + """re pattern for ``value``: full text, except NEP-51-variable reprs. + + ``np.float64(2.0)`` reprs as ``np.float64(2.0)`` on numpy>=2 but ``2.0`` + on the declared 1.20 floor, so numpy scalars pin the PREFIX only (the + same reason the existing test_v4_merge_ddd pin is prefix-matched). + """ + if isinstance(value, np.generic): + return re.escape(ANTICIPATION_MSG_PREFIX) + return re.escape(_expected_message(value)) + + +_ids = [f"{v!r}" for v in BAD_VALUES] + + +# =========================================================================== +# Roster guard +# =========================================================================== + + +class TestAnticipationRoster: + def test_anticipation_exposed_by_exactly_the_validated_roster(self): + discovered, seen = [], set() + for name in diff_diff.__all__: + obj = getattr(diff_diff, name) + if not isinstance(obj, type) or id(obj) in seen: + continue + seen.add(id(obj)) + if issubclass(obj, BaseEstimator): + params = inspect.signature(obj.__init__).parameters + if "anticipation" in params: + discovered.append(obj) + assert set(discovered) == set(VALIDATED_CLASSES) | {StaggeredTripleDifference}, ( + "The `anticipation` roster changed. A future estimator exposing " + "`anticipation` must join the family-wide validation policy " + "(ledger row M-144): validate via utils.validate_anticipation at " + "__init__ AND re-check on the fit path via the assignment form. " + "The deprecated StaggeredTripleDifference is the one documented " + "construction-permissive exception (fit-validated via the shared " + "engine)." + ) + + +# =========================================================================== +# Lane 1: bad values raise at __init__ +# =========================================================================== + + +class TestConstructorValidation: + @pytest.mark.parametrize("cls", VALIDATED_CLASSES) + @pytest.mark.parametrize("value", BAD_VALUES, ids=_ids) + def test_bad_value_raises_at_init(self, cls, value): + with pytest.raises(ValueError, match=_match_for(value)): + _make(cls, anticipation=value) + + +# =========================================================================== +# Lane 2: boundary / accepted values normalize to Python int +# =========================================================================== + + +class TestAcceptedValuesNormalize: + @pytest.mark.parametrize("cls", VALIDATED_CLASSES) + @pytest.mark.parametrize( + ("value", "expected"), + [(0, 0), (1, 1), (np.int64(2), 2), (np.uint64(2), 2)], + ids=["0", "1", "np.int64(2)", "np.uint64(2)"], + ) + def test_accepted_value_round_trips_as_int(self, cls, value, expected): + est = _make(cls, anticipation=value) + # `type is int` is the real pin: `np.uint64(2) == 2` is True even + # without normalization, so a bare equality check would be vacuous. + assert type(est.anticipation) is int + assert est.anticipation == expected + assert type(est.get_params()["anticipation"]) is int + + +# =========================================================================== +# Lane 3: set_params raises and rolls back +# =========================================================================== + + +class TestSetParamsTransactional: + @pytest.mark.parametrize("cls", VALIDATED_CLASSES) + def test_set_params_bad_value_rolls_back(self, cls): + est = _make(cls, anticipation=1) + before = est.get_params() + with pytest.raises(ValueError, match=re.escape(ANTICIPATION_MSG_PREFIX)): + est.set_params(anticipation=-1) + assert est.get_params() == before + + +# =========================================================================== +# Lane 4: fit-time direct-mutation defense +# =========================================================================== + +# The nine minus SpilloverDiD (whose fit-path pins live in +# tests/test_spillover.py) and TripleDifference (staggered fit needs a +# partition column; covered in lane 4b instead). +_FIT_MUTATION_CLASSES = [ + CallawaySantAnna, + SunAbraham, + ImputationDiD, + TwoStageDiD, + StackedDiD, + ContinuousDiD, + EfficientDiD, + WooldridgeDiD, +] + + +def _fit_kwargs_for(cls): + kwargs = dict(outcome="y", unit="u", time="t", first_treat="g") + if cls is ContinuousDiD: + kwargs["dose"] = "d" + return kwargs + + +class TestFitMutationDefense: + @pytest.mark.parametrize("cls", _FIT_MUTATION_CLASSES) + def test_mutated_anticipation_raises_at_fit(self, cls): + """``True`` is the discriminating mutation: bool is an int subclass + and not negative, so any legacy ``< 0``-style check accepts it — + only the shared helper rejects it. Each estimator's re-check runs + before any column handling, so a bare empty DataFrame suffices.""" + est = _make(cls) + est.anticipation = True + with pytest.raises(ValueError, match=re.escape(ANTICIPATION_MSG_PREFIX)): + est.fit(pd.DataFrame(), **_fit_kwargs_for(cls)) + + +# =========================================================================== +# Lane 4b: fit-path re-check NORMALIZES (assignment form, not validate-only) +# =========================================================================== + + +@pytest.fixture(scope="module") +def small_panel(): + """Public DGP: columns unit / period / outcome / first_treat.""" + return generate_staggered_data(n_units=30, n_periods=4, seed=1) + + +_PANEL_FIT_KWARGS = dict(outcome="outcome", unit="unit", time="period", first_treat="first_treat") + + +class TestFitNormalization: + @pytest.mark.parametrize( + "cls", + [ + CallawaySantAnna, + SunAbraham, + ImputationDiD, + TwoStageDiD, + StackedDiD, + ContinuousDiD, + EfficientDiD, + WooldridgeDiD, + ], + ) + def test_mutated_numpy_scalar_normalized_by_fit(self, cls, small_panel): + """A mutated ``np.uint64`` must be re-assigned as a Python int by + the fit-path re-check BEFORE any ``g - 1 - anticipation`` arithmetic + (which raises OverflowError on unsigned scalars on SunAbraham / + TwoStageDiD / StackedDiD, measured on numpy 2.4.5). A validate-only + call that discards the helper's return fails this lane.""" + df = small_panel + if cls is ContinuousDiD: + df = df.copy() + rng = np.random.default_rng(0) + dose_map = { + u: (rng.uniform(0.5, 2.0) if ft > 0 else 0.0) + for u, ft in df.groupby("unit")["first_treat"].first().items() + } + df["dose"] = df["unit"].map(dose_map) + est = _make(cls) + est.anticipation = np.uint64(1) + fit_kwargs = dict(_PANEL_FIT_KWARGS) + if cls is ContinuousDiD: + fit_kwargs["dose"] = "dose" + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + est.fit(df, **fit_kwargs) + assert type(est.anticipation) is int + assert est.anticipation == 1 + + def test_triple_difference_engine_normalizes(self, small_panel): + """Engine coverage: TripleDifference's staggered fit routes through + the shared ``_staggered_triple_diff_engine`` re-check, whose + assignment form must normalize a mutated numpy scalar too.""" + df = small_panel.copy() + df["eligible"] = (df["unit"] % 2 == 0).astype(int) + est = TripleDifference() + est.anticipation = np.uint64(1) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + est.fit(df, partition="eligible", **_PANEL_FIT_KWARGS) + assert type(est.anticipation) is int + assert est.anticipation == 1 + + +# =========================================================================== +# Lane 4c: hausman_pretest classmethod normalizes its own argument +# =========================================================================== + + +class TestHausmanPretestNormalization: + def test_uint64_matches_int_result(self, small_panel): + """``EfficientDiD.hausman_pretest`` uses ``anticipation`` in its OWN + event-time arithmetic (``e < -ant``). Without normalization, + ``-np.uint64(1)`` silently WRAPS to 2**64-1 (no OverflowError) and + the pretest degrades to an all-NaN inconclusive result — so the pin + must compare against the Python-int result, not just assert no + exception.""" + kwargs = dict(_PANEL_FIT_KWARGS) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + r_int = EfficientDiD.hausman_pretest(small_panel, anticipation=1, **kwargs) + r_u64 = EfficientDiD.hausman_pretest(small_panel, anticipation=np.uint64(1), **kwargs) + assert np.isfinite(r_int.statistic) and np.isfinite(r_int.p_value) + assert r_u64.statistic == r_int.statistic + assert r_u64.p_value == r_int.p_value + assert r_u64.df == r_int.df diff --git a/tests/test_spillover.py b/tests/test_spillover.py index 723f09b0c..f10334b4e 100644 --- a/tests/test_spillover.py +++ b/tests/test_spillover.py @@ -4,6 +4,7 @@ Step 2+ surfaces are added incrementally as the implementation lands. """ +import re import warnings from typing import Dict, Optional @@ -2433,18 +2434,43 @@ def test_normal_butts_dgp_does_not_trigger(self): class TestSpilloverDiDAnticipationValidation: """anticipation must be a non-negative integer. Round-9 codex review - caught that fractional / negative values silently shifted timing. + caught that fractional / negative values silently shifted timing; the + family-wide sweep (ledger row M-144) moved the raise to construction + via the shared ``utils.validate_anticipation``, keeping the in-fit + re-check as the direct-mutation defense. """ - @pytest.mark.parametrize("bad_value", [-1, 0.5, 1.5, -0.1]) - def test_invalid_anticipation_raises_treatment_path(self, bad_value): + @pytest.mark.parametrize( + ("bad_value", "message"), + [ + # -1 takes the validator's NEGATIVE branch (no type suffix); + # the floats take the TYPE branch. + (-1, "anticipation must be a non-negative integer; got -1."), + (0.5, "anticipation must be a non-negative integer; got 0.5 (type float)."), + (1.5, "anticipation must be a non-negative integer; got 1.5 (type float)."), + (-0.1, "anticipation must be a non-negative integer; got -0.1 (type float)."), + ], + ) + def test_invalid_anticipation_raises_at_construction(self, bad_value, message): + with pytest.raises(ValueError, match=re.escape(message)): + SpilloverDiD( + rings=[0.0, 100.0], + conley_coords=("lat", "lon"), + anticipation=bad_value, + ) + + def test_mutated_anticipation_raises_at_fit_treatment_path(self): + """Direct-mutation defense on the treatment path: ``True`` is the one + value the OLD inline fit check accepted (bool is an int subclass and + not ``< 0``), so this pin detects a failure to actually swap in the + shared helper — the in-fit re-check must reject it.""" df = _make_butts_2period_dgp(seed=42) est = SpilloverDiD( rings=[0.0, 100.0], conley_coords=("lat", "lon"), - anticipation=bad_value, ) - with pytest.raises(ValueError, match="anticipation"): + est.anticipation = True + with pytest.raises(ValueError, match="anticipation must be a non-negative integer"): est.fit(df, outcome="y", unit="unit", time="time", treatment="D") @@ -3464,10 +3490,13 @@ def test_horizon_max_zero_with_event_study_raises(self): with pytest.raises(ValueError, match="horizon_max=0 is not supported"): est.fit(df, outcome="y", unit="unit", time="time", first_treat="first_treat") - def test_non_numeric_anticipation_raises_targeted_value_error(self): - """PR #456 R2 P2: anticipation must be validated BEFORE the ref_period - compatibility check; otherwise `-1 - self.anticipation` would raise a - raw TypeError on non-numeric input instead of the targeted ValueError.""" + def test_mutated_anticipation_raises_targeted_value_error(self): + """Direct-mutation defense on the event-study path (ledger row + M-144): construction now validates eagerly, so the value is mutated + AFTER construction. ``True`` is what the OLD inline fit check + accepted (bool is an int subclass and not ``< 0``), so this pin + detects a failure to actually swap the in-fit check for the shared + helper.""" df = generate_butts_staggered_dgp(seed=1) est = SpilloverDiD( rings=[0.0, 50.0, 200.0], @@ -3475,13 +3504,17 @@ def test_non_numeric_anticipation_raises_targeted_value_error(self): conley_coords=("lat", "lon"), event_study=True, horizon_max=2, - anticipation="1", # type: ignore[arg-type] ) + est.anticipation = True with pytest.raises(ValueError, match="anticipation must be a non-negative integer"): est.fit(df, outcome="y", unit="unit", time="time", first_treat="first_treat") def test_none_anticipation_raises_targeted_value_error(self): - """Same P2 fix: None anticipation must surface the targeted ValueError.""" + """PR #456 R2 ordering pin: the in-fit anticipation re-check must run + BEFORE the ref_period compatibility arithmetic. This test KEEPS + ``None`` deliberately — `-1 - None` would raise a raw TypeError if + the in-fit check ever moved after the ref-period arithmetic, whereas + ``True`` arithmetic evaluates cleanly and cannot detect reordering.""" df = generate_butts_staggered_dgp(seed=1) est = SpilloverDiD( rings=[0.0, 50.0, 200.0], @@ -3489,8 +3522,8 @@ def test_none_anticipation_raises_targeted_value_error(self): conley_coords=("lat", "lon"), event_study=True, horizon_max=2, - anticipation=None, # type: ignore[arg-type] ) + est.anticipation = None with pytest.raises(ValueError, match="anticipation must be a non-negative integer"): est.fit(df, outcome="y", unit="unit", time="time", first_treat="first_treat") diff --git a/tests/test_v4_matrix.py b/tests/test_v4_matrix.py index 51f06367c..d75d3f268 100644 --- a/tests/test_v4_matrix.py +++ b/tests/test_v4_matrix.py @@ -128,11 +128,12 @@ # HAD workflow-aggregate row (M-139, next free id - the reserved pool is # spent/earmarked) = 121; + the phase-3(b) DDD merge rows (M-140/M-141 carry # fit-time aggregate=/balance_e= onto the surviving TripleDifference, M-142 the -# pscore_trim tightening) = 124. +# pscore_trim tightening) = 124; + the CiC results-field rename (M-143) = 125; +# + the family-wide anticipation validation row (M-144) = 126. # Ids are never reused and terminal rows are never deleted, so the ledger # only grows - raise the floor when rows are added; a lower parse count # means scanner/format drift or an illegal row deletion. -ROW_COUNT_FLOOR = 125 +ROW_COUNT_FLOOR = 126 # Committed snapshot of the shipped id set ("ids are never deleted or reused" # contract - a delete-one-add-one edit keeps the count above the floor but trips @@ -186,6 +187,7 @@ (139, 139), (140, 142), (143, 143), + (144, 144), ] EXPECTED_INITIAL_IDS = frozenset( f"M-{n:03d}" for lo, hi in _INITIAL_ID_RANGES for n in range(lo, hi + 1) @@ -584,14 +586,14 @@ def test_initial_ids_never_deleted(): """The shipped id set is immutable: ids are never deleted or reused (spec section 11). ROW_COUNT_FLOOR alone would let a delete-one-add-one edit pass; this snapshot cannot. - Extends as rows ship (121 as of 2b PR-4's HAD workflow-aggregate row: + Extends as rows ship (126 as of the family-wide anticipation validation row: Phase 1 + diagnostic-family + M-092/M-093 + M-094..M-096 + the M-097..M-115 public-function completeness sweep + M-117..M-120/M-122 + M-123/M-124 + M-125 + M-126 + M-127..M-131 + M-132..M-135 + - M-136..M-138 + M-139 + M-140..M-142).""" + M-136..M-138 + M-139 + M-140..M-142 + M-143 + M-144).""" missing = sorted(EXPECTED_INITIAL_IDS - set(_ROW_IDS)) assert not missing, f"ledger rows deleted (ids are permanent): {missing}" - assert len(EXPECTED_INITIAL_IDS) == 125 + assert len(EXPECTED_INITIAL_IDS) == 126 def test_version_tuple_pads_to_three_components(): @@ -1055,14 +1057,18 @@ def _changes_at_4_0(row): Keyed on the two LIFECYCLE version fields only - a symbol removed at 4.0, or one whose deprecation warning starts firing at 4.0 (the ``field-flip`` family, removed - at 5.0). 108 of the 125 rows qualify. + at 5.0). 108 of the 126 rows qualify. - The 17 that do not, and why (this enumeration is the contract - a reader of the + The 18 that do not, and why (this enumeration is the contract - a reader of the guide must be able to trust that nothing 4.0-relevant was dropped): - 12 ``behavior`` rows with ``introduced_in: 3.9`` and no dep/rem: already shipped in 3.9, so there is no 4.0 action. They get their own guide section, not an appendix row. + - 1 ``behavior`` row with ``introduced_in: 4.0`` and no deprecation/removal + fields (``M-144``, the post-cut anticipation validation tightening): it lands + AT 4.0 but removes/deprecates nothing, so it appears in the guide's + "Remaining 4.0 changes" prose, not the ledger-derived appendix. - ``M-062``, ``M-063``: aliases, introduce-only / all lifecycle fields null. - ``M-031``, ``M-082``: ``deprecated_in: 3.9`` with ``removed_in: null``, because the NAME ``time`` survives with a new meaning. Their 4.0 enforcement rows