CausalLab — A/B testing and experiment statistics, built and evaluated from scratch

Most teams can run the test. Fewer can say what the p-value actually cost them.
Exact Student-t p-values, Wilson intervals, power and sample size, CUPED variance reduction, and O'Brien-Fleming / Pocock sequential monitoring — no scipy, no statsmodels.

---

What this is

The full experiment-analysis stack, one module per piece:

• stats — normal CDF (via math.erf), inverse normal (Acklam's algorithm), the regularized incomplete beta (Lentz's continued fraction), and the tests built on them: one- and two-proportion z, Welch's t
• intervals — Wilson (score) and Wald intervals, difference-of-proportions CI, Welch mean-difference CI
• power — achieved power and required sample size for two-proportion and Welch t designs
• cuped — CUPED variance reduction from a pre-experiment metric
• sequential — Bonferroni and Lan-DeMets alpha spending (O'Brien-Fleming and Pocock), with a group-sequential monitor for a running A/B
• data — seeded synthetic experiments (binary, continuous, pre/post) with known ground truth

Every p-value is exact for its model — the Student-t one goes through the incomplete beta, not a normal approximation, and is pinned to textbook values in the tests.

The standard eval

Four analyses over seeded 1000-per-arm experiments:

$ causalab demo
binary     p_c 0.0950  p_t 0.1660  z +4.713  p 0.0000  significant
continuous t +3.860 (df 1997.8)  p 0.0001  significant
cuped      reduction 0.526  p_raw 0.0000  p_adj 0.0000
sequential crossed=True  null_crossed=False

• Binary A/B (injected lift 0.10 → 0.15): the pooled z-test fires at z = 4.71; the report shows both Wilson intervals, the 95% CI on the difference, the relative lift, the achieved power at this n, and the n that would have been required (683/arm for 80% power — the test ran 47% oversized).
• Continuous A/B (10.0 → 10.5, sd 2): Welch's t = 3.86 with the exact incomplete-beta p-value.
• CUPED: the synthetic pre metric is built to explain beta²·Var(pre) of the post variance; the realized theta is 0.799 (true 0.8) and the variance drops 52.6% — right on the theoretical β²σ²ₚᵣₑ / (β²σ²ₚᵣₑ + σ²ₙₒᵢₛₑ) = 2.56/5.06 ≈ 0.51.
• Sequential: the live experiment (true lift) crosses the O'Brien-Fleming boundary at look 2 of 4; the null experiment (no lift) never crosses. Both are asserted by tests, not just shown.

causalab report --out report.html renders the decision tables, the power/sample-size card, the CUPED raw-vs-adjusted comparison, and the sequential |z| trajectory against the boundary (live vs. null overlaid).

Quickstart

pip install -e .

# the full eval
causalab demo --json

# a two-proportion z-test on a seeded A/B
causalab test --p0 0.10 --p1 0.15 --n 1000

# power / sample size
causalab power --p0 0.10 --p1 0.15              # n required for 80% power
causalab power --p0 0.10 --p1 0.15 --n 1000     # achieved power

# CUPED on a pre/post experiment
causalab cuped --effect 0.4

# sequential monitoring (OBF or Pocock)
causalab sequential --looks 4 --method obf

# the HTML report
causalab report --out report.html

Everything accepts --json, and every seed is fixed by default so runs are reproducible.

Using it as a library

from causalab import two_proportion_z, wilson, sample_size_two_proportion

z, p, pool = two_proportion_z(150, 1000, 100, 1000)   # treatment, control
print(z, p, wilson(150, 1000))
print(sample_size_two_proportion(0.10, 0.15, power=0.80))  # 683 per arm

Why it's built this way

• Exact p-values, audited against tables. The t-distribution p-value is the regularized incomplete beta I_{df/(df+t²)}(df/2, ½) via Lentz's continued fraction. The tests pin it to the textbook: p(t = 2.228133, df = 10) = 0.05 to 1e-5, the Cauchy identity p(t = 1, df = 1) = 0.5 to 1e-9, and the large-df approach to the normal tail.
• The spending functions spend exactly alpha. Both O'Brien-Fleming (2 − 2Φ(Φ⁻¹(1−α/2)/√t)) and Pocock (α·ln(1 + (e−1)t)) are checked at the endpoints (0 at t→0, α at t=1), for monotonicity, and that their per-look increments sum to α to 1e-12. The OBF shape (huge early boundary, final boundary above 1.96) and the Pocock shape (flat-ish, fast early spend) are both asserted.
• Honest baselines in the sequential demo. The same monitor is run on a null experiment with the same seed structure; the test requires the live one to cross and the null one not to. A sequential procedure that can't be shown not to false-alarm on a fixed seed is just a plot.
• CUPED is tested both ways. With a real effect the adjusted p must be ≤ the raw p (and both significant); on a no-effect pre/post the adjusted p must not manufacture significance.
• Sample size walks to the minimum. The closed-form starting point can overshoot by a step; sample_size_two_proportion walks back to the smallest n with achieved power ≥ target, and the test asserts power(n) ≥ 0.8 > power(n−1).

The report

report.render_report produces a single-file HTML document:

• Binary A/B — per-arm n / conversions / rate / Wilson 95%, then the decision row (diff, CI, lift, z, p, verdict)
• Continuous A/B — Welch t row with df, CI, verdict
• Power & sample size — achieved power at the actual n vs. the n required
• CUPED — raw vs. adjusted (diff, t, p) with theta and the variance-reduction share
• Sequential — |z| trajectory per look against the OBF boundary, live and null overlaid, with the per-look table
• No external assets, no script tags, prints clean, byte-identical on re-run

_{Sample report — rendered from the seeded standard eval via causalab report --out report.html.}

Tests

52 offline tests, no network, no GPU:

• stats: Φ and Φ⁻¹ identities (symmetry, round-trip to 1e-6, known quantiles), the incomplete beta (endpoint/complement identities, a hand-integrated value), t p-values pinned to the table, hand-computed z and Welch t values (z = 5/3 exact; Welch df = 4 exact on the 3-point example)
• intervals: Wilson known value for 10/100, bounds-in-[0,1] properties, Wald comparison, hand-computed difference CIs (including t₀.₉₇₅,₄ = 2.776445)
• power: known achieved power (0.9237 at n=1000), monotonicity in n / effect / power target, the round-trip power(n) ≥ 0.8 > power(n−1)
• cuped: theta recovery (±0.05 of the true 0.8), variance reduction in the theoretical band, p-sharpening with a real effect, no manufactured significance on a null
• sequential: spend-function endpoints and monotonicity, spends summing to α to 1e-12, OBF/Pocock boundary shapes, cross-for-live-and-not-for-null on fixed seeds, determinism
• benchmark + report + CLI: structure, determinism, decision sanity (the injected effects are all detected), byte-identical HTML, no external assets, exit codes, --json shape

python -m pytest tests -q

Design notes

• Normal quantile without tables. Acklam's rational approximation covers the central region with two-piece rational functions and the tails with the symmetry transform; the tests check round-trips to 1e-6 across the central and tail regions.
• The incomplete beta uses the symmetry branch. I_x(a,b) = 1 − I_{1−x}(b,a) when x ≥ (a+1)/(a+b+2), with Lentz's method on the continued fraction in each branch — the standard Numerical-Recipes construction, re-implemented here because the project's rule is zero dependencies.
• Welch, not Student. The t-test uses the Welch–Satterthwaite degrees of freedom, which is what "unequal variance A/B" actually needs; the equal-variance case falls out as a special case.
• Spending → discrete construction. Per-look critical values are Φ⁻¹(1 − Δᵢ/2) from the cumulative-spend increments (Jennison–Turnbull), so the sum of per-look type-I budgets is exactly α by construction.
• Wilson over Wald by default in the report. Wald intervals can leave [0, 1] and misbehave near the boundaries; Wilson (score) stays inside and is the one shown.

Roadmap

• Fisher's exact test for small-count experiments (hypergeometric, exact)
• A full group-sequential power computation (simulating the boundary-crossing distribution on the seeded normal lattice) rather than the normal-approximation power
• Mixed-effects and cluster-randomized adjustments for experiments with grouped units
• Delta-method CIs for ratio metrics (revenue per user), where the plain mean CI is wrong
• Interference-aware designs (switchback testing) for marketplace experiments

---

© 2026 Adithya N Raj