Replace Java DUKE with a Splink matching engine

[FabianHofmann]

Note

AI generated draft
Fully experimental

Side experiment, companion to #301 (which swaps DUKE for a rapidfuzz+numpy backend). This one replaces the Java/JVM DUKE record-linkage engine with a pure-Python backend built on Splink + DuckDB, and makes it the sole matching engine. Matching no longer needs Java or the bundled DUKE .jar binaries.

Summary

powerplantmatching.linkage.match() builds a fixed-parameter Splink (Fellegi-Sunter) model: the per-field m/u probabilities are set directly from the former DUKE field weights (Comparison.xml / Deleteduplicates.xml), so the model needs no per-call EM training and is deterministic and stable across the many small per-country slices the pipeline produces. A 0.5 prior reproduces DUKE's belief update; only the relative Bayes factor between levels matters for ranking, so one calibrated match_probability threshold per mode sets the operating point.

Rationale and benchmark numbers are in analysis/linkage_findings.md.

Changes

New engine

• powerplantmatching/linkage.py — match() (list of two frames → record linkage, single frame → deduplication). Jaro-Winkler name levels, exact categoricals, capacity min/max ratio, haversine distance ladder.
• analysis/benchmark_splink.py — reproducible ground-truth + benchmark harness.

Removed

• duke.py, package_data/duke_binaries/*.jar (520K), Comparison.xml, Deleteduplicates.xml
• LICENSES/Apache-2.0.txt (only the DUKE jars used it) and its REUSE / MANIFEST / docs references
• test/test_duke.py

Wiring & config

• matching.py / cleaning.py call match() directly; **dukeargs → **kwargs across matching.py, collection.py, accessor.py
• Rename parallel_duke_processes → parallel_processes
• splink>=4 added to pyproject.toml dependencies; docs (API reference, license notes, release notes)

Results (vs DUKE, ground truth from powerplants.csv)

Record linkage (GEO × GPD, 12 countries, GT = 567 pairs):

backend	precision	recall	F1	time
DUKE	0.790	0.771	0.780	6.7 s
splink (th 0.70, default)	0.778	0.718	0.747	2.6 s
splink (th 0.50)	0.700	0.802	0.748	2.6 s

~2.6× faster, within ~3 F1-points; recall exceeds DUKE at a lower threshold (raw recall ceiling 0.82).

Deduplication (GPD, 8 countries): the powerplants.csv dedup clusters are DUKE's own output (circular), so the fair metric is merge behaviour — Splink makes 459 merges vs DUKE's 454, 86 % of them principled (same fueltype + within 5 km + high name similarity), at ~7× speed (3.2 s vs 21.7 s).

Compatibility

⚠️ Breaking:

• Java is no longer required (and no longer used).
• parallel_duke_processes is renamed to parallel_processes — existing configs must update the key.
• Direct imports of powerplantmatching.duke no longer resolve.

Testing

• test/test_linkage.py — 6 passed (format parity, identical-row matching, singlematch uniqueness, empty input, symmetric dedup pairs, geo proximity raises score)
• test/test_cleaning.py — 3 passed (real-data aggregate_units → match() dedup path)
• ruff check clean; reuse lint compliant (used licenses: MIT, CC0-1.0, CC-BY-4.0)

Draft — opening for review of the fixed-parameter modelling choice (vs Splink's usual unsupervised EM training, which is unstable on the per-country slices) and the threshold calibration. The DUKE baseline in benchmark_splink.py must be run on master since the binaries are removed here.