End-to-end football analytics pipeline engineered for production. Provider-agnostic ingestion via a Dart microservice (StatsBomb · Opta · Wyscout adapter pattern), a four-layer dbt medallion architecture with a universal 105×68 m coordinate system, calibrated XGBoost models for xG/xP, value-iteration xT, cosine-similarity recruitment model, K-Means set-piece clustering, and a fully automated BI visualisation layer — all validated through GitHub Actions CI on 147 World Cup 2022 matches (524,457 events).
Five portfolio-grade charts are generated automatically on every CI run and uploaded as a 90-day artifact. The images below are sourced from the latest successful pipeline run.
| xT Surface Heatmap | Shot Map with xG Bubbles |
|---|---|
 |
 |
| Team xG vs Goals Scored | Set-Piece Delivery Clusters |
|---|---|
 |
 |
| Player xP Ranking (Top 20) |
|---|
 |
Charts are generated by
scripts/visualise.pyusing mplsoccer and matplotlib on live pipeline data. The latest PNG artifacts are available under Actions → xForge V2 Pipeline → analytics-visualisations-{run_id}.
StatsBomb Open Data / Opta F24 / Wyscout v3
│
▼
Dart Ingestion Service HTTP :8090 · Adapter Pattern
├── StatsBombAdapter 120×80 → UnifiedEvent
├── OptaAdapter 100×100 → UnifiedEvent
└── WyscoutAdapter 100×100 → UnifiedEvent
│
▼
PostgreSQL 15 · fact_events LIST-partitioned by competition_id
│
▼
dbt Medallion Pipeline
├── Bronze Type-cast pass-through, provider tests
├── Silver 105×68 m normalisation, spatial range tests
│ └── silver_pass_links Recipient extraction for pass network
├── Gold Player / team aggregations with ML values
└── Marts BI-ready tables — player metrics, leaderboards
├── mart_pressing_metrics PPDA per team per match
└── mart_pass_network Player-to-player pass pairs
│
├── ML Models (run before Gold/Marts — write back to fact_events)
│ ├── xG Model XGBoost · calibrated goal probability per shot
│ ├── xT Model Value iteration · 16×12 grid · 192 cells
│ ├── xP Model XGBoost · pass completion probability (AUC 0.897)
│ └── Player Sim. cosine NearestNeighbors · top-10 similar players
│
├── Materialized Views mv_team_xg · mv_shot_locations (zero-downtime refresh)
├── BI Visualisations 5 PNG charts · mplsoccer · 90-day CI artifacts
├── PDF Match Report 5-page mplsoccer report per match
└── SportsCode XML Top-25 xT events · Hudl-compatible
flowchart TD
SB["StatsBomb Open Data\n147 matches · 524,457 events"]
OA["Opta F24 JSON\nvia OptaAdapter"]
WY["Wyscout v3 JSON\nvia WyscoutAdapter"]
subgraph DART["Dart Ingestion Service — :8090"]
SA["StatsBombAdapter\n120×80"]
OPT["OptaAdapter\n100×100"]
WSC["WyscoutAdapter\n100×100"]
PW["PostgresWriter\nON CONFLICT DO NOTHING"]
end
subgraph PG["PostgreSQL 15"]
FE["fact_events\nLIST-partitioned · competition_id\nxt_value · xp_value · xg_value"]
DIM["dim_matches · dim_players\ndim_teams · dim_competitions"]
AUX["xt_surface 192 cells\nset_piece_clusters\nmodel_registry\nmv_team_xg\nplayer_similarity_scores"]
end
subgraph DBT["dbt Medallion"]
BR["Bronze\ntype-cast pass-through"]
SL["Silver\n105x68 m normalisation\nspatial range tests\nsilver_pass_links"]
GD["Gold\nplayer + team aggregations"]
MT["Marts\nmart_pressing_metrics · mart_pass_network\nmart_player_metrics · mart_team_summary"]
end
subgraph ML["ML Pipeline"]
XG["xG Model\nXGBoost · calibrated"]
XT["xT Model\nValue iteration · 16x12"]
XP["xP Model\nXGBoost · AUC 0.897"]
KM["K-Means\nSet-piece clusters k=6"]
PS["Player Similarity\ncosine NearestNeighbors · top-10"]
end
subgraph SERVE["Serving Layer"]
VIZ["BI Visualisations\n5 PNG charts · CI artifacts"]
PDF["PDF Match Report\n5-page mplsoccer"]
XML["SportsCode XML\n25 top-xT events · Hudl"]
SUP["Apache Superset\n7 charts · dashboard"]
end
SB --> SA
OA --> OPT
WY --> WSC
SA & OPT & WSC --> PW --> FE & DIM
DIM & FE --> BR --> SL
SL --> XG & XT & XP & KM & PS
XG & XT & XP & KM --> FE
PS --> AUX
FE --> GD --> MT
MT & AUX --> VIZ & PDF & XML & SUP
Returns {"status":"ok"} — used by Docker/K8s liveness probes. Always public.
Ingests events for a single match.
Request:
{"provider": "statsbomb", "match_id": 3869685}Optional for Opta / Wyscout (file):
{"provider": "wyscout", "match_id": 1234567, "file_path": "/data/wyscout_match.json"}Optional for Wyscout (API with Bearer token):
{"provider": "wyscout", "match_id": 1234567, "url": "https://...", "api_key": "TOKEN"}Response (200):
{"provider": "statsbomb", "match_id": 3869685, "written": 3401}Error responses: 400 (invalid input), 401 (bad token), 502 (upstream fetch failed)
Authentication: Bearer token via Authorization: Bearer <API_TOKEN> header. Bypassed when API_TOKEN env var is not set (local dev / CI only).
Rate limiting: 60 requests per minute per IP (sliding window). HTTP 429 with Retry-After header when exceeded.
Security: HTTPS-only upstream URLs enforced in all adapters (SSRF protection). Bearer token compared in constant time (timing-attack resistance). Request body capped at 10 MB (DoS protection).
Returns xG values for all shot events in a match. Backed by a partial index on fact_events(match_id) WHERE xg_value IS NOT NULL.
Query params: limit (default 200, max 2000) · offset (default 0)
Response (200):
{
"match_id": 3869685,
"limit": 200,
"offset": 0,
"events": [
{"event_id": "a1b2c3d4-...", "xg_value": 0.142},
{"event_id": "e5f6g7h8-...", "xg_value": 0.031}
]
}Error responses: 400 (invalid match_id or params), 401, 429 (rate limit)
Returns up to 10 most similar players by cosine similarity across 12 aggregated features (goals/90, xG/90, progressive passes, pressures, etc.). Requires player_similarity.py to have been run first to populate player_similarity_scores.
Query params: position (optional) — filters results to a specific position group (e.g. CM, Left Center Back). Letters and spaces only, max 60 chars. Blocks <script>, ../, and control characters.
Response (200):
{
"player_id": 5503,
"similar": [
{"player_id": 6012, "player_name": "Luka Modrić", "position": "CM", "similarity_score": 0.947, "rank": 1},
{"player_id": 7234, "player_name": "Toni Kroos", "position": "CM", "similarity_score": 0.931, "rank": 2}
]
}Error responses: 400 (invalid player_id or position format), 401, 429
Decision-ready match intelligence: was the scoreline fair, which team pressed harder, and who created the most threat?
Response (200):
{
"match_id": 3869685,
"home_team": "Barcelona",
"away_team": "Real Madrid",
"home_score": 4,
"away_score": 0,
"xg_home": 2.81,
"xg_away": 0.43,
"xg_diff": 2.38,
"pressing_intensity": {
"home": {"label": "High", "ppda": 6.2},
"away": {"label": "Medium", "ppda": 11.7}
},
"key_threats": [
{"rank": 1, "player_name": "Lionel Messi", "team": "Barcelona", "total_xt": 0.412},
{"rank": 2, "player_name": "Luis Suárez", "team": "Barcelona", "total_xt": 0.287},
{"rank": 3, "player_name": "Andrés Iniesta", "team": "Barcelona", "total_xt": 0.241}
]
}| Field | Meaning |
|---|---|
xg_diff |
home_xg − away_xg. Positive = home team had the better of it on quality chances. |
pressing_intensity.label |
PPDA-derived: High < 10 · Medium 10–15 · Low > 15 |
pressing_intensity.ppda |
Passes allowed per defensive action. Lower = more aggressive press. null when insufficient data. |
key_threats |
Top-3 players by cumulative xT (passes + carries into higher-threat zones). |
Error responses: 400 (invalid match_id), 401, 404 (match not found), 429
Returns top-N statistically similar players with per-stat deltas and a plain-English recruitment recommendation. Backed by the pre-computed cosine similarity model (player_similarity.py).
Query params: top_n (default 3, max 10)
Response (200):
{
"target": {
"player_id": 5503,
"player_name": "Luka Modrić",
"position": "CM",
"stats": {"avg_xg": 0.042, "pass_completion_pct": 88.3, "avg_xt_per_pass": 0.0041, "total_shots": 47}
},
"candidates": [
{
"rank": 1,
"player_id": 7234,
"player_name": "Toni Kroos",
"position": "CM",
"similarity_score": 0.943,
"stats": {"avg_xg": 0.038, "pass_completion_pct": 91.2, "avg_xt_per_pass": 0.0044, "total_shots": 39},
"delta": {"avg_xg": -0.004, "pass_completion_pct": 2.9, "avg_xt_per_pass": 0.0003}
}
],
"recommendation": "Replacing Luka Modrić: best match is Toni Kroos (similarity 0.94). xG/shot 0.038 vs 0.042 (lower), pass completion 91.2% vs 88.3% (higher)."
}Error responses: 400 (invalid player_id or top_n), 401, 404 (player not found), 429
| Capability | V1 | V2 | V3 |
|---|---|---|---|
| Data providers | StatsBomb only | StatsBomb + Opta | + Wyscout v3 |
| Ingestion language | Python | Dart microservice, HTTP API | Wyscout adapter (file + URL+Bearer) |
| Coordinate system | StatsBomb 120×80 (raw) | Universal 105×68 m — single source of truth | wyscout_100x100 branch in coord_normalise macro |
| Data layer | Single staging schema | Bronze / Silver / Gold / Marts medallion | + silver_pass_links |
| Coordinate tests | None | dbt spatial range tests on every Silver run | wyscout added to accepted_values |
| xG model | Post-hoc rescaled | CalibratedClassifierCV (Platt scaling) | xG feeds player similarity features |
| BI output | Superset (local only) | 5 static PNGs via CI — shareable artifacts | + PPDA mart + pass network mart |
| CI coverage | Lint + unit tests | Full end-to-end pipeline on 524,457 events | 131 tests (115 Python · 66 Dart) |
| Data volume | Single match | 147 WC 2022 matches, bulk incremental loader | same |
| Recruitment | None | finishing_quality metric in mart_player_metrics |
Cosine similarity model — top-10 similar players per player |
| Pressing analytics | None | None | PPDA per team per match with High/Medium/Low label |
| Pass network | None | None | Player-to-player pass pairs with completion rate |
| Decision layer | None | None | /summary: xg_diff + PPDA + key threats · /replacement: stat deltas + recommendation string |
| Model | Algorithm | Input | Result |
|---|---|---|---|
| xT Surface | Value iteration (15 passes) | Silver events · 16×12 grid | 192 cells, max xT = 0.298 |
| xG Classifier | XGBoost + Platt scaling | Silver shots — distance, angle, pressure | Calibrated goal probability |
| xP Classifier | XGBoost | Silver passes — start/end coords, distance, pressure | AUC 0.897 · log-loss 0.293 · 118,187 training passes |
| Set-piece Clustering | K-Means k=6 | Corner + shot locations (105×68 m) | 12 cluster centroids (6 corner zones, 6 shot zones) |
| Press Trigger Detector | Rule-based sequence | Ball recovery + 3 defensive actions / 5 s | 165 press triggers detected (WC 2022) |
| Player Similarity | scikit-learn NearestNeighbors (cosine) | 12 aggregated features per player — shots, goals, xG, passes, completion rate, location, pressure | Top-10 most similar players per player; idempotent write-back to player_similarity_scores |
xP engineering note: all three ML models read from the Silver layer and write xp_value / xg_value / xt_value back to fact_events before dbt Gold and Marts materialise — ensuring mart_player_metrics.avg_xp is populated on every run.
xG calibration: CalibratedClassifierCV(XGBClassifier(), cv=5, method='sigmoid') — Platt scaling ensures xG=0.30 represents a genuine ~30% conversion rate, not merely a ranking score. A Brier score and expected-vs-actual goal check gate every training run.
Coordinate normalisation: a single dbt macro normalise_x(col, coord_system) converts any provider's raw coordinates to the 105×68 m universal pitch. Adding a third provider requires one new adapter file and one macro branch — zero changes to downstream models.
Pass quality scouting: xP isolates pass difficulty from completion rate. A midfielder completing high-difficulty passes (low xP) in high-threat zones (high xT) appears on no traditional completion-rate leaderboard — but surfaces immediately in mart_player_metrics.
Opponent set-piece analysis: K-Means clustering of 1,384 corners and 4,904 shots across 147 WC matches reveals six repeatable delivery zones per event type. Coaching staff receive cluster centroids and member counts without manual video tagging.
Video integration: the pipeline generates a SportsCode/Hudl-compatible XML file containing the 25 highest-xT events per match. Analysts open the file directly in Hudl Sportscode — no manual timestamp entry.
Striker recruitment: finishing_quality = goals − total_xG separates clinical finishers from shot-volume players. Available in mart_player_metrics; directly queryable in Superset without custom SQL.
Hidden gem scouting: the player similarity model indexes every player on 12 aggregated features (shot profile, pass profile, location, pressure resistance). A scout running SELECT * FROM player_similarity_scores WHERE player_id = :target ORDER BY rank receives the 10 most statistically similar players — surfacing affordable alternatives to a transfer target across any provider's dataset.
Pressing intensity analysis: mart_pressing_metrics computes PPDA per team per match using all three providers' data. A PPDA of 6.2 (High) means the team forced an error or blocked every 6.2 opponent passes — directly comparable across StatsBomb, Opta, and Wyscout feeds without any manual normalisation.
Post-match decision intelligence: GET /api/v1/matches/{id}/summary delivers three actionable signals in a single API call: xg_diff (was the scoreline fair?), pressing_intensity per team (PPDA-derived High/Medium/Low), and key_threats (top-3 xT contributors). Analysts get a structured JSON verdict instead of raw event rows — the pipeline answers the question, not just provides the data.
Transfer target replacement: GET /api/v1/players/{id}/replacement queries pre-computed cosine similarity scores and returns top-N candidates with per-stat deltas (xG, pass completion, xT/pass) and a plain-English recommendation string. A scout can surface affordable alternatives to a £60M target with a single HTTP request.
| Metric | Value |
|---|---|
| Matches ingested | 147 |
| Total events | 524,457 |
| xP training passes | 118,187 |
| xP model AUC | 0.897 |
| Set-piece cluster centroids | 12 |
| Press triggers detected | 165 |
| xT grid cells | 192 (16×12, 105×68 m) |
| CI pipeline duration | ~11 min (full end-to-end) |
| BI chart artifacts | 5 PNGs · 90-day retention |
Every push to main runs the full pipeline against a live PostgreSQL instance:
checkout
→ install Python 3.11 deps
→ init schema (01_schema.sql + 02_v2_migration.sql)
→ bulk ingest 147 matches (ingest_season=true)
→ build + start Dart ingestion service
→ ingest match events via Dart HTTP API
→ dbt bronze (run + test)
→ dbt silver (run + test — spatial range gate)
→ xG model (XGBoost · calibrated)
→ xT model (value iteration)
→ xP model (XGBoost · writes xp_value to fact_events)
→ player similarity model (cosine NearestNeighbors · writes player_similarity_scores)
→ dbt gold (run + test)
→ dbt marts (run + test — avg_xp now populated)
→ refresh materialized views
→ export SportsCode XML
→ tactical models (K-Means · press trigger)
→ generate PDF match report
→ generate analytics visualisations (5 PNG charts)
→ upload artifacts (XML · PDF · 5 PNGs · dbt logs)
→ pipeline summary (GitHub Step Summary)
The Silver spatial range tests (location_x ∈ [0, 105], location_y ∈ [0, 68]) act as a hard gate — if any event falls outside the universal pitch boundary after coordinate normalisation, the pipeline fails before ML models train.
xforge/
├── .github/
│ └── workflows/
│ └── pipeline_v2.yml # Full end-to-end CI pipeline
├── config/
│ └── superset_config.py # Superset secret key, DB URI, feature flags
├── dart_ingestion/
│ ├── Dockerfile # Multi-stage AOT compile — ~10 MB image
│ ├── pubspec.yaml
│ └── lib/
│ ├── main.dart # Shelf HTTP server — POST /ingest, GET /health
│ │ # GET /api/v1/matches/{id}/xg
│ │ # GET /api/v1/matches/{id}/summary ← xg_diff · PPDA · key threats
│ │ # GET /api/v1/players/{id}/similar
│ │ # GET /api/v1/players/{id}/replacement ← stat deltas · recommendation
│ ├── middleware/
│ │ ├── bearer_auth.dart # Constant-time Bearer token guard
│ │ └── rate_limit.dart # Sliding-window 60 req/min per IP
│ ├── models/unified_event.dart
│ ├── adapters/
│ │ ├── adapter_interface.dart
│ │ ├── statsbomb_adapter.dart # 120×80 → UnifiedEvent
│ │ ├── opta_adapter.dart # 100×100 → UnifiedEvent + SSRF guard
│ │ └── wyscout_adapter.dart # 100×100 → UnifiedEvent (Wyscout v3) + SSRF guard
│ └── db/postgres_writer.dart # Bulk INSERT · queryXgValues · querySimilarPlayers
│ # queryMatchSummary · queryReplacementCandidates
├── dbt_project/
│ ├── macros/
│ │ └── coord_normalise.sql # normalise_x / normalise_y — single normalisation point
│ └── models/
│ ├── bronze/ # Type-cast pass-through + provider tests
│ ├── silver/ # 105×68 m normalisation + spatial range tests
│ │ # silver_pass_links — recipient extraction for pass network
│ ├── gold/ # Player / team aggregations
│ └── marts/ # mart_player_metrics · mart_team_summary
│ # mart_match_summary · mart_competition_leaderboard
│ # mart_pressing_metrics (PPDA) · mart_pass_network
├── docs/
│ └── screenshots/ # CI-generated PNGs committed from latest artifact
├── scripts/
│ ├── init/
│ │ ├── 01_schema.sql # Tables, partitions, indexes
│ │ └── 02_v2_migration.sql # V2 columns, materialized views
│ ├── massive_ingestion.py # Incremental bulk StatsBomb loader
│ ├── xg_model.py # XGBoost xG + Platt calibration
│ ├── xt_model.py # Value-iteration xT surface
│ ├── predictive_models.py # XGBoost xP + chunked prediction write-back
│ ├── tactical_models.py # K-Means clustering + press trigger detection
│ ├── player_similarity.py # Cosine NearestNeighbors recruitment model
│ ├── visualise.py # 5 PNG BI charts — mplsoccer + matplotlib
│ ├── report_generator.py # 5-page PDF per match
│ ├── xml_generator.py # SportsCode/Hudl XML — top-25 xT events
│ ├── refresh_materialized_views.py
│ └── setup_superset.py # Autonomous Superset bootstrap — 7 charts + dashboard
├── docker-compose.yml # Local: Postgres + Superset + pgAdmin
├── requirements.txt
└── LICENSE
Prerequisites: Docker >= 24, Docker Compose v2, 4 GB RAM minimum.
git clone https://github.com/bbasaranemir/xforge.git
cd xforge
docker compose up -d postgres
psql -h localhost -U analytics -d football_db -f scripts/init/01_schema.sql
psql -h localhost -U analytics -d football_db -f scripts/init/02_v2_migration.sqlRun the Dart ingestion service:
docker compose up -d dart_ingestion
curl http://localhost:8090/health
# → {"status":"ok"}
curl -X POST http://localhost:8090/ingest \
-H "Content-Type: application/json" \
-d '{"provider":"statsbomb","match_id":3869685,"competition_id":43}'
# → {"written":3401}
# After running ML models (player_similarity.py, xt_model.py):
curl http://localhost:8090/api/v1/matches/3869685/summary
# → {"xg_diff":2.38,"pressing_intensity":{"home":{"label":"High","ppda":6.2},...},"key_threats":[...]}
curl "http://localhost:8090/api/v1/players/5503/replacement?top_n=3"
# → {"target":{...},"candidates":[...],"recommendation":"Replacing Luka Modrić: best match is ..."}Run the dbt medallion:
cd dbt_project
dbt deps
dbt run --select bronze silver gold marts
dbt test --select silver # spatial range gateLaunch Superset with pre-built dashboards:
docker compose up -d superset
python scripts/setup_superset.py
# → 7 charts and Matchday Analytics dashboard bootstrapped at http://localhost:8088Navigate to Actions → xForge V2 Pipeline → Run workflow. Set ingest_season: true to load all 147 WC 2022 matches before running models. The full pipeline completes in approximately 11 minutes; five PNG charts are uploaded as a 90-day artifact under analytics-visualisations-{run_id}.
To update the screenshots in this README after a successful run:
# Download analytics-visualisations-{run_id}.zip from the Actions artifact panel,
# extract to docs/screenshots/, then:
git add docs/screenshots/*.png
git commit -m "docs: update BI visualisation screenshots from CI run {run_id}"
git pushdim_competitions ─┐
dim_seasons ├──► fact_events (LIST-partitioned by competition_id)
dim_matches │ │
dim_players │ ├── xt_value (value-iteration xT)
dim_teams ────────┘ ├── xp_value (XGBoost pass completion probability)
└── xg_value (XGBoost calibrated goal probability)
│
┌────────────┼────────────────────┐
▼ ▼ ▼
xt_surface set_piece_clusters model_registry
(192 cells) (12 centroids) (AUC, log-loss)
mv_team_xg (REFRESH CONCURRENTLY)
mv_shot_locations
player_similarity_scores (cosine top-10 per player)
analytics_analytics_marts.*
├── mart_player_metrics (avg_xg · avg_xp · finishing_quality)
├── mart_team_summary (xG · shot counts per team per match)
├── mart_match_summary (per-match aggregates)
├── mart_pressing_metrics (PPDA · High/Medium/Low per team per match)
├── mart_pass_network (player-to-player pass pairs)
└── mart_competition_leaderboard (xT per match rank)
Partitioning: fact_events uses PostgreSQL LIST partitioning on competition_id. The CI environment adds --shm-size 256m to the Postgres service container to support large cross-partition joins during mart materialisation.
Materialised views: REFRESH MATERIALIZED VIEW CONCURRENTLY is used throughout — BI tools and Superset see no read-lock downtime during refresh cycles.
StatsBomb Open Data — used under the StatsBomb Open Data Licence. This project is not affiliated with or endorsed by StatsBomb.
MIT