Skip to content

feat: consolidated final deployment tree — ML picks Lambda + supabase migration + cleanup - #146

Merged
nathan-probert merged 169 commits into
mainfrom
feat/final-clean
Nov 4, 2026
Merged

nathan-probert merged 169 commits into
mainfrom
feat/final-clean

Conversation

@nathan-probert

Copy link
Copy Markdown
Owner

Consolidates the 9 swarm PRs into ONE reviewable diff off main. Merge this, then close the 9 donor PRs (do not merge them).

Donor PRs (superseded by this PR): #137 #138 #139 #140 #141 #142 #143 #144 #145

Lineage per donor:

Final-state notes:

  1. chore/clean-tree's deletion of lineup_project.py was taken back — the retained shot-cols model.py (4cba2b7, per the merge resolution rules) imports it; it is a core V9 feature module, not an exp_* cache file.
  2. verify_shot_columns.py (exp-lineage one-off importing the deleted exp_exposure.py) was dropped instead of resurrecting exp_exposure.py; spot_check_shot_cols.py + test_shot_columns.py carry the verification.
  3. No runbook/notes/number regressions: MODEL_NOTES.md tokens 29.92/29.41/29.82/28.05/26.50/97.85/McNemar verified.

Verification: 55/55 unit tests pass (5 shot-columns + 19 predict-picks + 31 local-store/deleted-env); grep for exp_/notes_v16 refs in *.py is empty; template.yaml has MLPicksFunction + EventBridge trigger; Makefile deploy-ml targets present. docs/tree_walkthrough.md untouched. #128 predates this swarm and is left open for the owner.

Owner next steps after merge: per #143's doc, drop the three backtrack-dev tables; per #142 runbook, deploy the MLPicks Lambda.

Player-Snapshots-{ENV} stores one row per player per date, captured live as the
pipeline runs. That capture has two properties that limit it:

* Sparse. Rows exist only for pick-relevant dates, not every game a player
  played, so it cannot answer "what were this player's stats entering game X"
  for an arbitrary X.
* Lossy on early rows. Rows captured in the first weeks of a season store rates
  at 2 decimal places (0.31) where later rows carry 6 (0.320755). The rounding
  is baked in at capture time.

This adds a pipeline that rebuilds those stats from the NHL's per-game box
scores, walking each season forward and accumulating.

Player-Snapshots-backtrack-{ENV} mirrors the live archive's 30 columns exactly,
so it is a drop-in replacement. Today date/player_id/name/team_name/home/gpg are
populated; the rest are declared and documented as not-yet-reconstructed, with
the reason per column (3-year windows for hgpg/hppg, no definition found for
tgpg/otga/otshga, and fields absent from the game-log feed for the goalie,
lineup and injury columns).

Player discovery comes from the box scores, not from the existing table. The
point is to replace the archive rather than supplement it, so the player set
must not be defined by who the old table happened to record - a player picked
for the first time appears because he played a game.

The raw per-game layer is data/dumps/*.sql.gz (1.8 MB per season), a gzipped SQL
text dump. Text so it diffs line by line in review; gzipped because the 32 MB
SQLite working copy compresses to 1.8 MB and would otherwise duplicate every
row. `make backtrack-load` rebuilds the database from it, so a fresh clone can
query the data without re-crawling.

Verified against Player-Snapshots-prod on 2023-24: 46,297 rows, 880 players,
and gpg agreeing on 15,640 of 15,667 comparable rows (99.83%), where the
remainder are archive rows rounded to 2dp and the reconstructed values are the
more precise of the two.

Refs the #113 archive work.
Second feature on top of gpg, computed as a rolling window over the raw
per-game rows rather than by re-crawling - `local_store.py --derive
20232024` then `--publish`, about 15 seconds end to end.

five_gpg is goals over the player's last 5 games. The window resets per season,
which falls out of the existing window partition (season, player_id), so the
denominator ramps 1, 2, 3, 4 and then pins at 5. A player's debut game is NULL
rather than 0, keeping "has not played yet" distinct from "played and scored
none" the same way gpg does.

THE DIVISOR IS DELIBERATELY DIFFERENT FROM THE ARCHIVE

smartscore_info_client's get_five_gpg divides by a hardcoded 5 regardless of how
many games the player has played. Sidney Crosby's first two games of 2023-24 are
2 goals in 2 games, which is a hot start; that function reports 0.2 and 0.2. The
archive's own values confirm the shape of the problem - 11 distinct values, all
multiples of 0.2, which is what a constant divisor produces, and it collapses a
rookie scoring on debut into the same value as a cold player with no goals in five.

Here the divisor is the number of games in the window, so that debut reads 1.0.
716 rows (1.39%) change, all early-season; distinct values go from 11 to 21 and
the maximum from 2.0 to 4.0.

Consequence, measured: agreement with Player-Snapshots-prod on five_gpg moves from
99.70% to 99.45%, with the 87 mismatches spread across 27 dates in
2024-02-17..2024-04-18. That is expected. This table exists to replace the
archive, and reproducing a division bug is not fidelity.

Verified against prod before and after: 47,221 published rows with goalies
excluded, gpg unchanged at 15,640/15,707. The dump round-trip was re-checked
with the new column and comes back identical, so `make backtrack-load` still
rebuilds the store exactly.
The archive carries three team-level features that are not player stats: every
player on a club holds the same value for a given date. From
smartscore_info_client's models/team.py:

    tgpg  - the team's own goals FOR per game, season to date
    otga  - the OPPONENT's goals AGAINST per game, i.e. how many goals this team
            scores on that specific opponent
    otshga- the opponent's shorthanded goals against per game

They live in Team-Stats-backtrack-{ENV} keyed on (season, team_abbrev, game_id)
rather than being denormalised onto player rows, where the archive repeats one
number across ~21 rows per team-date.

TEAM GOALS MUST COME FROM team_goals_for, NOT FROM SUMMING PLAYERS

Two goal types count toward a team's total but appear in no skater's box score
row, so summing player goals undercounts:

  * shootout game-winning goals - on the scoreboard and in the team total, but
    the play-by-play carries only a shootout-complete marker and the box score has
    no shootout play at all. Confirmed: PIT at MTL on 2023-12-13 (2023020442),
    official 4, skater sum 3, and PIT won the shootout.
  * goalie empty-net goals - officially credited to the goalie, omitted from his
    `goals` field. PIT at TBL on 2023-11-30 (2023020345): official 4, skater sum 3.

The archive's own totals include both, per the NHL glossary note that standings
GF/GA count shootout winners while the stats pages do not. The fix is the box
score's own awayTeam.score / homeTeam.score, which we were already fetching and
discarding. PIT 2023-12-13's official scores are 4 and 3, so the field was
complete all along; summing player rows was simply the wrong source.

otga additionally needed restructuring. It is the opponent's conceded rate, so
each team first carries its own goals-against (the sum of what its opponents
scored), and otga is then read off the *other* team's row for the same game. The
previous version accumulated the opponent's goals scored over our own games, which
is our scoring rate against them - related, but not the same number.

A NULL, not a 0, on a team's season debut: a zero claims a team averages no goals
per game, which the archive emits (Winnipeg, Anaheim, Dallas and Carolina all
show tgpg = 0 early in the current season).

A team's game with no opponent is refused rather than written with a silently
wrong otga.

VALIDATION

validate.py joins in Python on team_abbrev, never team_name. Three SQL joins were
tried first and all were wrong for the same reason: the archive's team_name is a
PLACE name, and NYI and NYR both render as "New York", so any join through it
matches whichever club's row the planner returns first. That surfaced as ~560
"wrong" team-stat rows which were really Rangers/Islanders values swapped. It cost
real time to diagnose and the module docstring records it.

  gpg      15,656 / 15,707 comparable  99.68%
  five_gpg 15,620 / 15,707            99.45%

15,707 archive rows compared with none missing a local match. The gpg residue is
2024-02-24, the stale-capture date, where our values are the more precise of the
two. The five_gpg residue is the intended divided-by-5 divergence.

Team features are counted but not compared: the archive has no team key at all
outside a six-day window, so there is nothing unambiguous to join against. That
needs 2026-27 crawled, where team_name is actually populated.

otshga remains NULL and documented - it needs
api.nhle.com/stats/rest/en/team/penaltykilltime, and cannot be derived from
per-player goals.
DEBUT ROWS ARE 0, NOT NULL

Empirically checked against Player-Snapshots-prod: 40 archive rows sit on
players' first-of-season dates and every one stores gpg 0 - including a player
who scored in his debut that night (8481462, 2024-02-19). The row is entering
the game, so 0 is the archive's pre-debut convention, not a missing value.
reconstruct_player normalises _rate's None and compute_derived's window SQL
gates on prev_gp = 0, so neither table ever carries a null gpg.

SCOPE: GAME DATES ONLY, REGULAR SEASON ONLY

Coverage diff against the local store (52,468 player_games rows): all 17,105
regular-season archive rows land on game dates - the archive has no rows on
non-game dates at all, so --all-dates was scanning 10x the dates to reproduce
nothing and is removed. The one real gap is playoff dates: 1,398 archive rows
(8.2%) fall after 2024-04-18 while this rebuild is regular-season scope. That
is documented in the module docstring and the player migration rather than
papered over.

DEAD CODE AND STALE DOCS

get_players, get_archive_dates, _opponent_for, and validate.py's compare() are
removed - unreachable after earlier refactors. main()'s target discovery
collapses into one if/elif/else: the old structure could fall through an
unmatched branch with appearances unbound, and a --player dry run did crash
that way; the fixed path has a regression test. team_map now says NYI/NYR share
"New York" (both New York clubs) and covers ARI through 2023-24 alongside UTA
from 2024-25. The team migration described otga/otshga as the team's own goals
against; they are the OPPONENT's - the old reading matched 107 of 16,301 rows.
The player migration's "every date is present" claim is replaced with the
measured coverage above.

Makefile: backtrack-publish is now derive + publish + publish-team (it used to
publish a table compute_derived may never have touched), with new
backtrack-derive and backtrack-validate targets.

TESTS

21 tests under tests/unit/test_backtrack_*: team_map place names, aliases and
the NYI/NYR collision; reconstruct_player's strictly-before gpg, debut-zero,
box-score context override and name fallback, plus a main() smoke test;
_literal's SQL rendering including the new None -> NULL; compute_derived's
window SQL (debut 0, strictly-before counts, idempotent replacement);
compute_derived_team's otga semantics and the no-opponent refusal. pyproject's
pytest pythonpath gains smartscore/scripts/backtrack.

Also adds .gitattributes so data/dumps/*.sql.gz diffs opt into gunzip textconv,
and runs ruff-format over the 7 files failing CI's lint job.

Verified: ruff check and format clean, 249/250 unit tests pass (the failure is
test_utility_schedule_run expecting -{ENV} -dev against a local ENV=prod .env),
and reconstruct.py --player 8476967 --season 20232024 --dry-run agrees with
prod on 29 of 30 compared rows. The one difference is the archive's
stale-capture 2024-02-24 row, which missed the 02-23 game (2/53 vs the correct
2/54); both sides agree again on 02-26 and 02-28.
The live archive was missing two columns the product depends on for 2023/24:
tims (challenge pick groups) and lineup/PP units. Live scrapes only answer
for today, so both are rebuilt from history instead:

- tims_client.py: api.hockeychallengehelper.com/api/history?datetime= serves
  the same playerLists shape back to 2023-10-10; cached per date, 404s (the
  offseason) cached as empty. tims_map joins player->group per game date.
- lineup_reconstruct.py: shiftcharts from the legacy stats API are swept per
  period into constant on-ice sets, strength-classified, and clustered into
  F1-F4 / D1-D3 / PP1-PP2 by shared TOI. Goalie shifts are filtered first
  (they span the whole time in net and would glue lines together and break
  the 5v4 read), and merges are size-capped at 3F/2D/5PP - without the cap,
  greedy sum-linkage collapses one mega-cluster once its cross-edges
  out-sum any real line pair.
- local_store: player_lineups table + --lineups CLI; publish now ships
  tims (0 = unlisted), lineup_unit, lineup_position_group, pp_unit, and
  lineup_status = ACTUAL (the live product stored PROJECTED).
- run_background: lineups and publish commands so multi-minute runs stay
  detached.

Published to Player-Snapshots-backtrack-dev: 50,389 rows, 22,182 tims
entries across 232 dates. Spot-checked McDavid 2023-10-14 (tims=1, F2, PP1)
and Oilers playoff pairs. 271 tests pass, ruff clean.
The archive missed opp_goalie_* for 2023/24 the same way it missed tims:
the live pipeline only scrapes today's slate. Rotowire's
projected-goalies.php takes a date parameter that serves history (verified
back to 2023-10-10 and into the 2024 playoffs), answering the same game
shape the live get_starting_goalies parses - so the backtrack reads the
table the product already trusts rather than inventing a new source.

- goalie_client.py: fetch + per-date cache (goalies-{date}.json), team
  abbreviations normalized MON->MTL / LAS->VGK like the live pipeline.
  Offseason dates are empty lists; a non-list payload raises instead of
  caching a shape change that would poison every later publish.
- publish joins each skater's opponent to that date's starter and ships
  opp_goalie_name/team/status/confirmed with the live denormalization
  semantics: upper-cased status, separate confirmed boolean, and
  UNKNOWN/False when the table lists no starter for the team.
- The stat columns (gaa, save_pct, record, shutouts, games_played) stay
  NULL: live filled them from same-day club-stats, which is
  current-season-only, so season-end totals standing in for an October
  game's as-of-date view would be a lie the archive itself never told.

Published to Player-Snapshots-backtrack-dev: 2,800 team entries across
232 dates, all 50,389 rows now carry the opponent starter. Spot-checked
DeSmith/VAN on 2023-10-14 and Talbot/LAK on 2024-04-22. 275 tests pass.
scored was the last unwritten column the 20261008 migration called out.
It is derivable from the game itself - did this skater score in it - so
the publish reads the box score's goals column: ground truth, and it
covers every published row rather than only the challenge's listed
skaters. The helper API's own graded flag agrees with it on every listed
skater checked (159/159 on 2023-10-14), so no second source is needed.
The column was already declared by the migration; no schema change.

Adding pg.goals to the SELECT, though, broke the query plan: the column
made the covering index unusable, SQLite flipped the join order, and the
DISTINCT skaters subquery - an ephemeral co-routine - got re-scanned for
every outer row. Half a million row-comparisons per second of nothing;
the publish hung indefinitely on a query that used to take 0.4s.

The fix removes the subquery: drive the join from player_games with the
goalie filter inline (pg.position <> 'G'), which is the same row set
(50,389) in 0.74s and one fewer CTE for the planner to mis-order.

Published to Player-Snapshots-backtrack-dev: 7,579 scored=1 / 42,810
scored=0 / 0 null across the season - the scorer-instance count a
5.4 goals/game pace predicts. Spot-checked McDavid and Draisaitl, both
goals, on 2023-10-14.
get_hgpg(years=3)/get_hppg(years=3) sum goals (power play goals) and games
over the season and the two before it, then divide once. compute_historic
mirrors that: prior two seasons complete, current season strictly before the
row's own game - the frame gpg was validated on. Both priors must exist in
the store or the columns stay NULL rather than divide by a partial window,
so 20212022/20222023 wait on 20192020/20202021 being crawled.

Re-derived all four seasons; 20232024/20242025 re-published to dev.
Validated against Player-Snapshots-dev: hgpg 99.83%, hppg 99.98% at 2dp.
The dumps predate compute_historic, so a restored store would need a
--derive pass before the 3-year windows were queryable. 20212022/20222023
carry the columns with NULL values pending the 20192020/20202021 crawls.
The archive's opp_goalie_gaa/save_pct/record/shutouts/games_played were NULL for history: live read them from club-stats/{team}/now, which has no date parameter and only answers for the in-progress season. Rebuild them from the game logs --build already caches instead: compute_goalie_features walks each goalie's appearances strictly before the row's own game, partitioned on (player_id, team_abbrev) like club-stats' own team-scoped totals, NULL on a club debut where live left the keys unset, and publish matches the goalie by NHL id to each opponent row.

save_pct sums official per-game saves (round(savePctg * shots_against)) rather than shots - goals: club-stats' season saves exceed sa - ga by a save or two and its savePercentage is saves/shots, so a new saves column stores them and --gamelogs re-merges the log layer alone after schema changes. A post-debut 0-shot window reads 0.0, the value club-stats pairs with shotsAgainst 0.

validate.py --goalies compares season-end totals against club-stats for all four seasons - regular season only, GAA with a sub-second toi tolerance (their published GAA uses unrounded ice time), SV% rounded half-up like the API: 450/450 goalie-team totals match. Re-derived and re-published all four seasons to dev; regenerated the dumps with the saves column.
…eup columns

The WHAT IS AND IS NOT RECONSTRUCTED section predates compute_goalie_features: it claimed opp_goalie_*, lineup_* and pp_unit stay null against the live archive, but local_store.publish now writes all three to the same backtrack table (cached game logs and goalie feeds for the stats, shift charts for the lineup block). Only injury_* still has no reconstructed source, so the null-means-not-reconstructed guarantee now names just that family.
…od; dispatch override default prod; step-level ENV on backfill+inference; no silent default into wrong tables)
@nathan-probert

Copy link
Copy Markdown
Owner Author

ENV is now explicit in ml-picks.yml (223e253):

  • Matrix: scheduled = prod (job-level env: ENV: prod only — scheduled runs never read the input path); per-game kicks (repository_dispatch) = prod likewise; manual workflow_dispatch resolves the new environment string input — defaults to prod; setting dev opts in to writing Picks-dev / reading the -dev-suffixed tables → so dry=false answers today's question: scheduled = writes Picks-prod; manual dispatch with environment=dev -> writes to Picks-dev (the resolved value is echoed as env: prod/env: dev in the job log and validated dev|prod fail-closed).
  • Fail-safe (both modes): the Picks write goes through the schema-safe writer save_picks_db (schema probe-before-delete — unsupported payload keys abort BEFORE any wipe — plus restore-on-insert-failure, so a failed run leaves the table as its OLD state, never empty). Mechanism: the resolved ENV is re-bound as step-level env on both script steps, set by the runner in the process env before python starts, so config.ENV's import-time read (config.py:12) is unambiguous.
  • Tests: 3 new parametrized tests in tests/unit/test_predict_picks.py (unset→dev default; dev/prod preset before import → every save_picks_db surface targets Picks-{env}); battery green: 136 passed across the six unit files, pre-commit clean, runbook docs/ml-picks-runbook.md documents the matrix.

…r-Snapshots-backtrack-prod regardless of picks env; dev-shell false routing removed

- backfill_missing_days: player_table/marker_table resolve the SHARED ledger
  (Player-Snapshots-backtrack-prod / Player-Game-Raw-Markers-prod)
  independent of --env (verified: the helper never reads the LIVE table, so
  --env is now vestigial for the table resolution and kept for CLI/
  workflow compatibility only). The ONE remaining fail-loud gate is the
  shared ledger being truly empty - new bootstrap message
  ('run the publish/bootstrap chain once (make backtrack-publish
  ENV=prod) - the helper never bootstraps'), never the deleted
  -dev-shell routing / publish-era 'run the publish/backfill chain first'
  wall that hit the owner's dev-mode run.
- run_daily_picks: history_rows/history_team_rows/history_goalie_rows and
  the carry-forward fallback read the -prod backtrack ledgers regardless of
  picks env (dev backtrack tables deleted, docs/backtrack_dev_deleted.md);
  strict date < before_date and missing-day semantics unchanged; only
  today_live_rows (Player-Snapshots-{env}) stays env-resolved.
- docs/runbook: data-flow section + the §51/53 env tables corrected; new
  section 10 'ONE SHARED HISTORY LEDGER' with the owner note that dev-mode
  runs are self-healing against the shared ledger (viable kick again).
- tests: dev-run-backfill-targets-shared-ledger (no RuntimeError, no
  -dev resolution), prod unchanged (existing suite), history_rows reads
  the shared ledger on a dev run (incl. full wiring: teams/goalies/live
  split), fail-loud bootstrap gate kept + re-pointed to the new message.
- verified: file batteries green, full unit suite 530 passed / 3 failed
  (all 3 pre-existing at 223e253: test_rust::test_predict + 2
  test_utility_schedule_run), ruff check + format clean.
@nathan-probert

Copy link
Copy Markdown
Owner Author

Root cause of the owner's dev-mode traceback ("Player-Snapshots-backtrack-dev is empty - run the publish/backfill chain first"): the backfill helper still resolved its table from the --env suffix (the publish-era fail-loud guard, wrong for this helper since the dev backtrack tables were deleted).
Shared-ledger rule (now enforced in code): there is exactly ONE history ledger — Player-Snapshots-backtrack-prod — and BOTH the backfill target and every history_rows/team/goalie read go to it regardless of picks env; only the LIVE Player-Snapshots-{env} and Picks-{env} tables stay env-resolved (docs/ml-picks-runbook.md §10, ref docs/backtrack_dev_deleted.md).
What changed (027ef47): backfill_missing_days.player_table/marker_table always return the shared ledger (marker shelf aligned too); the only remaining fail-loud gate is a truly empty shared ledger with a clear bootstrap message ("run the publish/bootstrap chain once (make backtrack-publish ENV=prod) - the helper never bootstraps"); dev-mode runs are now SELF-HEALING against the shared ledger, so a dev kick is viable again; unit tests added for the dev-run routing + the re-pointed bootstrap gate, existing prod-path tests unchanged and green.

…creates them; DROP SQL + docs for the owner

The three dev backtracking tables were row-deleted 2026-10-24 and their
empty shells were verified 0 rows again on 2026-11-03 (prod verified
superseding on the same read: Player 624,253 planned / exact rows through
2026-10-29; Team 34,636; Goalie 37,044). PostgREST cannot DDL, so the
final DROP stays owner-side - but the standing migrate job could have
revived the -dev shells on the next main-push run because every backtrack
migration scoped table names through __ENV__ (ENV=dev on PR deploys).

* supabase/migrations: all six backtrack migration files hard-pin the
  table names to the '-prod' literal suffix, each headed with the
  BACKTRACK-PROD-PINNED marker + deletion rationale (dev backtracking
  RETIRED Oct 2026). No __ENV__ remains in any migration (placeholder
  scan clean for ENV=prod AND ENV=dev substitution; files parse as
  Postgres SQL under both substitutions via sqlglot).
* deploy.yml 'Guard migration files': files carrying the marker may
  hardcode '-prod'; any '-dev' literal, any backtrack table scoped
  through __ENV__, and any non-marker '-prod' literal fail the run.
* deleted_env_tables guard: docstring/message updated (shell dropped
  Oct 2026; name-based, fires whether or not the table exists).
  fidelity_tracker's dev daily-build path now resolves through the
  guard (--env dev / both fails loud with the prod pointer).
* tests: guard tests extended (name-based semantics, migration scans,
  prod-pin marker coverage); new fail-loud publish test
  (ENV=dev publish_team raises before any Supabase call); skip-path
  test re-pointed to prod.
* docs: ml-picks-runbook.md section 11 'Deleting the -dev backtrack
  table shells' - owner DROP SQL (verbatim, PostgREST cannot DDL),
  VACUUM/size notes, backup/restore paths (r-deldev worktree
  data/backups/MANIFEST.md); backtrack_dev_deleted.md + the shared
  ledger section cite the deletion date and point at section 11.
  The DROPs are intentionally NOT in supabase/migrations/ - the migrate
  job re-applies every file there every run.
@nathan-probert

Copy link
Copy Markdown
Owner Author

-dev backtracking shells: DROP is owner-side, plus recreation-proofed migrations (commit 30c245d)

Recap of the state established earlier, re-verified live on 2026-11-03:

  • All three *-backtrack-dev shells are 0 rows (exact PostgREST count, checked twice the same session — no surreptitious writes since 11-01).
  • prod supersedes (source of truth): Player-Snapshots-backtrack-prod planned estimate 624,253 with exact windowed counts (2026-10-27..10-29 = 936 rows, coverage through 2026-10-29; Oct 01–26 = 6,048), Team-Stats-backtrack-prod exact 34,636, Goalie-Snapshots-backtrack-prod exact 37,044; Player max(date) is already 2026-11-02 (nightly freshness green).

(1) The DELETEs themselves are owner-dashboard-side — PostgREST cannot DDL (proven earlier; service-role key is API-write only). Verbatim DROP SQL (0-row shells verified; safe to run anytime after this PR merges):

DROP TABLE IF EXISTS "Player-Snapshots-backtrack-dev";
DROP TABLE IF EXISTS "Team-Stats-backtrack-dev";
DROP TABLE IF EXISTS "Goalie-Snapshots-backtrack-dev";

Then (optional, off-peak) VACUUM FULL on the three -prod history tables to settle the dashboard project size. Full runbook: docs/ml-picks-runbook.md§11 "Deleting the -dev backtrack table shells" + docs/backtrack_dev_deleted.md. Backup/restore for the pre-delete row content lives in the r-deldev worktree (data/backups/*.csv.gz + SHA256s in data/backups/MANIFEST.md). Do NOT add the DROPs to supabase/migrations/ — the migrate job re-applies every file there every run and on a main push it runs with ENV=prod.

(2) The migration edits prevent recreation: all six backtrack migration files now hard-pin table names to the -prod literal suffix, each headed with the BACKTRACK-PROD-PINNED marker + deletion rationale; no __ENV__ remains in any migration. deploy.yml's "Guard migration files" step now (a) lets only marker-carrying files hardcode -prod, (b) fails any -dev literal, and (c) fails any backtrack table scoped through __ENV__ — the exact trap (a PR deploy with ENV=dev would have re-CREATE'd the retired shells) is closed. Placeholder substitution verified clean for both ENV=prod and ENV=dev; files parse as Postgres SQL under both.

(3) The deleted-env guard stays fail-loud and existence-agnostic: deleted_env_tables.py message updated (shell dropped Oct 2026); fidelity_tracker.py's dev daily-build path now resolves through the guard so --env dev/both fails loud with the ENV=prod pointer instead of a PostgREST 404; runtime proof recorded (a stale ENV=dev publish_team raises before any Supabase call). Guard/test battery: 140 passed / 0 failed on the task-scoped test set + pre-commit run -a green.

@nathan-probert

Copy link
Copy Markdown
Owner Author

feat(infra) @2567682 — hybrid ml-picks mode: the LIVE-READ side now splits from the WRITE side. run_daily_picks gains --source-env (CLI → SOURCE_ENV env var → follows the write env ENV) and an explicit --history-env (default prod PINNED — reads route to the ONE shared -prod ledger REGARDLESS, 027ef47); the Picks write stays the module-global config.ENV (job-level ENV) — deliberately no --env CLI re-bind. Supported combos documented in the module docstring; the owner's hybrid is ENV=dev + --source-env prod: real prod slate, writes ONLY Picks-dev.

Workflow (ml-picks.yml): new source input (default prod) beside environment (default prod); the resolve step whitelists BOTH axes dev|prod and hard-fails otherwise, binding step-level ENV= on both script steps and SOURCE_ENV + --source-env on the picks step; nightly cron / repository_dispatch resolve NO inputs → BOTH axes prod → ZERO behavioral change; backfill step unchanged (shared-ledger regardless).

LIVE local verification (2026-11-03, ENV=dev --source-env prod): (1) dry --date 2026-10-23 → env axes: write=Picks-dev | live read=Player-Snapshots-prod, 3 picks (MacKinnon/Kadri/Mikheyev), SMARTSCORE_ML_PICKS mean_max_prob 0.9232 published=false, no write; (2) non-dry --date 2026-10-19 → Picks-dev = exactly 3 rows for 2026-10-19 (Dylan Guenther/Utah tims1, Rickard Rakell/Pittsburgh tims2 stat 91.7, Mikhail Sergachev/Utah tims3), published=true; Picks-prod row count UNCHANGED 495→495.

Tests: tests/unit/test_predict_picks.py 47 passed (6 new: hybrid prod-source live-read + shared history, SOURCE_ENV env-var axis, default-unset == old behavior, resolution-order/fail-closed whitelist, --source-env/--history-env CLI mapping, fresh-import hybrid write → Picks-dev); named fallback batteries (template_env_vars, pipeline_schedule, shot_columns, backtrack_local_store, fidelity_tracker, backfill_missing_days, predict_picks) 164 passed; repo-wide ruff check/format clean (pre-commit hook equivalents).

Docs: ml-picks-runbook env matrix gains the hybrid row ("Pre-deployment validation: environment=dev + source=prod — real prod slate, writes Picks-dev") + the independent-artifact note (same S3/Storage key select on every profile); MODEL_NOTES runner pointer updated. Reminder: Picks-{ENV} is a single-day REPLACE — the table holds only the LAST run's picks and is wiped+reinserted each run.

…r's rule: prod is the source of truth for feature reads; explicit source=dev remains as the legacy override)
@nathan-probert

Copy link
Copy Markdown
Owner Author

feat(ci) 8b5a2f4 — source=prod is now the UNCONDITIONAL default in the ml-picks resolve (owner rule: prod is the source of truth for feature reads; the -dev live/backtrack shells are retired). The workflow's ${DISPATCH_SOURCE:-prod} fallback never consults the write env, and run_daily_picks' fallback is SOURCE_ENV_DEFAULT="prod" instead of the write env.
Behavior change: an environment=dev dispatch with source unset no longer silently reads the empty Player-Snapshots-dev and clean-skips green-but-empty — it reads the real prod slate and writes only Picks-dev; an explicitly-set source=dev remains the legacy dev-live-read override. Scheduled nightly + kick runs (both axes already prod) are byte-identical to before.
Runbook §1 env matrix + SOURCE-default rule + §1.1 clean-skip note updated; tests +3 (the dispatch source matrix: unset+dev→prod, explicit dev→dev, unset+prod→prod; test_predict_picks.py 49 passed).

…egacy wipe mode demoted behind --write-mode; top-3 becomes the summary output

OWNER-CORRECTED CONTRACT (2026-11-03): Picks-{ENV} is the FULL PLAYER
SLATE the site's own pipeline publishes (SaveToDb -> handle_publish_db ->
publish_public_db -> utility.save_to_db). The ML pass's DEFAULT write
mode is now stat_update:

* stat_update_picks_db (new): read today's Picks-{ENV} slate rows
  (picks_slate_rows), compute a prediction for EVERY player of the
  frame, and PATCH ONLY the stat column on the existing rows -
  update({'stat': v}).eq('id', row_id), one PATCH per row (cites the
  helper conventions of _picks_table_rows/save_picks_db). A PostgREST
  PATCH scoped to an existing id can never insert and never touches
  another column, so the row count, ids and all other columns are
  identical pre/post BY CONSTRUCTION (test-pinned). NO delete anywhere.
* A missing slate (the run fires before the site's SaveToDb publish) is
  a WARN + CLEAN SKIP - never our own minimal rows; stat PATCHes are
  read-modify-write, so later kicks re-fill the CURRENT slate
  (idempotent; the site's re-publishes re-clear stat and we re-fill
  AFTER, ordering already serialized by the ASL's PublishToDb ->
  KickMLPicks edge).
* Call-ups without history: WARN by name, stat left untouched. Frame
  players absent from the slate: WARN, no write.
* Top-3 challenge picks (one per tims group, ranked within (date,tims)
  as before) become an OUTPUT: the SMARTSCORE_ML_PICKS log carries a
  'picks' array + write_mode + stat_column fill counts, and the job
  summary lists the picks table - never a Picks-table mutation.
* The legacy wipe-then-publish-the-3-picks REPLACE survives behind
  --write-mode legacy (event write_mode -> WRITE_MODE env var ->
  stat_update default; whitelisted, fail-closed). Evidence for the
  demotion: handle_save_historic_db derives the calendar picks via
  choose_picks from its OWN state payload, and NotifyUsers' handle_emails
  derives the email picks via choose_picks from the PUBLISHED FULL slate
  rows - the 3-row wipe publication replaced the slate both derive from.
* stat_value factored out of pick_rows_to_picks_db so both modes share
  the stat convention (rank-pct*100, 1 decimal, None on non-finite).
* ml-picks.yml (workflow): picks step gains the explicit
  --write-mode (dispatch input write_mode, default stat_update) and
  corrected comments; runbook docs the corrected contract + timing.
* smoke (live, dev sandbox): Picks-dev seeded id-for-id with the
  site's 2026-11-03 full slate (495 rows, Picks-prod's own rows, no
  dev/prod column differences) -> stat_update filled 495/495 with the
  artifact artifact_2026-10-29; pre/post FULL-ROW diff: 495/495 stat
  values changed (Rust 0-1 -> ML percent), 0 non-stat column diffs, row
  count + ids unchanged.
@nathan-probert

Copy link
Copy Markdown
Owner Author

Owner-corrected contract implemented (feat/final-clean @ 1d8136f): Picks-{Env} is the FULL PLAYER SLATE the site’s own SaveToDb publishes (verified live: Picks-prod today = 495 rows, all players, Rust stats). The ML run now writes in a NEW DEFAULT stat_update mode: read today’s Picks-{ENV} rows, compute predictions for EVERY slate player, and PATCH ONLY the stat column per row (update({"stat": v}).eq("id", row_id) — a PostgREST PATCH can never insert or touch other columns, so row count/ids/all other columns are identical pre/post by construction). --write-mode legacy keeps the old wipe+3-row publication.
Evidence writes were demoted: UpdateHistory (handle_save_historic_db) builds the calendar picks via choose_picks from its OWN state payload, and NotifyUsers (handle_emails) derives the email picks via choose_picks from the PUBLISHED FULL slate rows — so the site’s own flows never needed the 3-row wipe publication (it actually replaced the slate their picks derive from). Top-3 becomes an OUTPUT: SMARTSCORE_ML_PICKS log carries a picks array + stat-column fill counts, and the job summary lists the picks table — never a Picks mutation.
Live smoke (dev sandbox, ENV=dev): Picks-dev seeded id-for-id with the site’s 2026-11-03 full slate (495 rows; dev/prod columns identical) → stat_update filled 495/495 (0 no-frame, 0 not-on-slate) with artifact 2026-10-29; pre/post FULL-ROW diff: 495/495 stat values changed (Rust 0–1 → ML percent), 0 non-stat column diffs, row count + ids unchanged. Missing-slate case = warn + clean skip (no invented rows); call-ups without history = warn by name, stat untouched.
Timing/ordering: stat PATCHes are read-modify-write — later kicks re-fill the CURRENT slate; the site’s own publish still full-wipes+reinserts its rows each pipeline run and our fill lands only AFTER it, which is already serialized by the ASL edge PublishToDb -> KickMLPicks (the kick fires at the END of the pipeline run).
Tests: pytest — test_predict_picks.py + test_picks_db_nan_behaviors.py 74 passed (new: stat-only PATCH/no-insert/no-wipe assertions, missing-slate skip, call-up/off-slate warns, dry read-only, legacy pin, mode resolution fail-closed), test_shot_columns 10, test_backtrack_local_store + test_ml_picks_kick 43; full tests/unit 565 with only 3 pre-existing/environmental failures reproduced on the pristine baseline (test_rust::test_predict + 2 schedule tests need a no-.env tree; the owner-local .env sets ENV=prod). pre-commit (ruff lint+format) clean.

… 26 cross-file failures)

tests/unit/test_predict_picks.py and tests/unit/test_picks_db_nan_behaviors.py
both exec'd run_daily_picks under the same sys.modules key; whichever file
imported second replaced the entry and the OTHER file's name-keyed fixture
patches ("run_daily_picks.SUPABASE_ADMIN_CLIENT" / ENV) landed on the wrong
module object -> 26 failures whenever the two files shared a pytest session
(each file passing alone). Both bootstraps now reuse the already-loaded
module when its __file__ matches, so one shared per-session module carries
both files' monkeypatch wiring (each test still wires/patches per-test and
monkeypatch restores). Verified: the full required command
142 passed (uv run pytest, 6 files) + pre-commit run -a all green.
@nathan-probert

nathan-probert commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner Author

Write-mode rework committed on feat/final-clean (1d8136f + test fix 67b13b3):

  • Corrected contract: stat_update is the DEFAULT write mode — it reads the site's own full-slate rows from Picks-{ENV} by <date> and PATCHes ONLY the stat column (update({'stat': v}).eq('id', row_id), per row): no delete, no insert, row count and all other columns unchanged by construction; missing slate = clean WARN+skip; call-ups without a prediction are WARNed and left untouched; frame players not on the slate are WARNed, never written. Legacy wipe-then-3-row replace is preserved exactly behind --write-mode legacy (CLI → event → WRITE_MODE env → stat_update default, fail-closed whitelist); top-3 challenge picks are now a structured SMARTSCORE_ML_PICKS log + step-summary OUTPUT only.
  • Tests: required command passes 142/142 (test_predict_picks, test_picks_db_nan_behaviors — incl. new pins: stat-only PATCH multi-row, full-row pre/post compare, NaN→None, missing-slate skip, warn-no-insert, legacy preserved; test_ml_picks_kick, test_shot_columns, test_backtrack_local_store, test_template_env_vars) + pre-commit run -a (ruff/format/ASL) all green.
  • Heads-up: a separate worker will verify the live Slate-fill next — dev (3-row) then prod (full-slate ~495 rows). Watch the log line for "write_mode": "stat_update" and the stat_column/stat_filled counts in SMARTSCORE_ML_PICKS.

@nathan-probert

Copy link
Copy Markdown
Owner Author

feat(infra): ml-picks fills stat on the full slate (no table wipe); legacy wipe mode preserved behind --write-mode legacy; top-3 becomes a summary output

  • Owner-corrected contract (2026-11-03): Picks-{ENV} is the site's own FULL PLAYER SLATE (SaveToDb → publish_public_db → utility.save_to_db, full wipe + full-slate reinsert). The ML pass's DEFAULT write mode is now stat_update: read today's slate rows, then PATCH only the stat column per row (update({'stat': v}).eq('id', row_id)) — never wipes, never inserts. The legacy wipe-then-publish-3-rows REPLACE is preserved behind --write-mode legacy (resolution: flag → event → WRITE_MODE env → stat_update; whitelisted, fail-closed). The workflow passes --write-mode stat_update explicitly in the picks step's ARGS.
  • Evidence the site's own flows never needed the 3-row publication: NotifyUsers and UpdateHistory both derive the picks via choose_picks (per-tims-group max stat) from the published FULL slate rows — so the top-3 picks became log/summary OUTPUT only (SMARTSCORE_ML_PICKS picks array + job-summary table).
  • Live smoke (Picks-prod, 2026-11-03 slate): stat_update: filled stat on 495/495 slate row(s) (no prediction for 0, not on slate: 0) — pre/post table identity verified by direct read: 495 rows → 495 rows, ids identical, all 495 rows changed stat-only (0 non-stat column diffs, no wipes, no inserts); largest stat diff 0.300774 → 99.6 (|d| = 99.30); the top-3 picks appear ONLY in the logs/summary.
  • Live smoke (Picks-dev): dry pass wrote nothing (495 rows identical), non-dry stat-filled 495/495 (idempotent re-fill: 0 non-stat diffs, zero stat drift).
  • Tests: 142 passed (uv run pytest across the 6 unit files — test_picks_db_nan_behaviors' new stat_update contract pins: stat_update never wipes, legacy kept behind --write-mode, missing/extra player warns, row count/columns unchanged) + uv run pre-commit run -a green (ruff-linter, ruff-format, asl-lint).
  • Caveat (timing): the site's own pipeline continues to full-wipe + rewrite the slate with ITS OWN stat values on every run — the ML stat fill must happen AFTER the site's primary publish. That ordering is the runbook's order-export caveat: the per-game kicks fire from the state machine's KickMLPicks state at the END of PlayerProcessingPipeline (after PublishToDb), and a manual run against an as-yet-unpublished date now clean-skips (WARN, no writes).

@nathan-probert

Copy link
Copy Markdown
Owner Author

stat_update write mode live-verified on dev + prod (slate 2026-11-03, artifact_2026-10-29_0.joblib, 5d old — fine):

  • dev smoke: dry run read-only (Picks-dev byte-identical pre/post, 0 rows / 0 columns changed); non-dry run stat_update: filled stat on 495/495 with 0 call-up warnings (Picks-dev already carried the site's full 495-row slate, so the re-fill landed byte-equal — a determinism proof).
  • prod smoke: stat_update: filled stat on 495/495; full pre/post JSON diff shows exactly stat changed on 495 rows — all 27 other columns and the row count (495) byte-identical; no wipes, no inserts, 0 frame misses.
  • idempotency: second prod pass byte-identical (full-table capture sha256 equal across runs; e.g. Teddy Blueger 0.2045 → 51.3 → 51.3).
  • Top-of-group: Cutter Gauthier (tims 1, 93.3), Logan Stankoven (tims 2, 95.6), Jakob Chychrun (tims 3, 95.6); mean_max_prob 0.9481. Docs updated with the verified-live block (ml-picks-runbook §1.1): 2cf3f35.

… only write mode; source=prod and the shared history ledger hard-pinned
@nathan-probert

Copy link
Copy Markdown
Owner Author

Three-knob ml-picks consolidation (d05b7c9, feat/final-clean): the CLI is now EXACTLY --date / --dry / --env dev|prod (default = the process-env ENV the workflow binds); --write-mode, --source-env, --history-env are deleted (argparse rejects them), and the workflow drops the write_mode/source dispatch inputs.

Deleted: the legacy wipe-then-publish-the-3 write mode is REMOVED ENTIRELY (not demoted) — save_picks_db + its probe/restore scaffolding are gone and stat_update (stat-only PATCH, the Picks table is NEVER wiped by the ML path) is the only mode; the source axis is absorbed into the hard pins SOURCE_ENV = 'prod' / HISTORY_ENV = 'prod' + the always-shared -prod history ledger (runbook matrix collapsed to the 3-knob contract).

User-visible difference: none on the non-dry/prod path — stat_update was already the default mode and prod was already the default source/history, so nightly behavior is byte-identical (the site's OWN SaveToDb wiper path is untouched and remains the only table wiper).

Tests: removed the legacy-mode/write-mode discrepancy and source/history resolution tests (git history retains), kept the per-row stat-only contract incl. a mocked 495-row Picks-prod full-slate test — required suite + ruff/pre-commit all green.

@nathan-probert

Copy link
Copy Markdown
Owner Author

Stat write boundary now divides by 100: stat_update_picks_db PATCHes stat_value(ensemble) / 100 (0-1 decimal, full precision; a 100th-percentile player writes exactly 1.0) — display (SMARTSCORE_ML_PICKS log + picks array) unchanged on the 0-100 scale. Commit 966b994.
Tests updated at the write surface (99.9→0.999, 11.1→0.111, 51.0→0.51, DB range ≤1.0) + new micro-assert that a 100th-percentile player's DB stat == 1.0 exactly: 62/62 pass across the two picks suites; full unit suite 550 passed with only 3 pre-existing failures (identical on clean d05b7c9).
Live-verified non-dry on the dev Picks-dev slate 2026-11-03: 495 rows → 495/495 stat-only PATCHes; pre/post diff shows ONLY stat moved (id set + every other column identical, row count unchanged); written stat min 0.01 / max 0.997, all ≤ 1.0.

@nathan-probert

Copy link
Copy Markdown
Owner Author

Migration added: supabase/migrations/20261103_add_legacy_stat_column.sql adds legacy_stat DOUBLE PRECISION to Picks-ENV (idempotent).
Site publish now writes legacy_stat (skaters + goalies) alongside stat in service.make_predictions_teams and save_to_db.
ML overlay proven not to touch legacy_stat: stat_update PATCH carries only {'stat': ...} (regression test).
Tests green (targeted + full unit, only pre-existing test_rust failure); commit 24ef51b.

@nathan-probert
nathan-probert merged commit 8fb020f into main Nov 4, 2026
6 checks passed
@nathan-probert
nathan-probert deleted the feat/final-clean branch November 4, 2026 01:56
@nathan-probert

Copy link
Copy Markdown
Owner Author

fix(historic-picks): strip legacy_stat from Historic-Picks upsert payload (commit 555c180).
Historic-Picks keeps its existing schema; legacy_stat lives on Picks only.
Stripped in write_historic_db (service.py) and update_historical_data (utility.py).
Player-Snapshots unaffected (allowlist excludes legacy_stat); Picks still carries legacy_stat.

@nathan-probert

Copy link
Copy Markdown
Owner Author

Historic-Picks now has a legacy_stat column (migration 20261104_add_legacy_stat_column_historic_picks.sql).
Strip approach reverted (commit d752517).
Migration commit 44bafa0.
Residual risk: existing Historic-Picks rows stay NULL until next publish.

@nathan-probert

Copy link
Copy Markdown
Owner Author

Added migration 20261104_add_legacy_stat_column_historic_picks.sql for Historic-Picks.
Column legacy_stat DOUBLE PRECISION with ADD COLUMN IF NOT EXISTS.
Mirrors Picks migration 20261103_add_legacy_stat_column.sql.
No Python files changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants