Skip to content

Sync upstream into my preview branch - #10

Merged
github-actions[bot] merged 27 commits into
sourcejump:u/zasbufrom
zasbu:chore/sync-upstream
Nov 6, 2026
Merged

github-actions[bot] merged 27 commits into
sourcejump:u/zasbufrom
zasbu:chore/sync-upstream

Conversation

@zasbu

@zasbu zasbu commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Brings the latest sourcejump/sourcejump main (up to #22) into my preview branch, since the preview sync can't update u/* branches right now.

Resolved two overlaps with my Activity work:

  • web/src/app/activity/page.tsx: kept the board layout. Upstream's change there was the loading bar PR, which this already had.
  • api/crates/api/src/runs/recent.rs: kept upstream's stored was_record for the WR flag, plus total and offset from Show the total runs and a last page link on Activity #9. Query cache regenerated for the combined query.

API and web checks pass.

stackedgamer and others added 27 commits November 6, 2026 08:43
A staff member with only manage_bans could set invalidate_runs on a ban and
invalidate every valid run the player had, bypassing the invalidate_runs
privilege that every other invalidation path checks (revoking a key with
invalidate_runs already required it). The ban is now refused whole with 403,
and the ban form no longer offers the option to staff who lack the privilege.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replay upload links pointed into the public bucket, under the files domain,
with no type or size limit on the PUT. A server's key could host any file
(HTML included) on files.sourcejump.net for as long as the job hadn't
deleted it, and could replace a small upload with a huge one after the
listing, which the job then read whole into memory.

Uploads now go to the private staging bucket, as staff viewer-file uploads
already do. The job reads them with Staging::read, which stops past
MAX_REPLAY_BYTES however the file changed since it was listed, and only the
SJR it builds from a checked file reaches the public bucket.

The header check also let a NaN time or tickrate through, because of
the comparison: `(NaN - x).abs() > limit` is false, so a NaN passed. Both
are now refused explicitly, with a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first download of a replay rebuilds its shavit file and keeps it in the
public bucket. Its key was the SJR's with .replay in place of .sjr, and the
SJR's URL is public on every run page, invalidated runs included. So anyone
could fetch an invalidated run's shavit file straight from the files
domain, which the download route deliberately refuses to hand out.

Kept files now get their own random key part, and two first downloads at
once settle on whichever recorded its copy first, deleting the spare. On
start the API deletes files already kept under a derived key and forgets
them; the next download rebuilds each under a random key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing tied Steam's answer to the browser bringing it back. Someone could
start a sign-in to their own account, stop at the callback link Steam
returns, and send that link to someone else, whose browser would then be
signed in as them (login CSRF): anything that person did next, a post or
staff work in a shared browser, would land on the wrong account.

/v1/auth/steam now sets an sj_sign_in cookie with a random value, scoped to
the callback path, and puts the same value in the callback URL Steam signs.
The callback refuses an answer whose value doesn't match the cookie, and
clears the cookie either way. A sign-in started later in another tab
replaces the value, so only the latest one completes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rivilege

Require invalidate_runs to take a player's runs with a ban
Bind Steam sign-in to the browser that started it
…taging

Take replay uploads in the private staging bucket
Keep rebuilt shavit replays under their own random key
* Fixed inconsistent flag scales

* Fixed inconsistent flag scales

* Fixed inconsistent flag scales

* Fixed inconsistent flag scales
* Added a separator line between map name and server ip

* Tidied the server rows on phones

* Copy only the server address, and centre the copy icon

* Changed connected badge to a simple checkmark to reduce space used, moved player count to bottom right on mobile

* Slightly modified how the servers appear, and made the players graph a bit more compact to reduce vertical space wasted

* Slightly modified how the servers appear, and made the players graph a bit more compact to reduce vertical space wasted

* Slightly modified how the servers appear, and made the players graph a bit more compact to reduce vertical space wasted

* Slightly modified how the servers appear, and made the players graph a bit more compact to reduce vertical space wasted
* Added a page loading bar

* Added a page loading bar

* Added a page loading bar
External audit, 2026-11-06 (sourcejump#4, sourcejump#5, sourcejump#7, sourcejump#8 and sourcejump#14).

- Active bans are two views (migration 0047): `active_bans`, and
  `records_bans` for the ones that keep a player off the boards. The 13
  copies of the condition read them. Views, so the planner still runs a
  board's ban check as one anti-join.
- WR history, recent runs, the Discord feeds and the daily totals read the
  stored `runs.was_record` instead of working it out again. The Discord
  WR feed took any rank-1 run, so it announced ties as world records.
- Standings break ties on the SteamID, so offset pages keep one order. The
  web table drops repeats and stops at an empty page.
- A timed records ban ends with nothing written, so a job looks every five
  minutes for players it should put back on the Pre-2.0 standings.
- Daily totals recount older days that late, invalidated or restored runs
  changed, which a trigger on runs records (migration 0048).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pre-2.0 board's "out of" counted distinct maps over every legacy run; it now
counts legacy maps that have a run, one index probe each (~68 ms to ~25 ms on the
full legacy import).

The map list counted every map's players and bonuses before paging, whatever the
sort. Only the popularity sorts need every map's player count now; the page
counts its own rows' players and bonuses. On 6.7k maps and 179k personal bests,
the other sorts drop from ~250-440 ms to ~55-230 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y map names

Fixes from the 2026-11-06 audit (sourcejump#9, sourcejump#10, sourcejump#15, sourcejump#19, dot map names):

- Replay checks, SJR encoding, route tracing and SJR decoding run in
  spawn_blocking, as map images already do: files up to 128 MB at gzip's
  highest level took seconds of CPU on the async workers.
- Storage PUTs (viewer files up to ~256 MB) get 10 minutes and streamed
  downloads 30 minutes instead of the client's 30 s, which covered the whole
  body and cut off large or slow transfers.
- The heartbeat checks its per-IP limit before it looks the key up, so
  over-limit senders cost no database query. Tested with a closed pool.
- Map names made only of dots are refused: names end up in storage keys and
  URL paths.
- Tests that the description and release PUTs need manage_maps, and a success
  test for the description.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g with it

External audit, 2026-11-06 (sourcejump#3, sourcejump#12, sourcejump#18), measured on a synthetic 3M-run
database: on a board of 30k players, every run took seconds and most
timed out at the 5 s statement limit.

- A personal best rebuilds the totals of the players whose points changed
  and the run's own player, not everyone on its track. Totals are summed
  per player through the primary key; joined on the list, the planner read
  every personal best.
- Record-when-set is recounted from the board's earliest affected run on.
  The best before it is the latest stored record before it, found through
  a new partial index (migration 0049), so a run arriving now reads itself.
- World record holders are read from the board's best time instead of
  checking every personal best for a faster one.
- Lock order is the same everywhere: map row, personal bests, standings
  lock. A full recount locks personal_bests first and a tier change its
  map, so neither can deadlock with a run being stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…every PB

"Most players" is the map list's default sort, and it counted distinct players
over every personal best on each view: about 225 ms at the legacy site's size
(6.7k maps, 179k PBs), growing with every PB. `maps.players` now holds the
count, filled by the migration and recounted in `standings::refresh_runs`,
which every writer of personal bests (ingest, invalidation, restore) already
calls with the maps' rows locked. The recount runs before the standings lock,
which every ingest waits on; the map row locks are what keep it right. It
stays out of `refresh_all`, which locks personal bests before map rows.

The popularity sorts now cost what A to Z does, about 50 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A new world record moves the points of everyone on its board. The recount
summed each of those players' totals again from all their personal bests,
which read nearly every personal best on a big board. The points update now
returns each row's old and new points (RETURNING old/new, Postgres 18), and
totals move by that change. The run's own players are still summed again,
as their completed count may change. Board recounts for a run cover its own
style only; a tier change still covers every style.

Ingest now recounts last in its transaction, so the standings lock, which
every recount waits on, is held for less time.

refresh_track touched personal_bests only after taking the standings lock,
while refresh_all locks the table first: a tier change and a policy publish
at once could deadlock (40P01). It now locks the table before the standings
lock, as the documented lock order says.

Bench (3M runs, debug build, median), new WR: 30k-player board 11.2 s ->
2.7 s, 3k-player board 1.1 s -> 0.23 s, 200-player board 78 -> 28 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep replay work and key lookups off the hot path, and refuse dot-only map names
Move totals by the change on a new WR, and fix a tier-change deadlock
Store each map's player count so the default map sort stops counting every PB
Keep run ingest fast on big boards, and stop full recounts deadlocking with it
Read each board rule from one place, and fix what the copies got wrong
A board's points all follow from its best time and tier, so those are
stored per board (standings_boards), as the totals were counted with
them, and pages work a personal best's points out on read
(personal_best_points). personal_bests.points is dropped.

- A full recount writes the boards and the totals only. It rewrote every
  personal best, which was 4 of its 4.5 minutes at 10x the legacy data;
  it now takes 13-22 s there. It runs after the API starts listening,
  and it no longer locks personal_bests, so ingest on other maps goes on.
- A board recount compares the board's best and tier with the stored
  ones. Unchanged (a PB that isn't a WR): only the runner is summed
  again. Changed (a new WR or tier): everyone else's total moves by the
  change, and the runner is summed again.
- Recounts never write personal bests now, so the refresh_track and
  refresh_all locks that kept them from deadlocking with ingest go.
  standings_boards has no foreign key to maps, since checking one locks
  the map row ingest holds while it waits for a recount.

Rolling back past migration 0051 needs the backup taken before it, as
older binaries read the dropped column (docs/deploy.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Store each board's counted best and tier instead of every PB's points
# Conflicts:
#	api/.sqlx/query-02235b882665d8454409c6721c6e7a34ffa7328ea8650f68b3ed0bad729c1e34.json
#	api/.sqlx/query-0bfbe0266373668248628f241c61ed7472c84e03eb8177379011350cb79c84e8.json
#	api/.sqlx/query-79f8a533b9ffccdb99dad6b682bac3b451a29743fab3da841bd22e572864ff19.json
#	api/crates/api/src/runs/recent.rs
@github-actions
github-actions Bot merged commit 14073f5 into sourcejump:u/zasbu Nov 6, 2026
3 checks passed
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Merged into u/zasbu. It'll be live at https://zasbu-sj.okwhiteli.st in a few minutes.

@zasbu
zasbu deleted the chore/sync-upstream branch November 9, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants