Skip to content

fix: make masterfile reload safe under concurrent queries - #19

Open
jfberry wants to merge 1 commit into
masterfrom
fix/concurrent-masterfile-reload
Open

fix: make masterfile reload safe under concurrent queries#19
jfberry wants to merge 1 commit into
masterfrom
fix/concurrent-masterfile-reload

Conversation

@jfberry

@jfberry jfberry commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Problem

Reloading the masterfile while queries are running is a data race that crashes consumers. Golbat hit this in production as fatal error: concurrent map read and map write inside QueryPvPRank (ohbem.go:408) — root-caused and worked around app-side in UnownHash/Golbat#403, but the underlying issue is in gohbem and affects every consumer that reloads:

  1. LoadPokemonData mutates live maps. It unmarshals into the existing o.PokemonData, and encoding/json reuses non-nil maps — writing keys into the very Pokemon/Costumes maps concurrent QueryPvPRank/CalculateCp calls are reading. This is the production crash.
  2. WatchPokemonData publishes via an unsynchronized struct assignment. o.PokemonData = pokemonData is a multi-word write with no happens-before edge, so readers can see a torn mix of old/new fields, or a new map pointer before its contents are visibly published (arm64).
  3. ClearCache overwrites a live sync.Map (o.compactRankCache = sync.Map{}) while queries are mid-Load/Store on it — clobbering its internal state.

All three are flagged deterministically by go test -race with the included regression test.

Fix

The masterfile snapshot and the rank cache computed from it now live in one immutable bundle behind an atomic.Pointer:

  • Reloads (LoadPokemonData, FetchPokemonData, the watcher) build a fresh bundle — fresh maps, empty cache — and publish it with a single atomic store. Readers already holding the old bundle finish on it undisturbed.
  • Every query path loads the bundle once per call, so data and cache always come from the same masterfile version. (To be precise: on master the watcher stored new data before clearing the cache, but since cache entries are keyed by (cpCap, base stats) their values are masterfile-version-independent, so that window was benign for correctness — the atomic swap just removes it and reclaims the stale entries in one step.)
  • ClearCache swaps in a same-data/empty-cache bundle via CAS.
  • calculateAllRanksCompact no longer lazily writes o.RankingComparator on the query path (that write also raced concurrent queries).

Compatibility

  • Public API is unchanged; Golbat's full test suite passes against this branch via a replace directive.
  • The exported PokemonData field is kept: populating it directly before first use still works (it seeds the first snapshot), and reloads mirror into it for legacy readers. Its doc comment now spells out that it must not be read concurrently with a reload. If you'd rather unexport it in a future breaking release, happy to follow up.
  • VERSION left untouched — assuming you prefer to bump on release.

Verification

  • New TestReloadConcurrentWithQueries fails on master under -race (races 1 and 3 above) and passes here; ran with -count=3.
  • Full suite passes under -race; benchmarks confirm the cache path still hits (BenchmarkQueryPvPRankCached ~25× faster than uncached).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BMJMXHUiq6Pnce9KshgFTC

Reloading the masterfile while queries run crashed consumers with
"fatal error: concurrent map read and map write" (hit in production by
Golbat, see UnownHash/Golbat#403):

- LoadPokemonData unmarshaled into the live PokemonData struct;
  encoding/json reuses non-nil maps, writing into the very maps
  QueryPvPRank readers walk with no lock.
- WatchPokemonData published a new PokemonData via an unsynchronized
  multi-word struct assignment (torn reads, no happens-before).
- ClearCache overwrote the live sync.Map while queries were mid
  Load/Store on it.

The masterfile snapshot and the rank cache computed from it now live in
one immutable bundle behind an atomic.Pointer. Reloads build a fresh
bundle (fresh maps, empty cache) and publish it with a single atomic
store; readers pick up the bundle once per call. This also closes the
window where a reload served new masterfile data with ranks cached from
the old one (data was stored before the cache was cleared), and stops
calculateAllRanksCompact from lazily writing RankingComparator on the
query path.

The exported PokemonData field is kept for backward compatibility: it
still seeds the first bundle when populated directly, and reloads mirror
into it for legacy readers, but internal code no longer reads it after
publication.

Regression test exercises LoadPokemonData and ClearCache concurrently
with QueryPvPRank and CalculateCp under -race.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BMJMXHUiq6Pnce9KshgFTC
@jfberry

jfberry commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Heads-up: after opening this I found #18, which identified the same data races back in February (§1.6 of its DETAIL_REVIEW.md, including the lazy RankingComparator write) and fixes them as part of a broader modernization, using an RWMutex plus an atomic.Pointer[sync.Map] for the cache rather than the immutable-bundle swap here. The two PRs conflict in every shared region, so they can't both land as-is.

I've left a comment on #18 comparing the two concurrency approaches and asking @lenisko how he'd prefer to resolve the overlap. Whichever vehicle wins, the TestReloadConcurrentWithQueries regression test here is portable to either design.

Context for why this exists: the LoadPokemonData in-place unmarshal crashed Golbat in production (concurrent map read and map write in QueryPvPRank) — root cause and app-side fix in UnownHash/Golbat#403.

@lenisko

lenisko commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Feel free to merge @jfberry since Golbat is single consumer so far, I will close other one later. Please also bump https://github.com/UnownHash/gohbem/blob/master/ohbem.go#L18 to 0.13.0

@jfberry

jfberry commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Golbat actually uses the current build in a race free way now; but I will take 18+19 and merge them together then to make gohbem reliable (and I guess we don't know for sure if there are other users or not!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants