Repository navigation
feat(game): world-scale game services (watched views, fact batches, replication) - #292
Conversation
World cells are the write model; each zone keeps a pushed read model of
the cells it declares in CelldZone::watch. A watching read leaves an
in-memory lease, the cell pushes {topic, version, view} after commits
and alarms that change its view, the zone keeps the highest version per
topic and reconciles every 45 ± 15 s. Content rollout and dev settings
ride the same path, so the 1 s polls are gone; Mob Pit's hood, borough
and city reads become typed watches.
A view unchanged apart from its version is not pushed, so N rooms
waking (room_awake) do not cost N² pushes.
[[zone-world-views-pushed]] [[world-scale-game-services]]
The fact feed hands a lane's waiting facts to the sink at once
(FactSink::commit_batch, max_batch 16, ZONE_FACT_BATCH on celld).
CommandFactSink groups consecutive facts for one aggregate; the celld
object host sends them as one POST {kind}/{shard}/__batch, which
serve_cell runs entry by entry through the idempotent wait path (own
command id, principal and stamp) before one outbox drain. The zone
deletes settled facts from its outbox in one batched delete.
A batch stops at its first retryable failure and is retried from there,
attempts counting per fact, so order and exactly-once are unchanged.
Game code is unchanged.
[[zone-fact-batches]] [[world-scale-game-services]]
Every entity state is now a shared revision. Each changed entity's update record is encoded once per patch from its previous state, and clients with an older baseline share one record per baseline. Per-client work is selection and copy, and the bytes stay identical (a randomized test checks this against the old per-client path). Distance-based update rates (InterestConfig::update_rates, on by default): a known entity updates every patch within 0.5x the view radius, every 2nd patch up to 0.8x, and every 4th beyond, staggered by id. Creates, removes, globals and #[replicated(.., every_patch)] kinds are never held. [[replication-encode-once-lod]]
A sink that commits a batch one by one delivered no settlement until the whole batch was done (16 facts behind slow storage held back every settlement). FactSink::max_batch (default 1) and FactHost::max_fact_batch let a sink declare batching; the celld object host does, other sinks get one fact at a time as before. [[zone-fact-batches]]
Agents get one skill for the game layer: plain Rust zones, facts settled into cells (batched on celld), interest-managed replication with update rates, and zones watching world cells as pushed read models. [[world-scale-game-services]]
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
A watched neighbour room gave each observer the whole room from its
centre, so interest management stopped at the border. Clients now send
Focus {x, y} (tag 0x07, observers only) with the player's position in
the neighbour's coordinates; Zone::on_observer_focus recentres that
observer's view. Mob Pit applies VIEW_RADIUS, so interest and update
rates continue across rooms. JS: ZoneWatch.setFocus, throttled and
resent after reconnect.
[[observer-focus-neighbour-interest]]
|
Added 36054d8: neighbour rooms now apply the player's interest radius. Clients send |
A sink opted into batches by overriding both `FactSink::max_batch` and `FactSink::commit_batch` (a host: `max_fact_batch` and `dispatch_fact_batch`). Overriding only the size once batched a sink through the default one-by-one `commit_batch`, which held settlements back until the whole batch committed. Now one override opts in and supplies the batch commit: `FactSink::batches() -> Option<&dyn FactBatchSink>` and `FactHost::fact_batches() -> Option<&dyn FactBatchHost>`, both `None` by default. The feed batches only a sink that returns one (up to `FactFeed::with_max_batch` / `ZONE_FACT_BATCH`); the rest get one fact at a time. `DurableFacts`, `CommandFactSink`, `CelldObjectHost` and the test sinks moved over; behaviour is unchanged. New test: a sink without batches settles each fact as it commits. [[zone-fact-batches]]
mobpit-recovery wakes N rooms with one bot each, waits for (or runs) the kill, probes every room with real joins and reports when each answered (p50/p90/max, per 10 s), then each room's ticks over a hold. Bots record when their Welcome came (joined_after). Rooms report a wake trace in /info: when the object was built, its first call, and the awaited storage spans of its first wake and join. [[celld-fleet-recovery-hops]]
`AggregateCell::dispatch_idempotent_batch` runs commands in order, each exactly as `dispatch_idempotent` (own ledger entry, identity, session), under one command guard and one alarm reschedule instead of one per command. A rejected command leaves the earlier ones committed and the later ones run; the first retryable result (in progress, 408, 429, 5xx) ends the batch; one result per command attempted. The `__batch` wire moves to `distributed_game::cell::run_fact_batch` (host-testable); `serve_cell` parses, calls it and drains the outbox once. An entry that does not parse is answered in its place. Each entry is still its own transaction: a handler runs between its reservation and its commit, outside the synchronous cell SQL transaction, and must see the entries before it committed. Tests: a batch runs in order and replays as a whole; a rejected entry leaves the others committed; a replayed `__batch` body answers only replays and credits once. [[zone-fact-batches]]
Mob Pit's crawler and presence cells used their own router, which took `__batch` for a command name and answered 400. The object host only fell back to one-by-one on 404, so it read that 400 as the first fact's permanent refusal: a batch of kills for one crawler dropped its first kill. The cells now use `serve_cell`, and the object host sends the facts one by one whenever the batch request as a whole gets a 4xx other than 408/429, so each fact gets its own answer. [[zone-fact-batches]]
Plain integer counts that do not move with machine load, so A/B runs of
one build compare exactly: WorkCounters (records encoded per patch,
creates/updates sent, updates held by rates, Delta bytes, fact commands,
cell and __batch requests, outbox deletes, world reads and pushes) in
GET /zones/{zone}/info as `work`, with the switches in effect; watched
cells answer GET /{kind}/{shard}/__watchers with WatcherStats.
New A/B switches: ZONE_UPDATE_RATES=0, ZONE_WORLD_PUSH=0. The zone-side
reconcile schedule moves into world::watch::WorldSync, with a fake-clock
test that a reconcile repairs missed pushes within one period.
[[framework-work-counters]] [[zone-world-views-pushed]]
[[replication-encode-once-lod]] [[zone-fact-batches]]
Each step prints the rooms' and watched cells' work counters normalised per room patch, room-second or room-minute; --ab VAR=VAL with --pairs N alternates fresh-celld runs (A,B then B,A) and prints medians per side and of the per-pair change. Skill and README "Checking your work". [[framework-work-counters]] [[mobpit-zone-load-test]]
[[zone-fact-batches]]
Fact rates follow the fighting, which moves with machine load; the ratios are what a fact-batch A/B compares. [[framework-work-counters]] [[zone-fact-batches]]
Hello offered only version 1 and decoders reject trailing bytes, so Welcome could not grow and a new client message (Focus) made an old celld zone close the connection: servers had to deploy first. Servers (native net, celld zone object) now answer v1 or v2 in the client's version (ZoneHost::speaking) and refuse others with VERSION_MISMATCH "protocol ..." before the ticket is touched. Welcome v2 appends runtime features (bit 0 FOCUS) and the zone's world origin (Zone::origin). The JS SDK offers 2, resends the same Hello once with 1 to a v1-only server (remembered per URL, not surfaced as an error) and sends Focus only when FOCUS is announced. Golden fixtures cover v1 and v2; PROTOCOL.md gains a negotiation section. Game surfaces keep protocolVersion 1 (payload encodings are unchanged). [[zone-protocol-v2-negotiation]]
Each game converted the player's position into every neighbour's coordinates and called setFocus, and Mob Pit overrode on_observer_focus only to use its view radius. Zones now declare their world origin and player view radius once (Zone::origin, Zone::view_radius); the default on_observer_focus gives a focused observer that radius. The JS GameClient takes the player's position with one call (setPosition(x, y, zoneId?), local to a zone or world) and focuses every observed zone with an origin itself, also after its Welcome; setFocus stays for unplaced zones. Mob Pit declares both and drops its override; its client and the SvelteKit loop drop their coordinate math. [[zone-placement-auto-focus]]
[[zone-protocol-v2-negotiation]] [[zone-placement-auto-focus]]
celld holds a room's frames until the writes issued before them are durable. A wake wrote the zone id, the status and the activity fact before the first player's Welcome (and the join wrote the status again), so under a recovery burst Welcomes waited on the bucket. Now the wake only reads; the zone id is written once per zone, the status only when it changed and in the same batch as the facts, after the Welcome. Own celld node on the Hops RustFS, 150 fresh rooms: wake joins 89 -> 150 of 150 in the ramp; after a SIGKILL p50 38 -> 16 s, p90 44 -> 31 s; 2709 -> 2157 txns shipped in recovery. [[celld-fleet-recovery-hops]]
Probes and the staying bot reuse the room's principal (a reconnecting player, not new Crawler and Presence cells per attempt), send the game's content version, log why a wake bot was lost and count rooms that lose their player again; t0 is when the socket dropped. [[celld-fleet-recovery-hops]]
…lcome [[celld-fleet-recovery-hops]]
[[framework-work-counters]] [[zone-world-views-pushed]] # Conflicts: # distributed_game/src/celld_zone/object.rs
The watched world views (f93b24a) added the `hood` and `borough` zone messages; the generated client was not regenerated, so CI's `game-client --check` failed in the game demos smoke job. [[zone-world-views-pushed]]
Hood, borough and city are separate watched cells, so a waking room's views arrive independently. The test waited for the hood tier only and then asserted the borough count and city banner at once (failed in CI and 2 of 3 local runs). Wait for each, and retry a polled tiers read that fails while celld is busy. [[zone-world-views-pushed]]
Same hood/borough messages as e487715; the SvelteKit copy of the generated client was stale, so `npm run check:games` failed in CI. [[zone-world-views-pushed]]
One tick of walking is exactly the 0.25 m bound (5 m/s over a frame of up to 50 ms); crossing into the next room's coordinates adds f32 rounding, so CI saw 0.251 m (also on #291). Bound 0.26 m: a real doorway jump is a whole room or a visible snap.
A cell pushed each view once and logged only transport errors; a busy zone's 5xx/429 went unnoticed, so it caught up only at its next reconcile (45 ± 15 s). That is too late for a content rollout's freeze (the content headless test timed out waiting for both rooms frozen in CI). Retry after 0.25, 1 and 3 s on transport errors, 5xx and 429; a zone answers every parsed push with 204, so other 4xx are not retried. [[zone-world-views-pushed]]
Stacked on #291. Three changes from the Mob Pit load test, each behind the game layer's existing API (game code is unchanged apart from Mob Pit opting in to watches).
A. Zones watch world cells instead of polling (f93b24a)
CelldZone::watch→ typedWatches<M>; cells addCellWatchers).{topic, version, view}when its view changes; the zone keeps the highest version and reconciles every 45 ± 15 s.room_awake) caused N² pushes, and the first after-runs were much worse.B. Fact settlement in batches (f903581, e4af194)
FactSink::commit_batch/max_batchandFactHost::dispatch_fact_batch/max_fact_batch. The celld object host sends a lane's consecutive facts for one cell as onePOST …/__batch(ZONE_FACT_BATCH, default 16), and the zone deletes them from its outbox in one batched delete.__batchentry is still its own cell transaction; a single-transactionAggregateCellbatch is left open in the task.C. Replication: encode once, distance-based update rates (cd04f3e)
HostConfig::interest.update_rates;#[replicated(.., every_patch)]opts a kind out).Docs: README sections and a new
distributed-gameagent skill (2053dc5).Tests:
distributed_gameall-features (194) anddistributed_cli(316) pass. On celld:mobpit_smoke5/5 (including the crash mid-fight exactly-once test), and the headless mobpit / bosses / events / restarts / network / neighbours / prediction-wasm suites pass.Load numbers are not final. The machine sat at load 35–68 on 18 cores from other sessions, so the same config swung from 0 % to 98 % dropped between rounds.
[[world-scale-game-services]]