Skip to content

feat(game): world-scale game services (watched views, fact batches, replication) - #292

Merged
patrickleet merged 26 commits into
feat/mobpit-zone-gatewayfrom
feat/world-scale-services
Oct 24, 2026
Merged

patrickleet merged 26 commits into
feat/mobpit-zone-gatewayfrom
feat/world-scale-services

Conversation

@patrickleet

Copy link
Copy Markdown
Collaborator

Stacked on #291. Three changes from the Mob Pit load test, each behind the game layer's existing API (game code is unchanged apart from Mob Pit opting in to watches).

A. Zones watch world cells instead of polling (f93b24a)

  • World cells are the write model; zones keep pushed read models (CelldZone::watch → typed Watches<M>; cells add CellWatchers).
  • A watching read leaves an in-memory lease; the cell pushes {topic, version, view} when its view changes; the zone keeps the highest version and reconciles every 45 ± 15 s.
  • Content rollout and dev settings use the same path, so the 1 s polls are gone.
  • A view unchanged apart from its version is not pushed. Without that, rooms waking (room_awake) caused N² pushes, and the first after-runs were much worse.

B. Fact settlement in batches (f903581, e4af194)

  • FactSink::commit_batch / max_batch and FactHost::dispatch_fact_batch / max_fact_batch. The celld object host sends a lane's consecutive facts for one cell as one POST …/__batch (ZONE_FACT_BATCH, default 16), and the zone deletes them from its outbox in one batched delete.
  • A batch stops at its first retryable failure and is retried from there, with attempts counted per fact, so order and exactly-once are unchanged. Sinks that don't batch keep getting one fact at a time.
  • Each __batch entry is still its own cell transaction; a single-transaction AggregateCell batch is left open in the task.

C. Replication: encode once, distance-based update rates (cd04f3e)

  • Each changed entity is encoded once per patch and shared; the bytes match the old per-client path (randomized test).
  • Far entities update every 2nd or 4th patch (HostConfig::interest.update_rates; #[replicated(.., every_patch)] opts a kind out).
  • Patch build micro-benchmark: 64 clients 1.30 → 0.19 ms, 128 clients 4.64 → 0.78 ms.

Docs: README sections and a new distributed-game agent skill (2053dc5).

Tests: distributed_game all-features (194) and distributed_cli (316) pass. On celld: mobpit_smoke 5/5 (including the crash mid-fight exactly-once test), and the headless mobpit / bosses / events / restarts / network / neighbours / prediction-wasm suites pass.

Load numbers are not final. The machine sat at load 35–68 on 18 cores from other sessions, so the same config swung from 0 % to 98 % dropped between rounds.

  • One alternating local A/B for part A: refusals on saturated world cells fell from 85 to 25 at 128 rooms (move-only) and from 12 to 3 at 64 rooms (attacking).
  • Before/after on a quiet machine and on Hops are still open in the GitKB tasks.

[[world-scale-game-services]]

World cells are the write model; each zone keeps a pushed read model of
the cells it declares in CelldZone::watch. A watching read leaves an
in-memory lease, the cell pushes {topic, version, view} after commits
and alarms that change its view, the zone keeps the highest version per
topic and reconciles every 45 ± 15 s. Content rollout and dev settings
ride the same path, so the 1 s polls are gone; Mob Pit's hood, borough
and city reads become typed watches.

A view unchanged apart from its version is not pushed, so N rooms
waking (room_awake) do not cost N² pushes.

[[zone-world-views-pushed]] [[world-scale-game-services]]
The fact feed hands a lane's waiting facts to the sink at once
(FactSink::commit_batch, max_batch 16, ZONE_FACT_BATCH on celld).
CommandFactSink groups consecutive facts for one aggregate; the celld
object host sends them as one POST {kind}/{shard}/__batch, which
serve_cell runs entry by entry through the idempotent wait path (own
command id, principal and stamp) before one outbox drain. The zone
deletes settled facts from its outbox in one batched delete.

A batch stops at its first retryable failure and is retried from there,
attempts counting per fact, so order and exactly-once are unchanged.
Game code is unchanged.

[[zone-fact-batches]] [[world-scale-game-services]]
Every entity state is now a shared revision. Each changed entity's
update record is encoded once per patch from its previous state, and
clients with an older baseline share one record per baseline. Per-client
work is selection and copy, and the bytes stay identical (a randomized
test checks this against the old per-client path).

Distance-based update rates (InterestConfig::update_rates, on by
default): a known entity updates every patch within 0.5x the view
radius, every 2nd patch up to 0.8x, and every 4th beyond, staggered by
id. Creates, removes, globals and #[replicated(.., every_patch)] kinds
are never held.

[[replication-encode-once-lod]]
A sink that commits a batch one by one delivered no settlement until the
whole batch was done (16 facts behind slow storage held back every
settlement). FactSink::max_batch (default 1) and FactHost::max_fact_batch
let a sink declare batching; the celld object host does, other sinks get
one fact at a time as before.

[[zone-fact-batches]]
Agents get one skill for the game layer: plain Rust zones, facts settled
into cells (batched on celld), interest-managed replication with update
rates, and zones watching world cells as pushed read models.

[[world-scale-game-services]]
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5fff3990-7fe0-430c-96df-15bf107eae6e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

A watched neighbour room gave each observer the whole room from its
centre, so interest management stopped at the border. Clients now send
Focus {x, y} (tag 0x07, observers only) with the player's position in
the neighbour's coordinates; Zone::on_observer_focus recentres that
observer's view. Mob Pit applies VIEW_RADIUS, so interest and update
rates continue across rooms. JS: ZoneWatch.setFocus, throttled and
resent after reconnect.

[[observer-focus-neighbour-interest]]
@patrickleet

Copy link
Copy Markdown
Collaborator Author

Added 36054d8: neighbour rooms now apply the player's interest radius. Clients send Focus {x, y} (0x07, observers only) with the player's position in the neighbour's coordinates. Zone::on_observer_focus recentres the observer's view; Mob Pit uses VIEW_RADIUS. Headless check: neighbour mobs drawn only within the view radius (0 of 6 from the middle of the room, 2 of 6 by the wall). Compatibility: an older celld zone closes the connection on this tag, so deploy servers first.

A sink opted into batches by overriding both `FactSink::max_batch` and
`FactSink::commit_batch` (a host: `max_fact_batch` and
`dispatch_fact_batch`). Overriding only the size once batched a sink
through the default one-by-one `commit_batch`, which held settlements
back until the whole batch committed.

Now one override opts in and supplies the batch commit:
`FactSink::batches() -> Option<&dyn FactBatchSink>` and
`FactHost::fact_batches() -> Option<&dyn FactBatchHost>`, both `None`
by default. The feed batches only a sink that returns one (up to
`FactFeed::with_max_batch` / `ZONE_FACT_BATCH`); the rest get one fact
at a time. `DurableFacts`, `CommandFactSink`, `CelldObjectHost` and the
test sinks moved over; behaviour is unchanged. New test: a sink without
batches settles each fact as it commits.

[[zone-fact-batches]]
mobpit-recovery wakes N rooms with one bot each, waits for (or runs)
the kill, probes every room with real joins and reports when each
answered (p50/p90/max, per 10 s), then each room's ticks over a hold.
Bots record when their Welcome came (joined_after). Rooms report a
wake trace in /info: when the object was built, its first call, and
the awaited storage spans of its first wake and join.

[[celld-fleet-recovery-hops]]
`AggregateCell::dispatch_idempotent_batch` runs commands in order, each
exactly as `dispatch_idempotent` (own ledger entry, identity, session),
under one command guard and one alarm reschedule instead of one per
command. A rejected command leaves the earlier ones committed and the
later ones run; the first retryable result (in progress, 408, 429,
5xx) ends the batch; one result per command attempted.

The `__batch` wire moves to `distributed_game::cell::run_fact_batch`
(host-testable); `serve_cell` parses, calls it and drains the outbox
once. An entry that does not parse is answered in its place.

Each entry is still its own transaction: a handler runs between its
reservation and its commit, outside the synchronous cell SQL
transaction, and must see the entries before it committed.

Tests: a batch runs in order and replays as a whole; a rejected entry
leaves the others committed; a replayed `__batch` body answers only
replays and credits once.

[[zone-fact-batches]]
Mob Pit's crawler and presence cells used their own router, which took
`__batch` for a command name and answered 400. The object host only
fell back to one-by-one on 404, so it read that 400 as the first fact's
permanent refusal: a batch of kills for one crawler dropped its first
kill.

The cells now use `serve_cell`, and the object host sends the facts
one by one whenever the batch request as a whole gets a 4xx other than
408/429, so each fact gets its own answer.

[[zone-fact-batches]]
Plain integer counts that do not move with machine load, so A/B runs of
one build compare exactly: WorkCounters (records encoded per patch,
creates/updates sent, updates held by rates, Delta bytes, fact commands,
cell and __batch requests, outbox deletes, world reads and pushes) in
GET /zones/{zone}/info as `work`, with the switches in effect; watched
cells answer GET /{kind}/{shard}/__watchers with WatcherStats.

New A/B switches: ZONE_UPDATE_RATES=0, ZONE_WORLD_PUSH=0. The zone-side
reconcile schedule moves into world::watch::WorldSync, with a fake-clock
test that a reconcile repairs missed pushes within one period.

[[framework-work-counters]] [[zone-world-views-pushed]]
[[replication-encode-once-lod]] [[zone-fact-batches]]
Each step prints the rooms' and watched cells' work counters normalised
per room patch, room-second or room-minute; --ab VAR=VAL with --pairs N
alternates fresh-celld runs (A,B then B,A) and prints medians per side
and of the per-pair change. Skill and README "Checking your work".

[[framework-work-counters]] [[mobpit-zone-load-test]]
Fact rates follow the fighting, which moves with machine load; the
ratios are what a fact-batch A/B compares.

[[framework-work-counters]] [[zone-fact-batches]]
Hello offered only version 1 and decoders reject trailing bytes, so
Welcome could not grow and a new client message (Focus) made an old
celld zone close the connection: servers had to deploy first.

Servers (native net, celld zone object) now answer v1 or v2 in the
client's version (ZoneHost::speaking) and refuse others with
VERSION_MISMATCH "protocol ..." before the ticket is touched. Welcome v2
appends runtime features (bit 0 FOCUS) and the zone's world origin
(Zone::origin). The JS SDK offers 2, resends the same Hello once with 1
to a v1-only server (remembered per URL, not surfaced as an error) and
sends Focus only when FOCUS is announced. Golden fixtures cover v1 and
v2; PROTOCOL.md gains a negotiation section. Game surfaces keep
protocolVersion 1 (payload encodings are unchanged).

[[zone-protocol-v2-negotiation]]
Each game converted the player's position into every neighbour's
coordinates and called setFocus, and Mob Pit overrode on_observer_focus
only to use its view radius.

Zones now declare their world origin and player view radius once
(Zone::origin, Zone::view_radius); the default on_observer_focus gives a
focused observer that radius. The JS GameClient takes the player's
position with one call (setPosition(x, y, zoneId?), local to a zone or
world) and focuses every observed zone with an origin itself, also after
its Welcome; setFocus stays for unplaced zones. Mob Pit declares both
and drops its override; its client and the SvelteKit loop drop their
coordinate math.

[[zone-placement-auto-focus]]
[[zone-protocol-v2-negotiation]] [[zone-placement-auto-focus]]
celld holds a room's frames until the writes issued before them are
durable. A wake wrote the zone id, the status and the activity fact
before the first player's Welcome (and the join wrote the status
again), so under a recovery burst Welcomes waited on the bucket.

Now the wake only reads; the zone id is written once per zone, the
status only when it changed and in the same batch as the facts, after
the Welcome. Own celld node on the Hops RustFS, 150 fresh rooms:
wake joins 89 -> 150 of 150 in the ramp; after a SIGKILL p50 38 ->
16 s, p90 44 -> 31 s; 2709 -> 2157 txns shipped in recovery.

[[celld-fleet-recovery-hops]]
Probes and the staying bot reuse the room's principal (a reconnecting
player, not new Crawler and Presence cells per attempt), send the
game's content version, log why a wake bot was lost and count rooms
that lose their player again; t0 is when the socket dropped.

[[celld-fleet-recovery-hops]]
[[framework-work-counters]] [[zone-world-views-pushed]]

# Conflicts:
#	distributed_game/src/celld_zone/object.rs
The watched world views (f93b24a) added the `hood` and `borough` zone
messages; the generated client was not regenerated, so CI's
`game-client --check` failed in the game demos smoke job.

[[zone-world-views-pushed]]
Hood, borough and city are separate watched cells, so a waking room's
views arrive independently. The test waited for the hood tier only and
then asserted the borough count and city banner at once (failed in CI
and 2 of 3 local runs). Wait for each, and retry a polled tiers read
that fails while celld is busy.

[[zone-world-views-pushed]]
Same hood/borough messages as e487715; the SvelteKit copy of the
generated client was stale, so `npm run check:games` failed in CI.

[[zone-world-views-pushed]]
One tick of walking is exactly the 0.25 m bound (5 m/s over a frame of up
to 50 ms); crossing into the next room's coordinates adds f32 rounding,
so CI saw 0.251 m (also on #291). Bound 0.26 m: a real doorway jump is a
whole room or a visible snap.
A cell pushed each view once and logged only transport errors; a busy
zone's 5xx/429 went unnoticed, so it caught up only at its next
reconcile (45 ± 15 s). That is too late for a content rollout's freeze
(the content headless test timed out waiting for both rooms frozen in
CI). Retry after 0.25, 1 and 3 s on transport errors, 5xx and 429; a
zone answers every parsed push with 204, so other 4xx are not retried.

[[zone-world-views-pushed]]
@patrickleet
patrickleet merged commit cd6f9bf into feat/mobpit-zone-gateway Oct 24, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant