Repository navigation
feat(game): Mob Pit through the framework gateway at /zones - #291
Merged
patrickleet merged 46 commits intoOct 24, 2026
Merged
Conversation
Routes a prefix (e.g. /zones) to a zone host such as celld's public listener: path forwarded unchanged, WebSocket upgrades always allowed, no credential forwarded (zones verify their own tickets). [[mobpit-hops-local-gateway]]
The services (game-demos mobpit) now serve behind the framework gateway:
/zones/{room} goes to the rooms on celld (--zones-origin, default
--celld-url), /internal and /dev to the services, / to --ui-origin when
set; --public-origin names the origin behind an ingress. Vite proxies
game traffic to the gateway only. Room paths move from /rooms to /zones,
the framework's term (also the celld zone test worker and docs).
[[mobpit-hops-local-gateway]]
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
mobpit-system Environment: CelldStack (celld-stack, RustFS) in mobpit-celld plus a local CA celld trusts for its HTTPS outbox relay. Per-checkout Environment: services + gateway (cargo from the mounted checkout), Vite and an nginx HTTPS relay endpoint in one pod, a worker deployer that celld-deploys cells/build on change, and an HTTPRoute on the Istio platform Gateway at <checkout>.gitkb.localhost. ./run.sh build runs the host builds the setup needs. [[mobpit-hops-local-gateway]]
new_mesh allows cleartext only to a cluster Service (*.svc), for relays whose pods a mesh encrypts (Istio ambient mTLS). The default stays HTTPS-only. [[mobpit-hops-local-gateway]]
celld's namespace and the Mob Pit pod join the ambient mesh; the worker relays to the mobpit Service with CELLD_QUEUE_RELAY_MESH=1. Drops the local CA, the nginx TLS relay and celld's trust mount. [[mobpit-hops-local-gateway]]
kind-hops' Dory kernel lacks IPv6 policy routing, so istio-cni cannot enroll ambient pods. celld and Mob Pit pods get sidecars instead. [[mobpit-hops-local-gateway]]
A failed stat or copy of the host build (missing files, host-share I/O errors) skips the round instead of exiting the pod under set -e.
The view, lag-compensation and other toggles sat below the stats, off screen; they now follow the hint with their keys. Shots draw a brief tracer to where they land.
RT fires (held: at the fire cooldown), R3 toggles first person, View toggles lag compensation; the HUD shows each toggle's key and button. C no longer also jumps (it toggles lag compensation).
The bolt travels from the shooter to the impact at 60 m/s (at most 0.4 s); the landing marker appears when it arrives.
Players get a bar over their head (the label keeps name and level); in first person your HP shows in the bottom HUD, amber then red when low.
The controlled entity hopped once per input interval against a smooth camera (visible jitter at 30 Hz on a faster display). Each recorded step now blends from what was shown over its interval; a Delta mid-step shifts the blend by the correction so only the error is eased. On by default; smoothSteps: false draws the prediction the instant it is made.
The fill was a second billboard at the back's exact spot (z-fighting: black flashes) scaled about its centre and jumping to each value. It is now the back's child, just in front, anchored at the left edge, and eases to the health.
Rust bots ramp ZONESxBOTS into Mob Pit rooms and report the worst room's dropped ticks / late p99, RTT, bytes per bot and celld CPU/RSS. ZONE_MAX_PLAYERS raises the room cap (rooms.maxPlayers on Hops); scripts/hops/mobpit-load.sh runs the bots in-cluster and samples celld with crictl. Bots report the close code. RustFS gets 4Gi: at 1Gi it OOMs under ~256 busy rooms and celld self-fences while recovering. [[mobpit-zone-load-test]]
World cells are the write model; each zone keeps a pushed read model of
the cells it declares in CelldZone::watch. A watching read leaves an
in-memory lease, the cell pushes {topic, version, view} after commits
and alarms that change its view, the zone keeps the highest version per
topic and reconciles every 45 ± 15 s. Content rollout and dev settings
ride the same path, so the 1 s polls are gone; Mob Pit's hood, borough
and city reads become typed watches.
A view unchanged apart from its version is not pushed, so N rooms
waking (room_awake) do not cost N² pushes.
[[zone-world-views-pushed]] [[world-scale-game-services]]
The fact feed hands a lane's waiting facts to the sink at once
(FactSink::commit_batch, max_batch 16, ZONE_FACT_BATCH on celld).
CommandFactSink groups consecutive facts for one aggregate; the celld
object host sends them as one POST {kind}/{shard}/__batch, which
serve_cell runs entry by entry through the idempotent wait path (own
command id, principal and stamp) before one outbox drain. The zone
deletes settled facts from its outbox in one batched delete.
A batch stops at its first retryable failure and is retried from there,
attempts counting per fact, so order and exactly-once are unchanged.
Game code is unchanged.
[[zone-fact-batches]] [[world-scale-game-services]]
Every entity state is now a shared revision. Each changed entity's update record is encoded once per patch from its previous state, and clients with an older baseline share one record per baseline. Per-client work is selection and copy, and the bytes stay identical (a randomized test checks this against the old per-client path). Distance-based update rates (InterestConfig::update_rates, on by default): a known entity updates every patch within 0.5x the view radius, every 2nd patch up to 0.8x, and every 4th beyond, staggered by id. Creates, removes, globals and #[replicated(.., every_patch)] kinds are never held. [[replication-encode-once-lod]]
A sink that commits a batch one by one delivered no settlement until the whole batch was done (16 facts behind slow storage held back every settlement). FactSink::max_batch (default 1) and FactHost::max_fact_batch let a sink declare batching; the celld object host does, other sinks get one fact at a time as before. [[zone-fact-batches]]
Agents get one skill for the game layer: plain Rust zones, facts settled into cells (batched on celld), interest-managed replication with update rates, and zones watching world cells as pushed read models. [[world-scale-game-services]]
A watched neighbour room gave each observer the whole room from its
centre, so interest management stopped at the border. Clients now send
Focus {x, y} (tag 0x07, observers only) with the player's position in
the neighbour's coordinates; Zone::on_observer_focus recentres that
observer's view. Mob Pit applies VIEW_RADIUS, so interest and update
rates continue across rooms. JS: ZoneWatch.setFocus, throttled and
resent after reconnect.
[[observer-focus-neighbour-interest]]
A sink opted into batches by overriding both `FactSink::max_batch` and `FactSink::commit_batch` (a host: `max_fact_batch` and `dispatch_fact_batch`). Overriding only the size once batched a sink through the default one-by-one `commit_batch`, which held settlements back until the whole batch committed. Now one override opts in and supplies the batch commit: `FactSink::batches() -> Option<&dyn FactBatchSink>` and `FactHost::fact_batches() -> Option<&dyn FactBatchHost>`, both `None` by default. The feed batches only a sink that returns one (up to `FactFeed::with_max_batch` / `ZONE_FACT_BATCH`); the rest get one fact at a time. `DurableFacts`, `CommandFactSink`, `CelldObjectHost` and the test sinks moved over; behaviour is unchanged. New test: a sink without batches settles each fact as it commits. [[zone-fact-batches]]
mobpit-recovery wakes N rooms with one bot each, waits for (or runs) the kill, probes every room with real joins and reports when each answered (p50/p90/max, per 10 s), then each room's ticks over a hold. Bots record when their Welcome came (joined_after). Rooms report a wake trace in /info: when the object was built, its first call, and the awaited storage spans of its first wake and join. [[celld-fleet-recovery-hops]]
`AggregateCell::dispatch_idempotent_batch` runs commands in order, each exactly as `dispatch_idempotent` (own ledger entry, identity, session), under one command guard and one alarm reschedule instead of one per command. A rejected command leaves the earlier ones committed and the later ones run; the first retryable result (in progress, 408, 429, 5xx) ends the batch; one result per command attempted. The `__batch` wire moves to `distributed_game::cell::run_fact_batch` (host-testable); `serve_cell` parses, calls it and drains the outbox once. An entry that does not parse is answered in its place. Each entry is still its own transaction: a handler runs between its reservation and its commit, outside the synchronous cell SQL transaction, and must see the entries before it committed. Tests: a batch runs in order and replays as a whole; a rejected entry leaves the others committed; a replayed `__batch` body answers only replays and credits once. [[zone-fact-batches]]
Mob Pit's crawler and presence cells used their own router, which took `__batch` for a command name and answered 400. The object host only fell back to one-by-one on 404, so it read that 400 as the first fact's permanent refusal: a batch of kills for one crawler dropped its first kill. The cells now use `serve_cell`, and the object host sends the facts one by one whenever the batch request as a whole gets a 4xx other than 408/429, so each fact gets its own answer. [[zone-fact-batches]]
Plain integer counts that do not move with machine load, so A/B runs of
one build compare exactly: WorkCounters (records encoded per patch,
creates/updates sent, updates held by rates, Delta bytes, fact commands,
cell and __batch requests, outbox deletes, world reads and pushes) in
GET /zones/{zone}/info as `work`, with the switches in effect; watched
cells answer GET /{kind}/{shard}/__watchers with WatcherStats.
New A/B switches: ZONE_UPDATE_RATES=0, ZONE_WORLD_PUSH=0. The zone-side
reconcile schedule moves into world::watch::WorldSync, with a fake-clock
test that a reconcile repairs missed pushes within one period.
[[framework-work-counters]] [[zone-world-views-pushed]]
[[replication-encode-once-lod]] [[zone-fact-batches]]
Each step prints the rooms' and watched cells' work counters normalised per room patch, room-second or room-minute; --ab VAR=VAL with --pairs N alternates fresh-celld runs (A,B then B,A) and prints medians per side and of the per-pair change. Skill and README "Checking your work". [[framework-work-counters]] [[mobpit-zone-load-test]]
[[zone-fact-batches]]
Fact rates follow the fighting, which moves with machine load; the ratios are what a fact-batch A/B compares. [[framework-work-counters]] [[zone-fact-batches]]
Hello offered only version 1 and decoders reject trailing bytes, so Welcome could not grow and a new client message (Focus) made an old celld zone close the connection: servers had to deploy first. Servers (native net, celld zone object) now answer v1 or v2 in the client's version (ZoneHost::speaking) and refuse others with VERSION_MISMATCH "protocol ..." before the ticket is touched. Welcome v2 appends runtime features (bit 0 FOCUS) and the zone's world origin (Zone::origin). The JS SDK offers 2, resends the same Hello once with 1 to a v1-only server (remembered per URL, not surfaced as an error) and sends Focus only when FOCUS is announced. Golden fixtures cover v1 and v2; PROTOCOL.md gains a negotiation section. Game surfaces keep protocolVersion 1 (payload encodings are unchanged). [[zone-protocol-v2-negotiation]]
Each game converted the player's position into every neighbour's coordinates and called setFocus, and Mob Pit overrode on_observer_focus only to use its view radius. Zones now declare their world origin and player view radius once (Zone::origin, Zone::view_radius); the default on_observer_focus gives a focused observer that radius. The JS GameClient takes the player's position with one call (setPosition(x, y, zoneId?), local to a zone or world) and focuses every observed zone with an origin itself, also after its Welcome; setFocus stays for unplaced zones. Mob Pit declares both and drops its override; its client and the SvelteKit loop drop their coordinate math. [[zone-placement-auto-focus]]
[[zone-protocol-v2-negotiation]] [[zone-placement-auto-focus]]
celld holds a room's frames until the writes issued before them are durable. A wake wrote the zone id, the status and the activity fact before the first player's Welcome (and the join wrote the status again), so under a recovery burst Welcomes waited on the bucket. Now the wake only reads; the zone id is written once per zone, the status only when it changed and in the same batch as the facts, after the Welcome. Own celld node on the Hops RustFS, 150 fresh rooms: wake joins 89 -> 150 of 150 in the ramp; after a SIGKILL p50 38 -> 16 s, p90 44 -> 31 s; 2709 -> 2157 txns shipped in recovery. [[celld-fleet-recovery-hops]]
Probes and the staying bot reuse the room's principal (a reconnecting player, not new Crawler and Presence cells per attempt), send the game's content version, log why a wake bot was lost and count rooms that lose their player again; t0 is when the socket dropped. [[celld-fleet-recovery-hops]]
…lcome [[celld-fleet-recovery-hops]]
[[framework-work-counters]] [[zone-world-views-pushed]] # Conflicts: # distributed_game/src/celld_zone/object.rs
The watched world views (f93b24a) added the `hood` and `borough` zone messages; the generated client was not regenerated, so CI's `game-client --check` failed in the game demos smoke job. [[zone-world-views-pushed]]
Hood, borough and city are separate watched cells, so a waking room's views arrive independently. The test waited for the hood tier only and then asserted the borough count and city banner at once (failed in CI and 2 of 3 local runs). Wait for each, and retry a polled tiers read that fails while celld is busy. [[zone-world-views-pushed]]
Same hood/borough messages as e487715; the SvelteKit copy of the generated client was stale, so `npm run check:games` failed in CI. [[zone-world-views-pushed]]
One tick of walking is exactly the 0.25 m bound (5 m/s over a frame of up to 50 ms); crossing into the next room's coordinates adds f32 rounding, so CI saw 0.251 m (also on #291). Bound 0.26 m: a real doorway jump is a whole room or a visible snap.
A cell pushed each view once and logged only transport errors; a busy zone's 5xx/429 went unnoticed, so it caught up only at its next reconcile (45 ± 15 s). That is too late for a content rollout's freeze (the content headless test timed out waiting for both rooms frozen in CI). Retry after 0.25, 1 and 3 s on transport errors, 5xx and 429; a zone answers every parsed push with 204, so other 4xx are not retried. [[zone-world-views-pushed]]
patrickleet
marked this pull request as ready for review
October 24, 2026 23:34
patrickleet
merged commit Oct 24, 2026
e8c5ec0
into
fix/celld-rooms-long-lived-state
23 checks passed
4 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #290.
What
BindingKind::Zones { origin }/NativeBinding::Zones): routes a prefix such as/zonesto a zone host (celld's public listener). Path forwarded unchanged, WebSocket upgrades always allowed, no credential forwarded (zones verify their own join tickets). An origin equal to the gateway's public origin is rejected.game-demos mobpitmountsNativeGatewayin front of its routes:/zones→--zones-origin(default--celld-url),/internal+/dev→ services,/→--ui-originwhen set.--public-originfor running behind an ingress. Upgrade lifetime 24 h (the 1 h default would cut room sockets)./rooms→/zonesfor room connections (Mob Pit worker, client, SvelteKit page, smoke test, the celld zone test worker and docs)./zones/{zone}is already the nativenetdriver's route.run.shproxy game traffic to the gateway only; headless tests reach rooms through it.mobpit-systemEnvironment:CelldStack(RustFS) inmobpit-celld; celld pods carry Istio sidecars.cargo runfrom the mounted checkout, target dir on a PVC) and Vite; a deployer that runscelld deploywhenevercells/buildchanges; an HTTPRoute on the IstioplatformGateway at<checkout>.gitkb.localhost../run.sh buildruns the host builds the setup needs.CelldQueueHttpPublisher::new_meshallows HTTP only to a cluster Service (*.svc.cluster.local/*.svc); the default stays HTTPS-only. The Mob Pit worker opts in withCELLD_QUEUE_RELAY_MESH=1; Istio sidecar mTLS carries celld → services. Ambient isn't used: kind-hops' Dory kernel has no IPv6 policy routing, so istio-cni can't enroll ambient pods.Tests
cargo test --features gateway-native --test gateway_ui(2 new tests) + routing/mounts/extensions/auth suites;gateway-workerwasm check.cargo test -p distributed_game --all-features: 177 passed (rebuildtests/celld/zone-workerwithworker-build --releasefirst).examples/game/server:cargo testpasses.tsc --noEmitclean.cargo test --lib --features workers-rs celld_queue: 8 passed (new mesh-policy test); the cells worker builds to wasm withnew_mesh.https://distributed-mobpit-world.gitkb.localhostjoins in 137 ms, ~30 ticks/s, RTT 5–31 ms; room infodroppedTicks: 0, late p99 5 ms. Istio metrics on the Mob Pit sidecar: celld →POST /internal/celld-queue/relay202 withconnection_security_policy="mutual_tls".EIOon kind-hops (node too, any repo), so Vite answers some page loads with 500. The deployer now skips a round on read errors instead of exiting.