crash-loop

A 2D infrastructure & automation puzzle. You are a Site Reliability Engineer at a fictional cloud provider: each level hands you a failing distributed system, and you compose a topology of nodes — load balancers, services, caches, queues — that survives simulated traffic inside an error budget and a resource budget. No physics, no twitch, no 3D. Backend over bullets.

Built by Team Three-Way Merge (Gabriel Felipe Guarnieri · Hector Guarçoni Machado · Marcos Winícios Silva Martins) for Software Engineering for Games — 2026/1.

Run it

▶ Play it live: https://oguarni.github.io/crash-loop/ — deployed from main by GitHub Actions on every push (no install required).

To run it locally:

npm install
npm run dev      # vite dev server, opens the playable build

Other scripts:

npm run build      # type-check (tsc --noEmit) + production bundle into dist/
npm run preview    # serve the production build locally
npm run typecheck  # type-check only
npm run test       # full Vitest suite (sim, board rules, progress)
npm run coverage   # Vitest with v8 coverage thresholds (>=80 / >=95 core)
npm run test:sim   # headless deterministic sim-check smoke harness (L01–L06)

L01 — "boot" (the first playable level)

svc-cart takes public traffic on a single replica and folds under load. Incoming traffic is 30 req/tick; one service handles 10.

• ingress is a single public entry point — it can only feed one downstream node, so you must route it through a load-balancer to fan traffic out.
• The load-balancer splits its inflow evenly across every downstream service.
• Budget caps you at $5.00 / 8 CPU / 8 MEM, and dropped requests must stay within the error budget (20).

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            ├──> service
                            └──> service

3 services × 10 = 30 req/tick (zero drops), at $4.50 — which also clears the gold tier (parCost).

Screen flow

The boot/title screen leads into a level-select menu — every level is a card showing its saved best, so a presenter can jump straight into any scenario (click a card, or press its number key 1–7). Inside a level, the rail's < menu affordance or a clean Esc returns to the select screen; you no longer have to clear levels in sequence to reach the later ones.

Controls

Every action is a press — one input model for mouse, finger and pen alike, via pointer events. Keys are shortcuts, never the only way in.

Action	How
Place	Press a component in the left rail, then press an empty spot in the work area
Wire	Select wire, press the source node, then the target node
Move	Select move, press a node to pick it up, then press where it should go. Holding and dragging works too. A node put down on another is nudged to the nearest free slot
Delete	Select delete, press a node or an edge
Run	Run > (or Enter) — simulate the traffic profile and score it
Pause	Pause (or Space / P) freezes a running sim; Resume continues
Help	The rail's ? help affordance (or ? / H) opens the help / legend overlay — controls, the node kinds this level uses, and a way back into the tutorial
Mute	M (or the rail's mute affordance) toggles all audio, including the ambient hum
Fullscreen	F (or the rail's <> full row, or the menu's <> fullscreen row) fills the screen and asks the device for landscape
Cancel / back	Esc unwinds one layer at a time: a carried node goes back, then a pending wire / selection clears, then fullscreen, and only a clean Esc returns to the level menu. Pressing the rail or HUD also abandons a carry

Clear a level and a Next > button appears on the result banner to advance.

Touch and small screens

The board is a fixed 960×600 logical surface that CSS scales to fit, so the same build runs on a phone:

• The move tool is a two-step gesture, not a drag. A press-and-drag is the one gesture a touch screen cannot express — a finger reports no hover, and the browser claims a held press for scrolling — so a press picks the node up and the next press puts it down. Dragging still works for anyone who prefers it, and both paths share one state machine (beginCarry / moveCarried / dropCarried).
• The chrome is tappable, not keyboard-only. The rail carries < menu and ? help rows, and the help overlay carries a how to play button, so a device without Esc, ? or T can still reach every screen.
• touch-action: none on the canvas hands every gesture inside the board to the game: no page pan, no pinch zoom, no double-tap zoom, no long-press callout.
• Coarse pointers get padded hit-boxes (inflate() in src/layout.ts), each pad kept under half the gap to its neighbour so a fat-finger tap can never land on the wrong row — asserted in src/render.smoke.test.ts.
• Phone-sized viewports go full-bleed (the hint line and frame are dropped so every pixel goes to the board), and a portrait nudge asks for landscape, where a 16:10 board is actually readable. It is dismissible.
• Fullscreen, with a landscape lock. Browser chrome comes straight out of a fixed-size board, so F — or the rail's <> full row, the menu's <> fullscreen row, or the portrait nudge's own button — hands that space back and asks the device for landscape. The lock is the part that matters: a browser honours one only for a fullscreen document, so a phone whose rotation is locked to portrait, which "rotate your device" cannot help at all, is turned by this and by nothing else. Everything is best-effort and degrades quietly.
• A phone takes the screen on its first touch, without being asked. That is where the cost of browser chrome is highest and where a control is hardest to find, so the tap that leaves the boot screen — already carrying the user activation the request needs — spends it on fullscreen too. Once only, and never again once the player leaves fullscreen by any route: a game that grabs the screen back on the next tap is arguing with them (src/main.ts).
• A tap fires the toggle on release, not on press. Transient user activation, which requestFullscreen() requires, arrives on pointerdown for a mouse but only on pointerup for a finger — and the board's preventDefault() suppresses the compatibility click that would otherwise carry it. So the press parks the intent and the release spends it (src/main.ts).
• iPhone Safari has no Fullscreen API at all (it lives on <video> only), so the route there is an installed launch: public/manifest.webmanifest declares display: fullscreen and orientation: landscape, and the apple-* meta tags say the same to the iOS versions that read no manifest. Added to the home screen, the game opens with no browser chrome. The controls stay drawn on that phone and open an "add to home screen" sheet instead of toggling, because a player who cannot find a fullscreen control reads it as broken, not as absent — fullscreenRoute() picks between the two, and answers none only for a desktop browser without the API or a launch that is already chrome-free.

Scoring & progress

Each run is graded into a tier — FAIL, PASS (error budget held), or GOLD (also at or under parCost). Beyond the tier, a run is scored on three axes, surfaced side by side on the result banner and carried as saved bests in the rail:

• cost — the dollar total of the topology (always live);
• cycles — request-ticks spent waiting in a buffer, so it only matters where a queue exists; other levels show a clean — rather than a misleading 0;
• coverage — the share of traffic that passed through a CI gate, live only where a gate is present (or required).

The best tier and the best of each relevant axis are kept per level in localStorage, so a cleared scenario shows its saved bests on return, and a run that beats a record flags a NEW BEST. Verdicts are never signalled by colour alone — a tier always pairs with a word (PASS / FAIL / GOLD). The title screen reports how many regions you've stabilised. Scoring is meta state only — it never feeds the deterministic simulation.

L02 — "first deploy"

A new svc-cart release cuts over to production, but every request must clear a canary deploy gate first. Traffic rises to 40 req/tick.

• requireBeforeSinks: ['gate'] — every path from ingress to a service must pass through a ci-gate, or the run is rejected ("untested traffic reached production"). It's a topology rule, checked before the traffic ever flows.
• A ci-gate forwards only 20 req/tick, so one gate throttles production — you need two, fanned out from the load-balancer.
• Budget caps you at $8.00 / 8 CPU / 8 MEM; the error budget is 40.

The dominant correct topology:

ingress ──> load-balancer ──┬──> ci-gate ──┬──> service
                            │              └──> service
                            └──> ci-gate ──┬──> service
                                           └──> service

lb splits 40 → 20/20 to the gates → 10/10 to four services (cap 10): zero drops, at $7.50 — which also clears the gold tier. A third gate or fifth service would breach the $8.00 budget, so this build is the unique solution.

L03 — "flapping cart"

svc-cart is flapping under a flood of repeated reads, and adding replicas is priced out. This level introduces the cache node: it serves a fixed fraction of its inflow locally (a cache hit) and forwards only the misses downstream — and, like a load-balancer, it splits those misses evenly.

• A cache has hitRate: 0.5: of 40 req/tick it serves 20 as hits and forwards 20 as misses. It's cheap in cost but heavy in memory (mem 2).
• Budget caps you at $4.50 / 6 CPU / 6 MEM; the error budget is 40.

The dominant correct topology:

ingress ──> cache ──┬──> service
                    └──> service

The cache serves 20 locally and forwards 20 → 10/10 into two services: zero drops, at $3.00 (gold). The cacheless brute force (lb + 4 services = $5.50) is over budget, so you can't out-spend the problem — you have to cache. Chaining a second cache does not help: it receives only the first one's misses — the reads that are not repeated — so it serves nothing and forwards all 20 into a replica that caps at 10.

L04 — "error budget"

This level flips the lesson: until now the goal was zero drops; here serving everything is deliberately unaffordable. Traffic holds at a steady 20 req/tick, spikes to 40 for five ticks, then settles back.

• Budget caps you at $5.00 / 6 CPU / 6 MEM; the error budget is 120.
• The zero-drop build (lb + 4 services = $5.50) is over budget — you cannot buy your way out of the spike.

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            └──> service

Two services (cap 20) serve the steady 20 with zero drops; during the spike they shed 100 requests total, which sits inside the 120 error budget — at $3.50 (gold). A safer lb + 3 services ($4.50) passes with only 50 drops but misses gold. The lesson: spend the error budget instead of overspending on capacity.

L05 — "chaos friday"

It's Friday and chaos is loose: replicas crash mid-run, without warning. A seeded schedule knocks out one service at a time (two incidents, five ticks each), and while a replica is down its capacity is 0 — everything routed to it is dropped. Traffic is a steady 20 req/tick.

• Budget caps you at $7.00 / 8 CPU / 8 MEM; the error budget is 55.
• The incident schedule lives entirely in a per-level seed — you can't see the exact timing, so you build for the failure, not around it.

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            ├──> service
                            ├──> service
                            └──> service

The lesson is resilience through redundancy. The load-balancer splits 20 evenly, so with four services each carries only 5 req/tick; when one crashes, only its 5/tick are shed — across two 5-tick incidents that's 50 dropped, inside the 55 error budget, at $5.50 (gold). Two or three services carry a bigger share (10 or ~7 per replica), so losing one blows the budget. Because the gold build is symmetric, which replica the seed picks never changes the outcome — so the run stays fully deterministic.

L06 — "back-pressure"

The counterpoint to L04: instead of dropping a spike, you buffer it. This level introduces the queue — the one stateful node, whose buffer carries across ticks. Traffic sits at a steady 10 req/tick, spikes to 40 for five ticks, then settles back, leaving room to drain.

• A queue drains up to 20 req/tick and holds up to 100 across ticks; when the buffer is full it sheds the overflow (back-pressure). Like a cache, it splits its released traffic evenly across downstream edges.
• Budget caps you at $5.00 / 6 CPU / 6 MEM; the error budget is 20.
• Peak provisioning (lb + 4 services = $5.50) is over budget — you must buffer, not out-spend, the spike.

The dominant correct topology:

ingress ──> queue ──┬──> service
                    └──> service

The queue releases 20/tick and buffers the surplus (peaking at exactly 100 during the spike), then drains it over the calm tail. Two services (cap 20) match the drain rate, so nothing is dropped, at $4.00 (gold). A single downstream service can't keep up with the queue's own drain and fails. Requests still buffered when the run ends count as dropped — you must drain in time.

L07 — "black friday"

The finale. It stacks every mechanic the campaign taught into one topology, so all three scoring axes are live at once. Peak read traffic sits at 32 req/tick, bursts to 56 for eight ticks (the Black Friday spike), then eases back over a long recovery tail; two seeded incidents knock a replica out mid-run.

• cache halves the heavy read load, so you provision for the misses, not the full arrival rate — without it the downstream is unaffordable;
• queue soaks the burst and bleeds the backlog off across the quiet ticks (this is the cycles axis — request-ticks spent waiting). It is required on every path to a replica: after the cache halves the load the burst is only 28 req/tick, so a second $1.00 gate would carry it unbuffered — a $7.00 build that beat par on every axis and dissolved the lesson;
• ci-gate is required before every replica, which drives coverage to 100%. Both rules ride on requireBeforeSinks: ['gate', 'queue'] — every path from ingress to a replica must cross each listed kind;
• chaos (as in L05) sheds a replica's share during each incident, so you spread the load across enough replicas to stay inside the error budget.
• Budget caps you at $9.00 / 10 CPU / 12 MEM; the error budget is 52.

The dominant correct topology:

ingress ──> cache ──> queue ──> ci-gate ──┬──> service
                                          ├──> service
                                          ├──> service
                                          └──> service

The cache halves the reads, the queue drains ≤20/tick and holds the burst, the gate forwards ≤20 to four replicas carrying ~4–5 req/tick each — so losing one to an incident sheds only its small share: 45 dropped, inside the 52 error budget, at $8.00 / 768 cycles / 100% coverage (gold). Three replicas shed too big a share on a crash (60 dropped) and fail; a fifth replica clears the drops but breaches the cost par.

Architecture

src/
  types.ts          shared domain types (incl. ChaosSpec)
  palette.ts        canonical Three-Way Merge palette
  layout.ts         geometry constants + hit-testing helpers (incl. touch inflate)
  sim/
    nodes.ts        per-kind specs (cost, capacity, fan-out, cache hit-rate, queue buffer)
    rng.ts          deterministic seeded PRNG (mulberry32) for chaos
    engine.ts       deterministic per-tick topological flow simulation
  levels/
    L01.ts          "boot"          — routing / load balancing
    L02.ts          "first deploy"  — deploy gate rule
    L03.ts          "flapping cart" — cache node
    L04.ts          "error budget"  — traffic spike, tight budget
    L05.ts          "chaos friday"  — seeded incident injection
    L06.ts          "back-pressure" — queue node (cross-tick buffering)
    L07.ts          "black friday"  — finale: cache + queue + gate + chaos
    index.ts        level register (played in order)
  game.ts           board state, editing rules, run/playback (framework-agnostic)
  progress.ts       persistent per-level scoring (localStorage, sim-independent)
  render.ts         all canvas drawing + shared hit-region layouts
  mobile.ts         touch host wiring (gesture guards, portrait nudge, install sheet)
  fullscreen.ts     Fullscreen API + landscape lock + route detection, guarded
  main.ts           DOM wiring, pointer input, the playback loop
scripts/
  sim-check.ts      headless deterministic verification harness (npm run test:sim)

Design notes

• The simulation is fully deterministic — the same topology, traffic profile and seed always produce the same result (a design pillar). The chaos mechanic (L05) is a seeded incident injection, not wall-clock noise: the schedule is a pure function of the level seed, generated once before the tick loop.
• Traffic flows through the graph in topological order each tick; a cycle is rejected as an invalid topology (a real DAG constraint).
• Node behaviour is data-driven: fanOut nodes split evenly, a hitRate node (cache) serves a fraction and forwards the rest, a buffer node (queue) holds traffic across ticks and drains at its capacity, and plain sinks (services) handle up to capacity and drop the overflow.
• The queue is the only stateful node: its buffer persists between ticks. Requests still held when the run ends are counted as dropped, so conservation (served + dropped === arrived) always holds and a solution must drain in time.
• game.ts holds no rendering or DOM code, so the rules are unit-testable and the renderer is replaceable. That includes the move gesture: beginCarry / moveCarried / dropCarried / cancelCarry live in the model, so main.ts only has to decide when a press means pick-up, drop, or drag — and the gesture is covered by unit tests rather than by clicking around a browser.

Roadmap

Shipped: L01 — boot · L02 — first deploy · L03 — flapping cart · L04 — error budget · L05 — chaos friday · L06 — back-pressure · L07 — black friday (finale). All six roadmap node kinds are live: ingress, load-balancer, service, ci-gate, cache, queue. Also shipped:

• Level select — a title-screen menu to jump into any scenario, no longer gated behind clearing levels in order.
• Multi-axis scoring — cost, cycles, and coverage surfaced side by side on the result banner and rail.
• In-game help / legend — a ? / H overlay (or the rail's ? help row) with the controls and the node kinds each level uses.
• Touch support — pointer-event input, the two-step move gesture, tappable chrome, padded coarse-pointer hit-boxes, a full-bleed phone layout and a portrait nudge. The same build plays on a phone in landscape.
• Fullscreen on mobile — entered automatically on a phone's first touch, and an F / tappable toggle that fills the screen and locks landscape, offered from the menu, the rail and the portrait nudge. Where the browser has no Fullscreen API (iPhone Safari) those controls explain the installable-manifest route instead of disappearing.
• Terminal polish — CRT vignette + contrast pass and a low ambient hum; IBM Plex Mono is now self-hosted, so a live demo needs no network at all.

Planned:

• Infrastructure as Code — declare part of a topology from a script/template.
• Narrative & NPCs — diegetic incident briefings and the senior SRE mentor.
• Thematic campaign — group levels into worlds (Ingress, Queues, Services, CI/CD, Data/Cache, SRE panel) with boss scenarios.