Polish: honest consistency docs, hardened failure demo, join/leave + CI

.github/workflows/ci.yml

@@ -0,0 +1,27 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: "1.22"
+      - name: format
+        run: test -z "$(gofmt -l cmd internal)"
+      - name: vet
+        run: go vet ./...
+      - name: test
+        run: go test ./... -race -count=1
+      - name: failure + rebalance demos
+        run: |
+          ./scripts/run-cluster.sh
+          ./scripts/failure-demo.sh
+          ./scripts/rebalance-demo.sh
+          ./scripts/stop-cluster.sh

Makefile

@@ -1,4 +1,4 @@
-.PHONY: build test race bench cluster stop demo fmt
+.PHONY: build test race bench cluster stop demo rebalance quickstart fmt ci
 
 build:
 	mkdir -p bin
@@ -23,5 +23,16 @@ stop:
 bench: build
 	./scripts/bench.sh
 
+quickstart:
+	./scripts/quickstart.sh
+
 demo: build
 	./scripts/failure-demo.sh
+
+rebalance: build
+	./scripts/rebalance-demo.sh
+
+ci:
+	test -z "$$(gofmt -l cmd internal)"
+	go vet ./...
+	go test ./... -race -count=1

README.md

@@ -1,8 +1,10 @@
 # ringcache
 
-Distributed **in-memory** cache. One Go process per node, a consistent-hash ring with virtual nodes, per-node LRU+TTL, and synchronous replication (`R≥2`).
+Distributed **in-memory** cache. One Go process per node, a consistent-hash ring with virtual nodes, per-node LRU+TTL, and synchronous replication (`R=2`).
 
-This is a systems MVP, not a product: no persistence, no auth, no dynamic membership, no linearizability. Numbers below were **measured** on this machine, not invented.
+**Consistency in one line:** a SET is successful when `acked ≥ 1`. That is **not** linearizable, **not** a quorum, and **not** durable. Replication is R=2 synchronous fan-out; last-writer-wins on coordinator `written_at`; nothing is written to disk.
+
+This is a systems MVP, not a product: no auth, no gossip membership, no WAL. Numbers below were **measured** on this machine, not invented.
 
 ```mermaid
 flowchart LR
@@ -30,23 +32,38 @@ A client talks to **any** node. That node is the coordinator: it hashes the key,
 
 ---
 
-## Run
+## 60-second run
 
-### Docker Compose (3 nodes)
+Needs Go 1.22+ and `curl`. Docker is optional.
 
 ```bash
-docker compose up --build
+git clone https://github.com/hiroshi-os/ringcache && cd ringcache
+./scripts/quickstart.sh
 ```
 
-Host ports: `node-a :8080`, `node-b :8081`, `node-c :8082`. Compose uses `network_mode: host` so the three processes can replicate over loopback (a Docker bridge on this nested CI VM drops 100% of veth-to-veth packets). On Docker Desktop (Mac/Windows), use `make cluster` instead.
+That builds `bin/ringcache`, starts **node-a/b/c** on `127.0.0.1:8080-8082`, SETs `user:1`, and GETs it from another coordinator.
 
-### Local processes (no Docker)
+Same thing by hand:
 
 ```bash
 make cluster          # 127.0.0.1:8080-8082
-# make stop           # tear down
+curl -sS -X PUT http://127.0.0.1:8080/v1/set \
+  -H 'Content-Type: application/json' \
+  -d '{"key":"user:1","value":"ok","ttl_ms":60000}'
+curl -sS 'http://127.0.0.1:8081/v1/get?key=user:1'
+./scripts/failure-demo.sh     # A: kill replica, still hit; B: partial ACK then miss
+./scripts/rebalance-demo.sh   # join/leave remaps ~1/N primaries
+make stop
 ```
 
+### Docker Compose (optional)
+
+```bash
+docker compose up --build
+```
+
+Host ports: `node-a :8080`, `node-b :8081`, `node-c :8082`. Compose uses `network_mode: host` so the three processes can replicate over loopback (a Docker bridge on this nested CI VM drops 100% of veth-to-veth packets). On Docker Desktop (Mac/Windows), host networking is ignored — use `make cluster`.
+
 Single node (self is implied if `-peers` is empty):
 
 ```bash
@@ -60,7 +77,7 @@ go run ./cmd/ringcache -id node-a -listen :8080
 JSON, no auth. `ttl_ms: 0` or omitted means **no expiry**.
 
 ```bash
-# SET
+# SET — HTTP 200 means acked≥1, not “all replicas applied”
 curl -sS -X PUT http://127.0.0.1:8080/v1/set \
   -H 'Content-Type: application/json' \
   -d '{"key":"user:1","value":"ok","ttl_ms":60000}'
@@ -83,9 +100,10 @@ Aliases without the `/v1` prefix (`/get`, `/set`, `/delete`) do the same thing.
 | Method | Path | Notes |
 | --- | --- | --- |
 | `GET` | `/v1/get?key=` | 200 hit, 404 miss, 503 no replica reachable |
-| `PUT`/`POST` | `/v1/set` | body `{key,value,ttl_ms}` — 200 if `acked≥1`, else 503 |
+| `PUT`/`POST` | `/v1/set` | body `{key,value,ttl_ms}` — **200 if `acked≥1`**, else 503 |
 | `DELETE` | `/v1/delete?key=` | best-effort delete on all owners |
 | `GET` | `/internal/kv` | local store only (replication path) |
+| `GET`/`POST`/`DELETE` | `/admin/members` | local ring membership only — **not gossip** |
 
 SET response (honest about partial writes):
 
@@ -107,29 +125,34 @@ hash32(id + "#" + i)     // FNV-1a 32-bit, i in [0, V)
 
 A key walks **clockwise** from `hash32(key)` and collects the first `R` **distinct** physical nodes. Those are the owners.
 
-Why virtual nodes: one point per node makes arcs (and therefore key load) wildly uneven. `V=150` chops the circle so each node owns many small arcs. Adding a node remaps about `1/N` of primaries, not a whole neighbor's slice. Tests in `internal/ring` check determinism, uniqueness, spread, and join remap fraction.
+Why virtual nodes: one point per node makes arcs (and therefore key load) wildly uneven. `V=150` chops the circle so each node owns many small arcs. Adding a node remaps about `1/N` of primaries, not a whole neighbor's slice. Tests in `internal/ring` check determinism, uniqueness, spread, join remap, and leave remap. Live join/leave: `./scripts/rebalance-demo.sh`.
 
 ---
 
-## Replication and consistency (read this)
+## Consistency model
 
-**Default `R=2`.** SET is **synchronous fan-out** to all `R` owners. The coordinator returns as soon as those RPCs finish (or time out, default 200ms). Success means `acked ≥ 1`, **not** a quorum of `R`.
+Read this before citing the demo as “HA cache.”
 
-| What we are | What we are not |
+| What a 200 SET means | What it does not mean |
 | --- | --- |
-| Last-writer-wins on coordinator `written_at` (unix-nano) | Linearizable / sequential |
-| Best-effort sync replication | Quorum R+W>N |
-| Read: first live owner with a non-expired value | Read-your-writes from an arbitrary node |
-| Opportunistic read-repair of other owners | Anti-entropy / hinted handoff |
+| ≥1 of the R owners applied the write | Linearizable or sequential consistency |
+| The coordinator finished (or timed out) a **sync fan-out to all R owners** | Quorum `R+W>N`, or “the cluster has the key” |
+| Last-writer-wins on coordinator `written_at` (unix-nano) | Compare-and-swap / transactions |
+| GET returns the first live owner with a non-expired value | Read-your-writes from an arbitrary node |
+| Opportunistic read-repair of other owners | Anti-entropy, hinted handoff, or key migration on join |
 | 2s fail-open breaker after a peer error | Failure detection / membership |
 
+**Defaults:** `R=2`, replica RPC timeout 200ms. The coordinator **always waits for every owner RPC** (or timeout). HTTP 200 is `acked ≥ 1`, **not** `acked == R`.
+
+**There is no persistence.** Restart = empty. No WAL, snapshot, or disk.
+
 **Failure modes that actually happen:**
 
-1. SET acks 1 of 2. Kill that replica → GET misses even though the cluster is “up”.
+1. SET acks 1 of 2. Kill that replica → GET misses even though another node is “up.”
 2. Two coordinators SET the same key concurrently → replicas can diverge until the next write or read-repair.
 3. TTL is computed on the receiving node (`now + ttl_ms`). Clock skew moves expiry.
-4. Membership is **static** (compose/env). A dead node stays on the ring; peers skip it briefly, then retry and eat the timeout.
-5. Restart = empty. There is no WAL, snapshot, or disk.
+4. Membership is **per process**. A dead node stays on the ring until someone `DELETE /admin/members`. Peers skip it briefly, then retry and eat the timeout.
+5. Join/leave remaps ~`1/N` primaries. The newly responsible node starts **empty** — we do not move values.
 
 Caches hide this with TTL. Do not put a source of truth here.
 
@@ -147,16 +170,27 @@ Caches hide this with TTL. Do not put a source of truth here.
 ## Failure demo
 
 ```bash
-# local 3-process cluster
-./scripts/failure-demo.sh
-
-# or, with compose already up:
-./scripts/failure-demo.sh --compose
+./scripts/failure-demo.sh            # local 3-process cluster
+./scripts/failure-demo.sh --compose  # compose already up, or starts it
 ```
 
-What it does: pick a key whose owners include **node-b**, SET it via node-a, GET all nodes, **kill node-b**, GET node-a and node-c.
+The key picker reads **`owners[]` only**. Grepping the whole `/ring` body matches `"node-b"` in the cluster node list and is wrong.
+
+**Scenario A — full ACK, kill one replica (expect HIT).** Pick a key whose `owners[]` include node-b. SET via node-a. Kill node-b. GET on :8080 and :8082 still hits the surviving owner. `:8081` stops accepting connections. This is **not** “the cluster heals”; the dead id remains on the ring.
+
+**Scenario B — partial ACK, then kill the only copy (expect MISS).** Kill node-b first. SET a key owned by `{node-a, node-b}` → `acked=1`, `failed=["node-b"]`, HTTP 200. Kill node-a. GET via node-c misses (both owners down). Success is `acked≥1`, not durability.
 
-**Real result on this tree (2026-09-13):** script picked `demo:failure:1` with owners `[node-b, node-a]`. SET acked 2/2. Before the kill, `:8081` served from `node-b`. After `kill` of node-b, GET on :8080 and :8082 still returned `"found":true` with `served_by: node-a`. `:8081` stopped accepting connections. This is **not** “the cluster heals”; the dead id remains on the ring. If SET had only acked node-b, the same kill would miss.
+**Real result on this tree (2026-09-13):** A picked `demo:failure-a:1` owners `[node-b, node-a]`, SET acked 2/2; after kill, :8080 and :8082 served `node-a`. B picked `demo:failure-b:1` owners `[node-b, node-a]`; SET while node-b down acked 1 (`failed=["node-b"]`); after killing node-a, GET :8082 returned **503** `all replicas unreachable`.
+
+---
+
+## Join / leave rebalance
+
+```bash
+./scripts/rebalance-demo.sh
+```
+
+Starts **node-d** on `:8083` and `POST /admin/members` on node-a/b/c (membership is local, not gossip). Samples 2000 primaries via `/ring?key=` before and after; expects ~`1/4` remapped (`N=3→4`). Leave restores the 3-node placement. Newly responsible nodes do **not** receive old values.
 
 ---
 
@@ -172,31 +206,34 @@ Harness: `cmd/bench` — real HTTP PUT `/v1/set` and GET `/v1/get` against all t
 
 | | |
 | --- | --- |
-| Date (UTC) | 2026-09-13T09:29:48Z |
+| Date (UTC) | 2026-09-13T10:55:13Z |
 | Hardware | Linux 6.12.94+ x86_64, 4 vCPU, Intel Xeon, ~16 GiB RAM (Cursor Cloud Agent VM) |
 | Go | go1.22.2 linux/amd64 |
 | Cluster | 3 processes on loopback (`127.0.0.1:8080-8082`), `R=2`, `V=150`, cap 10000 |
 | Load | `cmd/bench -n 4000 -c 32 -keys 1000` (real HTTP, not in-process) |
 
 | Phase | ok | errors | wall | ops/s | p50 | p95 | p99 | max | mean |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| SET | 4000 | 0 | 312ms | **12820** | 2.28ms | 4.76ms | 6.56ms | 10.18ms | 2.48ms |
-| GET | 4000 | 0 | 129ms | **31027** | 0.82ms | 2.68ms | 3.91ms | 5.84ms | 1.02ms |
-
-These are **this VM, this commit, loopback**. Docker NAT, a laptop, or a noisy neighbor will differ. Do not cite them as product SLOs.
+| SET | 4000 | 0 | 426ms | **9395** | 3.08ms | 6.72ms | 8.90ms | 18.17ms | 3.40ms |
+| GET | 4000 | 0 | 378ms | **10583** | 2.33ms | 7.94ms | 11.31ms | 19.45ms | 3.01ms |
 
-`docker compose up --build` was also run on this VM (host network). SET acked 2/2; after `compose stop node-b` on a key owned by node-b, GET on :8080 and :8082 still hit `node-a`.
+These are **this VM, this commit, loopback**. A quieter slot on the same VM earlier today printed ~12.8k SET / ~31k GET ops/s — same harness, same flags. Shared-CPU noise is real; do not cite either row as a product SLO. Re-run `./scripts/bench.sh` and replace the table for your machine.
 
 ---
 
 ## Tests
 
 ```bash
 go test ./... -race -count=1
+# or
+make ci
 ```
 
-- `internal/ring` — deterministic owners, unique replicas, vnode count, key spread, join remaps ~1/N
+- `internal/ring` — deterministic owners, unique replicas, vnode count, key spread, join/leave remap ~1/N
 - `internal/store` — LRU order, TTL expiry, LWW stale-write ignore, capacity-1 eviction
+- `internal/node` — 3-node `httptest` SET/GET/DELETE, partial ACK when an owner is down, join/leave
+
+CI (GitHub Actions) runs `gofmt`, `go vet`, `-race` tests, then `failure-demo.sh` and `rebalance-demo.sh` against `make cluster`.
 
 ---
 
@@ -230,5 +267,5 @@ go test ./... -race -count=1
 
 > **DRAFT** — do not paste onto a résumé until you have run the demo yourself and can defend every number and failure mode.
 
-- **DRAFT.** Built a 3-node Go in-memory cache with consistent hashing (`V` virtual nodes, FNV-1a), per-node LRU+TTL, and `R=2` synchronous fan-out; coordinators report `acked`/`failed` instead of pretending writes are atomic.
-- **DRAFT.** Measured real HTTP SET/GET throughput and p99 on a 3-node loopback cluster and documented single-node-kill behavior: reads continue only when at least one surviving replica acked the write (last-writer-wins, no quorum).
+- **DRAFT.** Built a 3-node Go in-memory cache with consistent hashing (`V` virtual nodes, FNV-1a), per-node LRU+TTL, and `R=2` synchronous fan-out; coordinators report `acked`/`failed` and treat success as `acked≥1` (not linearizable, not a quorum, no persistence).
+- **DRAFT.** Measured real HTTP SET/GET throughput and p99 on a 3-node loopback cluster and documented single-replica-kill vs partial-ACK-then-kill: reads continue only when a surviving replica acked the write (last-writer-wins).

cmd/ringcache/main.go

@@ -38,6 +38,9 @@ func main() {
 		Addr:              cfg.Listen,
 		Handler:           n.Handler(),
 		ReadHeaderTimeout: 5 * time.Second,
+		ReadTimeout:       15 * time.Second,
+		WriteTimeout:      15 * time.Second,
+		IdleTimeout:       60 * time.Second,
 	}
 
 	errCh := make(chan error, 1)
@@ -138,7 +141,8 @@ func runHealthcheck(listen string) error {
 	if !strings.HasPrefix(port, ":") {
 		port = ":8080"
 	}
-	resp, err := http.Get("http://127.0.0.1" + port + "/health")
+	client := &http.Client{Timeout: 2 * time.Second}
+	resp, err := client.Get("http://127.0.0.1" + port + "/health")
 	if err != nil {
 		return err
 	}