Hot-swappable protection config, sustained rate limits, and penalty box

CLAUDE.md

@@ -74,6 +74,11 @@ so the UDS default stays v1.
 ### ArcSwap for RouteTable
 `ArcSwap<RouteTable>` gives us a pointer-swap on writes (single atomic store) and a single `load()` on reads — no lock contention on the hot path. With ≤100 routes, rebuilding the full table on `updateConfig` costs ~microseconds (Arc pointer clones only). A partial-update scheme would be more complex without meaningful benefit.
 
+### ArcSwap for ProtectionConfig (hot-swappable)
+`ProtectionState.config: ArcSwap<ProtectionConfig>` holds *all* protection settings as one swappable snapshot — CIDR allowlist/blocklist, JA3 blocklist, rate limit parameters, concurrency cap, handshake timeout, and requireSni are all inside `ProtectionConfig`. This means one atomic `store()` in `updateConfig` reaches all checks for a listener with zero hot-path cost. `check()` calls `config.load()` once per connection, which is a single lock-free pointer load. Callers identify listeners by port (`JsHotConfig.protection: [{ port, protection }]`).
+
+**Invariant:** protection *presence* (None vs Some) is fixed at listener construction — the `Option<Arc<ProtectionState>>` in `ListenerState` never changes after `start()`. `update_config` returns an error for ports that match no listener or that were started without protection. The `symphony-server` reconcile path enforces this by including `hasProtection` in the per-listener signature: none→some and some→none transitions change the signature and trigger a seamless proxy recreate (SO_REUSEPORT, no bind gap); only contents-only changes reach the hot-swap path.
+
 ### DashMap throughout
 `DashMap` provides lock-free concurrent access via internal sharding. Used for:
 - `protection.rs`: per-IP state (`ip_table: DashMap<IpAddr, Arc<IpState>>`)
@@ -83,6 +88,23 @@ so the UDS default stays v1.
 ### Token bucket via AtomicU32 CAS (×1000 fixed-point)
 The rate limit uses a fixed-point token count (×1000) in an `AtomicU32` with CAS retry loops. `Relaxed` ordering is correct here because the token bucket is inherently approximate — a small window of double-allowing at refill time is acceptable and expected. No mutex needed on the hot path.
 
+Each IP now has **two independent token buckets** on `IpState`: a per-second bucket (`tokens`, `last_refill_ns`) and a sustained per-minute bucket (`sustained_tokens`, `sustained_last_refill_ns`). Both use the same ×1000 fixed-point and CAS idiom. Both are checked on admission; exhausting either blocks the connection. Max sustained burst: 4,294,967 connections (u32::MAX / 1000) — far above any realistic value.
+
+### Monotonic clock for all timing
+`now_ns()` returns a monotonic offset from a process-wide `Instant` anchor (`static START: OnceLock<Instant>`), NOT wall-clock nanoseconds. All penalty deadlines and bucket timestamps use this value. Trade-off: values cannot be interpreted as unix timestamps; they are only ever compared internally. Benefit: forward NTP steps cannot release a penalty-boxed IP early and backward steps cannot freeze bucket refills.
+
+### Penalty box via AtomicU64 deadline
+When `penaltyBox.durationMs > 0`, exhausting any rate limit sets `IpState.penalty_deadline_ns = now_ns() + duration_ns`. While `now < deadline`, connections are blocked as `PenaltyBoxed`. Each blocked attempt while boxed also debits the token buckets (lazy refill + consume); if a bucket exhausts, the deadline is reset to `now + duration_ns` (extension). The IP is readmitted once the deadline passes without further extension. Penalty state lives on `IpState` and survives config hot-swaps. A `durationMs` hot-swap affects new stamps only; existing deadlines run out on the old duration.
+
+### IP state eviction (spawned in start())
+`ProtectionState::evict()` is spawned as a periodic background task (60 s interval) per listener with protection, in `start()` via the `shutdown_tx` broadcast pattern. Eviction uses **lazy bucket projection** — it computes what the token level *would* be if refilled by the current time (`now - last_refill_ns`), rather than relying on the stored token value (which is only updated on access). An entry is retained if:
+- It is in the penalty box (`penalty_deadline_ns > now`).
+- It has active connections (`active > 0`).
+- Its per-second bucket would not yet be fully refilled (projected tokens < burst_fp).
+- Its sustained bucket would not yet be fully refilled (projected sustained_tokens < sustained_burst_fp).
+
+This prevents an attacker from resetting their sustained window by pausing long enough for the eviction interval to pass.
+
 ### SO_REUSEPORT per worker
 Each tokio worker thread gets its own listening socket on the same address via `SO_REUSEPORT`. The kernel distributes incoming connections across them using a hash of the 4-tuple. This eliminates the accept lock contention that would occur with a single accepting socket + channel dispatch, and scales linearly with CPU count.
 

README.md

@@ -17,7 +17,7 @@ symphony sits in front of your services and:
 - **Limits** routes with per-route token-bucket rate caps to prevent any one route from starving others
 - **Protects** connections with per-IP token-bucket rate limiting, concurrency limits, CIDR allowlist/blocklist, JA3 fingerprint blocking, TLS handshake timeout, and SNI-required enforcement
 - **Suspends** routes — hold incoming connections and fire an event; your code decides whether to proxy or reject each one
-- **Hot-swaps** routes and protection config without restarting or dropping existing connections
+- **Hot-swaps** routes and per-listener protection config (CIDR lists, JA3 blocklist, rate limits, concurrency caps, handshake timeout, requireSni) without restarting or dropping existing connections
 - Scales to **~1 million concurrent connections** via `SO_REUSEPORT`, tokio's multi-thread runtime, and lock-free data structures
 
 ---
@@ -137,7 +137,9 @@ Both fields accept PEM-encoded strings or `Buffer`. The cert chain may include i
 
 | Field | Type | Default | Description |
 |---|---|---|---|
-| `rateLimit` | `{ connectionsPerSecond, burst? }` | — | Token bucket per source IP |
+| `rateLimit` | `{ connectionsPerSecond, burst? }` | — | Per-second token bucket per source IP |
+| `sustained` | `{ connectionsPerMinute, burst? }` | — | Per-minute token bucket per source IP (independent of `rateLimit`) |
+| `penaltyBox` | `{ durationMs? }` | — | Block an IP for `durationMs` after any rate limit exhaustion |
 | `maxConcurrentPerIp` | `number` | `0` (unlimited) | Max simultaneous connections per source IP |
 | `allowlist` | `string[]` | `[]` | CIDRs that bypass all checks |
 | `blocklist` | `string[]` | `[]` | CIDRs that are always blocked |
@@ -440,13 +442,50 @@ The **carrier depends on `sourceAddressHeader`**:
 ```typescript
 protection: {
   rateLimit: { connectionsPerSecond: 50, burst: 100 },
+  sustained: { connectionsPerMinute: 300, burst: 300 },
+  penaltyBox: { durationMs: 600_000 }, // 10 minutes
   maxConcurrentPerIp: 200,
   allowlist: ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'],
   requireSni: true,
   tlsHandshakeTimeoutMs: 5000,
 }
 ```
 
+### Sustained rate limits
+
+Use `sustained` to enforce a per-minute cap that is independent of the per-second `rateLimit` bucket. Both buckets are checked on every connection; exhausting either one blocks the connection. This lets you allow short bursts while still capping total volume over longer windows:
+
+```typescript
+protection: {
+  rateLimit: { connectionsPerSecond: 50, burst: 200 }, // allow short bursts
+  sustained: { connectionsPerMinute: 600, burst: 600 }, // but cap at 10/s long-term
+}
+```
+
+### Penalty box
+
+When `penaltyBox` is configured, exhausting any rate limit places the source IP in a penalty box for `durationMs` (default 10 minutes). While penalized, all connections from that IP are rejected outright without touching the token buckets — protecting the proxy from re-assembling per-connection state under a sustained attack.
+
+**Extension semantics:** while an IP is boxed, symphony still debits its token buckets on each connection attempt (to measure whether the attack is continuing). If a bucket is exhausted (the IP is still sending at the excess rate), the penalty deadline is reset to `now + durationMs` — effectively extending the penalty by a full `durationMs` from the moment of continued excess. If the IP stops attacking, the buckets refill and debits succeed; the deadline is not extended, so the IP is readmitted once the original deadline expires.
+
+```typescript
+protection: {
+  rateLimit: { connectionsPerSecond: 50, burst: 100 },
+  penaltyBox: { durationMs: 600_000 }, // 10 minutes (default)
+}
+```
+
+Blocked events from penalized IPs have `reason: 'penalty_boxed'`. They appear under `penaltyBoxed` in `blockedIps()`:
+
+```typescript
+const info = proxy.blockedIps();
+// info.penaltyBoxed — IPs currently in the penalty box
+```
+
+Penalty state is stored on per-IP runtime state and survives a configuration hot-swap. `penaltyBox` can be added or removed via `updateConfig` without restarting.
+
+**Hot-swap note on active deadlines:** changing `durationMs` via `updateConfig` affects new penalty stamps immediately, but IPs already in the box retain their current deadline until it expires or is re-stamped by a continued rate-limit hit. There is no retroactive recalculation of existing deadlines.
+
 ### JA3 blocking
 
 JA3 fingerprints the TLS ClientHello by hashing a canonical string of the version, cipher suites, extensions, elliptic curves, and EC point formats using MD5. That MD5 is by specification, not a security claim — it allows lists of known-bad fingerprints to be compared cheaply. Collect JA3 fingerprints from your logs (available in the `blocked` event `ja3` field) and add known-bad clients:
@@ -479,7 +518,27 @@ Matching is case-insensitive. JA4 fingerprints are always emitted as lowercase.
 
 ### Hot-swapping protection config
 
-Protection config is per-listener and not currently hot-swappable via `updateConfig` (listeners would need to restart). To update protection, restart with a new config. Route changes do not require listener restarts.
+Protection config is per-listener and fully hot-swappable via `updateConfig`. Push a new config atomically to each listener by port — no restart needed, in-flight connections are unaffected:
+
+```typescript
+// Block a new CIDR range without restarting
+proxy.updateConfig({
+  protection: [
+    {
+      port: 443,
+      protection: {
+        blocklist: ['198.51.100.0/24'],  // added under attack
+        rateLimit: { connectionsPerSecond: 50, burst: 100 },
+        requireSni: true,
+      },
+    },
+  ],
+});
+```
+
+The entire `ProtectionConfig` is replaced atomically (one pointer swap). Any field not included in the new config reverts to its default. Existing per-IP rate-limit token buckets are preserved across a swap; if burst decreases, tokens are capped at the new ceiling on the next refill — no underflow.
+
+**Transitions (none→some, some→none):** when using `symphony-server`, adding or removing a `protection` block in the config file triggers a seamless proxy recreate via SO_REUSEPORT — no bind gap, existing connections unaffected. Only contents-only changes (same presence, different CIDR/rate/etc.) stay on the pure hot-swap path. When calling `updateConfig()` directly, a listener that was started without protection cannot gain it and returns an error; restart the listener to change protection presence.
 
 ---
 
@@ -503,9 +562,10 @@ const m = proxy.metrics();
 // m.pendingSuspended — connections currently held waiting for resolveConnection()
 
 const blocked = proxy.blockedIps();
-// blocked.rateLimited — IPs with depleted token buckets
+// blocked.rateLimited — IPs with a depleted per-second or sustained token bucket
 // blocked.concurrencyLimited — IPs at their maxConcurrentPerIp limit
 // blocked.cidrBlocklist — the configured static CIDR blocklist
+// blocked.penaltyBoxed — IPs currently in the penalty box
 
 setInterval(() => {
   console.log('active:', proxy.metrics().activeConnections);
@@ -523,9 +583,9 @@ proxy.updateConfig({
 });
 ```
 
-**What can be hot-swapped:** routes (destinations, TLS certs, suspension state).
+**What can be hot-swapped:** routes (destinations, TLS certs, suspension state) and per-listener protection (CIDR allowlist/blocklist, JA3 blocklist, rate limits, sustained rate limits, penalty box, concurrency caps, handshake timeout, requireSni).
 
-**What requires a restart:** listeners (bind address, port, protection config, idle timeout).
+**What requires a restart:** bind address, port, idle timeout, worker threads. When calling `updateConfig()` directly, protection presence (None↔Some) cannot change at runtime — the listener must be restarted. The `symphony-server` bin handles this automatically via seamless recreate.
 
 ---