ops: take Redis off the internet, put 5432 behind an allowlist

ops/selfhost/README.md

@@ -181,13 +181,73 @@ Order that matters, because two databases can otherwise drive the same app:
 **not** `supabase/.env` — that directory stays root-owned because it holds the
 database's God Mode keys and a deploy has no business reading them.
 
+## Network posture
+
+What dev2 exposes to the internet, and why:
+
+| Port | Open to | Why |
+| --- | --- | --- |
+| 22 | everyone | ssh |
+| 80 / 443 | everyone | nginx: the app, and the Supabase gateway |
+| 5432 | **allowlist** | Postgres, for the dev box only |
+| 6379 | **nobody** | Redis is loopback; the prober tunnels in |
+| 8000, 3100 | nobody | gateway and app, loopback behind nginx |
+
+Two rules of thumb the hard way:
+
+- **ufw does not gate a published Docker port.** Docker inserts its own
+  iptables rules ahead of ufw's chains, so a ufw rule for a container port is
+  decoration. `DOCKER-USER` is the chain Docker consults first and leaves
+  alone; `restrict-data-ports.sh` puts the 5432 allowlist there and persists it
+  to `/etc/iptables/rules.v4`.
+- **The allowlist has to include `172.16.0.0/12`**, not just this stack's
+  subnet. Other apps on the box run in their own compose projects on their own
+  bridges and reach Postgres through the published port, so they arrive from a
+  different docker subnet and a narrow allowlist cuts them off the moment they
+  start.
+
+`pg_hba` is the other half and is deliberately unchanged by any of this: TLS
+required, `postgres` role only, scram.
+
+Admin access is **Supabase Studio at `https://supabase.crawlproof.com`**,
+behind HTTP basic auth (`DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` in
+`supabase/.env`). It rides the existing TLS, so there is no separate admin port
+to open.
+
 ## Redis and the prober
 
 `scan.crawlproof.com` (164.92.111.224) is a DigitalOcean droplet running the
-nmap prober as a BullMQ consumer. It connects **outbound** to Redis using the
-`PROBER_REDIS_URL` repo secret, which today points at Railway. After the move
-that secret has to be repointed at dev2, and ufw opened to that one address —
-the Redis port is on loopback by default.
+nmap prober as a BullMQ consumer, and it is the **only** remote consumer of
+anything on dev2. It stays on its own droplet deliberately: it port-scans
+customer sites, and doing that from the box that serves crawlproof.com would
+put the app's own address behind the scanning traffic.
+
+Rather than open Redis to it, it tunnels:
+
+```sh
+# on the prober, once
+ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519_dev2
+# on dev2, with that public key
+ops/selfhost/server/authorize-prober-tunnel.sh "$(cat ~/.ssh/id_ed25519_dev2.pub)"
+# back on the prober
+ops/selfhost/server/prober-redis-tunnel.sh
+```
+
+Then set the `PROBER_REDIS_URL` repo secret to a `redis` scheme URL for user
+`default`, host `127.0.0.1`, port **6380**, with the password from
+`REDIS_PASSWORD` in `deploy.env` on dev2. Build it where you set the secret;
+do not write it down here.
+
+Two things that will catch you:
+
+- **The prober droplet already runs its own `redis-server` on 6379.** The
+  tunnel therefore binds **6380** locally. Using 6379 either fails to bind or,
+  worse, silently points the prober at the wrong Redis.
+- The key on dev2 is
+  `restrict,port-forwarding,permitopen="127.0.0.1:6379",command="/bin/false"`,
+  so it cannot open a shell or reach any other port. `restrict` turns
+  everything off including forwarding, which is why `port-forwarding` has to be
+  listed again after it.
 
 This repo's default branch is **`master`**, not `main`. `deploy-prober.yml` and
 `deploy-dev2.yml` both watch `master` for that reason, and `deploy-app.sh`

ops/selfhost/server/authorize-prober-tunnel.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+#
+# Authorise the prober's key on dev2 for ONE thing: a port-forward to Redis.
+#
+# `restrict` turns everything off (no pty, no agent/X11 forwarding, no
+# port-forwarding), then `port-forwarding` + `permitopen` turn exactly one
+# destination back on. So this key cannot get a shell and cannot reach any
+# other port on the box — it is strictly less access than the public port it
+# replaces.
+set -euo pipefail
+
+# The prober's public key, from `cat ~/.ssh/id_ed25519_dev2.pub` on
+# scan.crawlproof.com. Pass it as $1, or set PUBKEY.
+PUBKEY=${1:-${PUBKEY:-}}
+[ -n "$PUBKEY" ] || { echo "usage: authorize-prober-tunnel.sh '<ssh-ed25519 ... comment>'" >&2; exit 1; }
+case "$PUBKEY" in
+  ssh-*) ;;
+  *) echo "that does not look like a public key" >&2; exit 1 ;;
+esac
+OPTS='restrict,port-forwarding,permitopen="127.0.0.1:6379",command="/bin/false"'
+AK=/home/anthony/.ssh/authorized_keys
+
+install -d -o anthony -g anthony -m 700 /home/anthony/.ssh
+touch "$AK"; chown anthony:anthony "$AK"; chmod 600 "$AK"
+
+# Drop any previous copy of this key so re-running does not stack entries.
+# Match on the key material, not the comment, which anyone can change.
+KEYBODY=$(printf '%s' "$PUBKEY" | awk '{print $2}')
+grep -vF "$KEYBODY" "$AK" > "$AK.tmp" 2>/dev/null || true
+mv "$AK.tmp" "$AK"
+printf '%s %s\n' "$OPTS" "$PUBKEY" >> "$AK"
+chown anthony:anthony "$AK"; chmod 600 "$AK"
+
+echo "=== authorized_keys entries ==="
+sed 's/AAAA[A-Za-z0-9+/=]*/<key>/' "$AK"

ops/selfhost/server/prober-redis-tunnel.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+#
+# Give the prober a private path to dev2's Redis, so Redis needs no public
+# port at all.
+#
+# The prober is the only remote consumer of that queue. It stays on its own
+# droplet on purpose: it runs nmap against customer sites, and doing that from
+# the box that serves crawlproof.com would put the app's own IP behind the
+# scanning traffic.
+set -euo pipefail
+
+sudo tee /etc/systemd/system/redis-tunnel.service >/dev/null <<'EOF'
+[Unit]
+Description=SSH tunnel to dev2 Redis (BullMQ prober queue)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+User=ubuntu
+# -N: no remote command (the key is restricted to /bin/false anyway)
+# ExitOnForwardFailure: fail loudly instead of running a tunnel-less process
+#   that would let the prober "start" and silently never see a job.
+ExecStart=/usr/bin/ssh -NT \
+  -o ExitOnForwardFailure=yes \
+  -o ServerAliveInterval=30 \
+  -o ServerAliveCountMax=3 \
+  -o StrictHostKeyChecking=accept-new \
+  -o BatchMode=yes \
+  -i /home/ubuntu/.ssh/id_ed25519_dev2 \
+  -L 127.0.0.1:6379:127.0.0.1:6379 \
+  anthony@dev2.profullstack.com
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo systemctl daemon-reload
+sudo systemctl enable --now redis-tunnel.service
+sleep 5
+systemctl is-active redis-tunnel.service
+echo "--- tunnel listening? ---"
+ss -tlnp 2>/dev/null | grep 6379 || echo "NOT LISTENING"

ops/selfhost/server/restrict-data-ports.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+#
+# Restrict dev2's published Postgres port to an IP allowlist.
+#
+# The self-host kit publishes 5432 and guards it with pg_hba (TLS required,
+# `postgres` role only, scram). That is decent, but it still means anyone on
+# the internet can reach the port and try. Nothing outside our own
+# infrastructure has any reason to connect.
+#
+# pg_hba is deliberately NOT changed: the nichedb session running in this same
+# cluster depends on the `hostssl all postgres 0.0.0.0/0` rule, and postgres
+# config stays theirs. This is purely a network-layer allowlist.
+#
+# DOCKER-USER, not ufw: Docker publishes ports by inserting its own iptables
+# rules ahead of ufw's chains, so a ufw rule here would be decoration.
+set -euo pipefail
+
+PORT=${PORT:-5432}
+# Our dev box, where the migration tooling and both agent sessions run.
+ALLOW=${ALLOW:-67.205.189.229}
+
+echo "=== before ==="
+iptables -L DOCKER-USER -n --line-numbers | head -10
+
+# Idempotent: strip any previous version of these rules first.
+while iptables -D DOCKER-USER -p tcp --dport "$PORT" -j DROP 2>/dev/null; do :; done
+for ip in $ALLOW 127.0.0.1 172.16.0.0/12; do
+  while iptables -D DOCKER-USER -s "$ip" -p tcp --dport "$PORT" -j ACCEPT 2>/dev/null; do :; done
+done
+
+# Catch-all drop first, then insert the accepts above it (-I 1 prepends, so
+# the last inserted ends up on top).
+iptables -I DOCKER-USER 1 -p tcp --dport "$PORT" -j DROP
+iptables -I DOCKER-USER 1 -s 172.16.0.0/12 -p tcp --dport "$PORT" -j ACCEPT
+iptables -I DOCKER-USER 1 -s 127.0.0.1 -p tcp --dport "$PORT" -j ACCEPT
+for ip in $ALLOW; do
+  iptables -I DOCKER-USER 1 -s "$ip" -p tcp --dport "$PORT" -j ACCEPT
+done
+
+echo "=== after ==="
+iptables -L DOCKER-USER -n --line-numbers | head -10
+
+mkdir -p /etc/iptables
+iptables-save > /etc/iptables/rules.v4
+echo "persisted to /etc/iptables/rules.v4"
+
+echo "=== postgres still up and pg_hba untouched ==="
+docker exec supabase-db pg_isready -U postgres -h localhost
+grep -c 'hostssl all  postgres       0.0.0.0/0' \
+  /home/anthony/www/crawlproof.com/supabase/volumes/crawlproof/pg_hba.conf \
+  && echo "nichedb's hba rule intact"