diff --git a/ops/selfhost/README.md b/ops/selfhost/README.md index d1ff47e..4bdb2a6 100644 --- a/ops/selfhost/README.md +++ b/ops/selfhost/README.md @@ -181,13 +181,73 @@ Order that matters, because two databases can otherwise drive the same app: **not** `supabase/.env` — that directory stays root-owned because it holds the database's God Mode keys and a deploy has no business reading them. +## Network posture + +What dev2 exposes to the internet, and why: + +| Port | Open to | Why | +| --- | --- | --- | +| 22 | everyone | ssh | +| 80 / 443 | everyone | nginx: the app, and the Supabase gateway | +| 5432 | **allowlist** | Postgres, for the dev box only | +| 6379 | **nobody** | Redis is loopback; the prober tunnels in | +| 8000, 3100 | nobody | gateway and app, loopback behind nginx | + +Two rules of thumb the hard way: + +- **ufw does not gate a published Docker port.** Docker inserts its own + iptables rules ahead of ufw's chains, so a ufw rule for a container port is + decoration. `DOCKER-USER` is the chain Docker consults first and leaves + alone; `restrict-data-ports.sh` puts the 5432 allowlist there and persists it + to `/etc/iptables/rules.v4`. +- **The allowlist has to include `172.16.0.0/12`**, not just this stack's + subnet. Other apps on the box run in their own compose projects on their own + bridges and reach Postgres through the published port, so they arrive from a + different docker subnet and a narrow allowlist cuts them off the moment they + start. + +`pg_hba` is the other half and is deliberately unchanged by any of this: TLS +required, `postgres` role only, scram. + +Admin access is **Supabase Studio at `https://supabase.crawlproof.com`**, +behind HTTP basic auth (`DASHBOARD_USERNAME` / `DASHBOARD_PASSWORD` in +`supabase/.env`). It rides the existing TLS, so there is no separate admin port +to open. + ## Redis and the prober `scan.crawlproof.com` (164.92.111.224) is a DigitalOcean droplet running the -nmap prober as a BullMQ consumer. It connects **outbound** to Redis using the -`PROBER_REDIS_URL` repo secret, which today points at Railway. After the move -that secret has to be repointed at dev2, and ufw opened to that one address — -the Redis port is on loopback by default. +nmap prober as a BullMQ consumer, and it is the **only** remote consumer of +anything on dev2. It stays on its own droplet deliberately: it port-scans +customer sites, and doing that from the box that serves crawlproof.com would +put the app's own address behind the scanning traffic. + +Rather than open Redis to it, it tunnels: + +```sh +# on the prober, once +ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519_dev2 +# on dev2, with that public key +ops/selfhost/server/authorize-prober-tunnel.sh "$(cat ~/.ssh/id_ed25519_dev2.pub)" +# back on the prober +ops/selfhost/server/prober-redis-tunnel.sh +``` + +Then set the `PROBER_REDIS_URL` repo secret to a `redis` scheme URL for user +`default`, host `127.0.0.1`, port **6380**, with the password from +`REDIS_PASSWORD` in `deploy.env` on dev2. Build it where you set the secret; +do not write it down here. + +Two things that will catch you: + +- **The prober droplet already runs its own `redis-server` on 6379.** The + tunnel therefore binds **6380** locally. Using 6379 either fails to bind or, + worse, silently points the prober at the wrong Redis. +- The key on dev2 is + `restrict,port-forwarding,permitopen="127.0.0.1:6379",command="/bin/false"`, + so it cannot open a shell or reach any other port. `restrict` turns + everything off including forwarding, which is why `port-forwarding` has to be + listed again after it. This repo's default branch is **`master`**, not `main`. `deploy-prober.yml` and `deploy-dev2.yml` both watch `master` for that reason, and `deploy-app.sh` diff --git a/ops/selfhost/server/authorize-prober-tunnel.sh b/ops/selfhost/server/authorize-prober-tunnel.sh new file mode 100755 index 0000000..b23d531 --- /dev/null +++ b/ops/selfhost/server/authorize-prober-tunnel.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# +# Authorise the prober's key on dev2 for ONE thing: a port-forward to Redis. +# +# `restrict` turns everything off (no pty, no agent/X11 forwarding, no +# port-forwarding), then `port-forwarding` + `permitopen` turn exactly one +# destination back on. So this key cannot get a shell and cannot reach any +# other port on the box — it is strictly less access than the public port it +# replaces. +set -euo pipefail + +# The prober's public key, from `cat ~/.ssh/id_ed25519_dev2.pub` on +# scan.crawlproof.com. Pass it as $1, or set PUBKEY. +PUBKEY=${1:-${PUBKEY:-}} +[ -n "$PUBKEY" ] || { echo "usage: authorize-prober-tunnel.sh ''" >&2; exit 1; } +case "$PUBKEY" in + ssh-*) ;; + *) echo "that does not look like a public key" >&2; exit 1 ;; +esac +OPTS='restrict,port-forwarding,permitopen="127.0.0.1:6379",command="/bin/false"' +AK=/home/anthony/.ssh/authorized_keys + +install -d -o anthony -g anthony -m 700 /home/anthony/.ssh +touch "$AK"; chown anthony:anthony "$AK"; chmod 600 "$AK" + +# Drop any previous copy of this key so re-running does not stack entries. +# Match on the key material, not the comment, which anyone can change. +KEYBODY=$(printf '%s' "$PUBKEY" | awk '{print $2}') +grep -vF "$KEYBODY" "$AK" > "$AK.tmp" 2>/dev/null || true +mv "$AK.tmp" "$AK" +printf '%s %s\n' "$OPTS" "$PUBKEY" >> "$AK" +chown anthony:anthony "$AK"; chmod 600 "$AK" + +echo "=== authorized_keys entries ===" +sed 's/AAAA[A-Za-z0-9+/=]*//' "$AK" diff --git a/ops/selfhost/server/prober-redis-tunnel.sh b/ops/selfhost/server/prober-redis-tunnel.sh new file mode 100755 index 0000000..1383536 --- /dev/null +++ b/ops/selfhost/server/prober-redis-tunnel.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# +# Give the prober a private path to dev2's Redis, so Redis needs no public +# port at all. +# +# The prober is the only remote consumer of that queue. It stays on its own +# droplet on purpose: it runs nmap against customer sites, and doing that from +# the box that serves crawlproof.com would put the app's own IP behind the +# scanning traffic. +set -euo pipefail + +sudo tee /etc/systemd/system/redis-tunnel.service >/dev/null <<'EOF' +[Unit] +Description=SSH tunnel to dev2 Redis (BullMQ prober queue) +After=network-online.target +Wants=network-online.target + +[Service] +User=ubuntu +# -N: no remote command (the key is restricted to /bin/false anyway) +# ExitOnForwardFailure: fail loudly instead of running a tunnel-less process +# that would let the prober "start" and silently never see a job. +ExecStart=/usr/bin/ssh -NT \ + -o ExitOnForwardFailure=yes \ + -o ServerAliveInterval=30 \ + -o ServerAliveCountMax=3 \ + -o StrictHostKeyChecking=accept-new \ + -o BatchMode=yes \ + -i /home/ubuntu/.ssh/id_ed25519_dev2 \ + -L 127.0.0.1:6379:127.0.0.1:6379 \ + anthony@dev2.profullstack.com +Restart=always +RestartSec=5 + +[Install] +WantedBy=multi-user.target +EOF + +sudo systemctl daemon-reload +sudo systemctl enable --now redis-tunnel.service +sleep 5 +systemctl is-active redis-tunnel.service +echo "--- tunnel listening? ---" +ss -tlnp 2>/dev/null | grep 6379 || echo "NOT LISTENING" diff --git a/ops/selfhost/server/restrict-data-ports.sh b/ops/selfhost/server/restrict-data-ports.sh new file mode 100755 index 0000000..1894214 --- /dev/null +++ b/ops/selfhost/server/restrict-data-ports.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# +# Restrict dev2's published Postgres port to an IP allowlist. +# +# The self-host kit publishes 5432 and guards it with pg_hba (TLS required, +# `postgres` role only, scram). That is decent, but it still means anyone on +# the internet can reach the port and try. Nothing outside our own +# infrastructure has any reason to connect. +# +# pg_hba is deliberately NOT changed: the nichedb session running in this same +# cluster depends on the `hostssl all postgres 0.0.0.0/0` rule, and postgres +# config stays theirs. This is purely a network-layer allowlist. +# +# DOCKER-USER, not ufw: Docker publishes ports by inserting its own iptables +# rules ahead of ufw's chains, so a ufw rule here would be decoration. +set -euo pipefail + +PORT=${PORT:-5432} +# Our dev box, where the migration tooling and both agent sessions run. +ALLOW=${ALLOW:-67.205.189.229} + +echo "=== before ===" +iptables -L DOCKER-USER -n --line-numbers | head -10 + +# Idempotent: strip any previous version of these rules first. +while iptables -D DOCKER-USER -p tcp --dport "$PORT" -j DROP 2>/dev/null; do :; done +for ip in $ALLOW 127.0.0.1 172.16.0.0/12; do + while iptables -D DOCKER-USER -s "$ip" -p tcp --dport "$PORT" -j ACCEPT 2>/dev/null; do :; done +done + +# Catch-all drop first, then insert the accepts above it (-I 1 prepends, so +# the last inserted ends up on top). +iptables -I DOCKER-USER 1 -p tcp --dport "$PORT" -j DROP +iptables -I DOCKER-USER 1 -s 172.16.0.0/12 -p tcp --dport "$PORT" -j ACCEPT +iptables -I DOCKER-USER 1 -s 127.0.0.1 -p tcp --dport "$PORT" -j ACCEPT +for ip in $ALLOW; do + iptables -I DOCKER-USER 1 -s "$ip" -p tcp --dport "$PORT" -j ACCEPT +done + +echo "=== after ===" +iptables -L DOCKER-USER -n --line-numbers | head -10 + +mkdir -p /etc/iptables +iptables-save > /etc/iptables/rules.v4 +echo "persisted to /etc/iptables/rules.v4" + +echo "=== postgres still up and pg_hba untouched ===" +docker exec supabase-db pg_isready -U postgres -h localhost +grep -c 'hostssl all postgres 0.0.0.0/0' \ + /home/anthony/www/crawlproof.com/supabase/volumes/crawlproof/pg_hba.conf \ + && echo "nichedb's hba rule intact"