From 5dbcf9dfb1411c45c4b619d46bd7f4596287fa4f Mon Sep 17 00:00:00 2001 From: "protostatis.dev" Date: Thu, 13 Aug 2026 11:02:31 -0500 Subject: [PATCH] fix: remove host-wide WireGuard dependency --- .env.docker.example | 13 +--- .github/workflows/deploy.yml | 34 +++++++++- CLAUDE.md | 29 ++++----- deploy/push-to-ec2.sh | 78 +++++++++++++++++++---- deploy/setup-wireguard.sh | 83 ------------------------ docs/EC2_INSTANCES.md | 112 +++++++++++++++------------------ docs/PROXY_SETUP.md | 7 +-- docs/REDDIT_UNBROWSER_SETUP.md | 10 +-- 8 files changed, 169 insertions(+), 197 deletions(-) delete mode 100755 deploy/setup-wireguard.sh diff --git a/.env.docker.example b/.env.docker.example index 6087761..eab7e6a 100644 --- a/.env.docker.example +++ b/.env.docker.example @@ -17,21 +17,10 @@ WHALE_ALERT_API_KEY= # Etherscan (for on-chain data) ETHERSCAN_API_KEY= -# ============================================ -# WireGuard VPN (host outbound traffic) -# ============================================ -# WireGuard remains enabled for normal crawler dependencies. Reddit HTML uses -# the cookie-backed Unbrowser transport described in docs/REDDIT_UNBROWSER_SETUP.md. -# Run deploy/setup-wireguard.sh to configure WireGuard. -WG_PRIVATE_KEY=your_wireguard_private_key -WG_ADDRESS=10.x.x.x/32 -WG_DNS=10.64.0.1 -WG_PEER_PUBKEY=server_public_key -WG_ENDPOINT=server_ip:51820 - # ============================================ # Reddit Unbrowser Cookie Transport # ============================================ +# Production hosts use direct EC2 egress; do not configure a host-wide VPN. # The release workflow sets REDDIT_FETCH_MODE and the Unix socket itself. # Set this token only after configuring the local solver; never commit it. UNBROWSER_COOKIE_SERVICE_TOKEN= diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 9a964ce..f98bfd7 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -117,8 +117,38 @@ jobs: cd /home/ec2-user/crypto_sentiment_crawler - # Keep WireGuard active for non-Reddit crawler dependencies. Reddit - # HTML is fetched by Unbrowser with a local cookie refresh socket. + # Retire the legacy host-wide WireGuard tunnel before any + # DNS-dependent deployment step. Reddit now uses the cookie-backed + # Unbrowser transport; all other traffic uses direct EC2 egress. + sudo systemctl disable --now wg-quick@wg0 >/dev/null 2>&1 || true + if /sbin/ip link show wg0 >/dev/null 2>&1; then + echo "Stopping legacy WireGuard interface..." + if ! command -v wg-quick >/dev/null 2>&1; then + echo "ERROR: wg0 is active but wg-quick is unavailable" + exit 1 + fi + sudo wg-quick down wg0 + fi + if /sbin/ip link show wg0 >/dev/null 2>&1; then + echo "ERROR: Legacy WireGuard interface is still active" + exit 1 + fi + WG_UNIT_STATE=$(sudo systemctl is-enabled wg-quick@wg0 2>/dev/null || true) + case "$WG_UNIT_STATE" in + enabled|enabled-runtime|linked|linked-runtime) + echo "ERROR: Legacy WireGuard service remains enabled ($WG_UNIT_STATE)" + exit 1 + ;; + esac + sudo rm -f /etc/wireguard/wg0.conf + if ! timeout 15 getent ahostsv4 github.com >/dev/null 2>&1; then + echo "ERROR: Direct EC2 DNS is unavailable after WireGuard cleanup" + exit 1 + fi + if ! curl -fsS --connect-timeout 5 --max-time 15 https://github.com/robots.txt >/dev/null; then + echo "ERROR: Direct EC2 HTTPS egress is unavailable after WireGuard cleanup" + exit 1 + fi # Pull latest code. Preserve any EC2-local edits so deployment can # move to the release tag without discarding operational changes. diff --git a/CLAUDE.md b/CLAUDE.md index d6a979b..7d1bfb5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -68,26 +68,21 @@ cd /opt/crypto-sentiment && docker-compose up -d `deploy/push-to-ec2.sh ` — packages local code, uploads via SCP, rebuilds on EC2. Only use if CI/CD is broken. -## Reddit Access via WireGuard VPN +## Reddit Access via Unbrowser -Reddit blocks AWS/EC2 IP addresses. A WireGuard VPN on the EC2 host routes -outbound crawler traffic through a clean IP. +Production uses direct EC2 egress. Reddit HTML is fetched by Unbrowser with +cookies supplied by a supervised Mac solver over a mode-0600 SSH Unix socket. +Do not enable a host-wide VPN: a dead full tunnel also blocks CoinGecko, +Telegram, and every other outbound feed. -### Setup (one-time): - -1. Add WireGuard config to `/opt/crypto-sentiment/.env` (see `.env.docker.example` for template). -2. Run `deploy/setup-wireguard.sh /opt/crypto-sentiment/.env` on EC2. - -### Verify VPN is active: +See `docs/REDDIT_UNBROWSER_SETUP.md` for setup and recovery. Quick checks: ```bash -# Should show the VPN egress IP, not the EC2 IP -curl https://httpbin.org/ip - -# Check WireGuard status -sudo wg show +curl --unix-socket /opt/crypto-sentiment/run/reddit-cookie-solver.sock \ + http://localhost/healthz +docker exec crypto-crawler sh -c 'test "$REDDIT_FETCH_MODE" = unbrowser' ``` -### If crawler shows "No fresh posts" or 403 errors: -- VPN may be down: `sudo wg-quick up wg0`. -- Check status: `sudo systemctl status wg-quick@wg0`. +If the crawler shows repeated empty results, verify the solver/tunnel agents on +the Mac, the EC2 socket metadata, and the deployment canary before restarting +the crawler. diff --git a/deploy/push-to-ec2.sh b/deploy/push-to-ec2.sh index e3a745d..f40d2b0 100755 --- a/deploy/push-to-ec2.sh +++ b/deploy/push-to-ec2.sh @@ -56,17 +56,75 @@ echo "Deploying on EC2..." ssh -i "$KEY_PATH" -o StrictHostKeyChecking=no "ec2-user@$EC2_IP" << 'ENDSSH' set -e -# Extract application -sudo rm -rf /opt/crypto-sentiment/* +APP_DIR=/opt/crypto-sentiment +PERSIST_DIR=/tmp/crypto-sentiment-persist + +# Retire the legacy host-wide VPN before any destructive deployment step. +# Reddit uses the cookie-backed Unbrowser transport in production. +sudo systemctl disable --now wg-quick@wg0 >/dev/null 2>&1 || true +if /sbin/ip link show wg0 >/dev/null 2>&1; then + echo "Stopping legacy WireGuard interface..." + if ! command -v wg-quick >/dev/null 2>&1; then + echo "ERROR: wg0 is active but wg-quick is unavailable" + exit 1 + fi + sudo wg-quick down wg0 +fi +if /sbin/ip link show wg0 >/dev/null 2>&1; then + echo "ERROR: Legacy WireGuard interface is still active" + exit 1 +fi +WG_UNIT_STATE=$(sudo systemctl is-enabled wg-quick@wg0 2>/dev/null || true) +case "$WG_UNIT_STATE" in + enabled|enabled-runtime|linked|linked-runtime) + echo "ERROR: Legacy WireGuard service remains enabled ($WG_UNIT_STATE)" + exit 1 + ;; +esac +sudo rm -f /etc/wireguard/wg0.conf +if ! timeout 15 getent ahostsv4 github.com >/dev/null 2>&1; then + echo "ERROR: Direct EC2 DNS is unavailable after WireGuard cleanup" + exit 1 +fi +if ! curl -fsS --connect-timeout 5 --max-time 15 https://github.com/robots.txt >/dev/null; then + echo "ERROR: Direct EC2 HTTPS egress is unavailable after WireGuard cleanup" + exit 1 +fi + +# Preserve runtime state before replacing application files. +rm -rf "$PERSIST_DIR" +mkdir -p "$PERSIST_DIR" +for name in data logs backups run .env; do + if [ -e "$APP_DIR/$name" ]; then + mv "$APP_DIR/$name" "$PERSIST_DIR/$name" + fi +done + +# Extract application. Restore runtime state even if extraction fails. +restore_runtime_state() { + sudo mkdir -p "$APP_DIR" + sudo chown ec2-user:ec2-user "$APP_DIR" + for name in data logs backups run .env; do + if [ -e "$PERSIST_DIR/$name" ]; then + rm -rf "$APP_DIR/$name" + mv "$PERSIST_DIR/$name" "$APP_DIR/$name" + fi + done +} +trap restore_runtime_state EXIT + +sudo rm -rf "$APP_DIR" sudo tar -xzf /tmp/crypto-sentiment.tar.gz -C /opt -sudo mv /opt/app/* /opt/crypto-sentiment/ -sudo rmdir /opt/app -sudo chown -R ec2-user:ec2-user /opt/crypto-sentiment +sudo mv /opt/app "$APP_DIR" +sudo chown -R ec2-user:ec2-user "$APP_DIR" +restore_runtime_state +trap - EXIT +rm -rf "$PERSIST_DIR" -cd /opt/crypto-sentiment +cd "$APP_DIR" # Create data directories -mkdir -p data logs +mkdir -p data logs backups run # Check for .env file if [ ! -f .env ]; then @@ -77,12 +135,6 @@ if [ ! -f .env ]; then echo "" fi -# Setup WireGuard VPN if configured -if grep -q WG_PRIVATE_KEY .env 2>/dev/null; then - echo "Setting up WireGuard VPN..." - bash deploy/setup-wireguard.sh .env -fi - # Build and start services echo "Building Docker images..." docker-compose build --no-cache diff --git a/deploy/setup-wireguard.sh b/deploy/setup-wireguard.sh deleted file mode 100755 index 6b0406e..0000000 --- a/deploy/setup-wireguard.sh +++ /dev/null @@ -1,83 +0,0 @@ -#!/bin/bash -# Setup WireGuard VPN on EC2 for Reddit access -# Reads configuration from environment variables or .env file -# -# Required env vars: -# WG_PRIVATE_KEY - WireGuard private key -# WG_ADDRESS - VPN address (e.g., 10.x.x.x/32) -# WG_DNS - DNS server (e.g., 10.64.0.1) -# WG_PEER_PUBKEY - Server public key -# WG_ENDPOINT - Server endpoint (e.g., server_ip:51820) - -set -e - -# Load .env if it exists -ENV_FILE="${1:-.env}" -if [ -f "$ENV_FILE" ]; then - echo "Loading config from $ENV_FILE" - set -a - source "$ENV_FILE" - set +a -fi - -# Validate required vars -for var in WG_PRIVATE_KEY WG_ADDRESS WG_DNS WG_PEER_PUBKEY WG_ENDPOINT; do - if [ -z "${!var}" ]; then - echo "ERROR: $var is not set" - exit 1 - fi -done - -# Install wireguard-tools if needed -if ! command -v wg &> /dev/null; then - echo "Installing wireguard-tools..." - sudo yum install -y wireguard-tools -fi - -# Detect EC2 private IP and default gateway -EC2_IP=$(hostname -I | awk '{print $1}') -EC2_GW=$(/sbin/ip route show default dev eth0 | awk '{print $3}') -echo "EC2 private IP: $EC2_IP" -echo "EC2 gateway: $EC2_GW" - -# Write WireGuard config -echo "Writing /etc/wireguard/wg0.conf..." -sudo bash -c "cat > /etc/wireguard/wg0.conf << EOF -[Interface] -PrivateKey = $WG_PRIVATE_KEY -Address = $WG_ADDRESS -DNS = $WG_DNS -MTU = 1420 -PostUp = /sbin/ip rule add from $EC2_IP table main priority 90 2>/dev/null || true; /sbin/iptables -t mangle -A PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0xca6c; /sbin/iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff; /sbin/ip route replace 140.82.112.0/20 via $EC2_GW dev eth0; /sbin/ip route replace 185.199.108.0/22 via $EC2_GW dev eth0; /sbin/ip route replace 169.254.169.254 dev eth0 -PostDown = /sbin/ip rule del from $EC2_IP table main priority 90 2>/dev/null || true; /sbin/iptables -t mangle -D PREROUTING -i eth0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0xca6c 2>/dev/null || true; /sbin/iptables -t mangle -D PREROUTING -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff 2>/dev/null || true; /sbin/ip route del 140.82.112.0/20 via $EC2_GW dev eth0 2>/dev/null || true; /sbin/ip route del 185.199.108.0/22 via $EC2_GW dev eth0 2>/dev/null || true; /sbin/ip route del 169.254.169.254 dev eth0 2>/dev/null || true - -[Peer] -PublicKey = $WG_PEER_PUBKEY -Endpoint = $WG_ENDPOINT -AllowedIPs = 0.0.0.0/0 -EOF" - -sudo chmod 600 /etc/wireguard/wg0.conf - -# Stop existing VPN if running -sudo wg-quick down wg0 2>/dev/null || true - -# Start VPN -echo "Starting WireGuard..." -sudo wg-quick up wg0 - -# Enable on boot -sudo systemctl enable wg-quick@wg0 2>/dev/null || true - -# Verify -echo "" -echo "=== Verification ===" -VPN_IP=$(curl -s --max-time 10 https://httpbin.org/ip | grep -oP '"origin": "\K[^"]+' 2>/dev/null || echo "failed") -echo "VPN IP: $VPN_IP" -echo "EC2 IP: $EC2_IP" - -if [ "$VPN_IP" != "failed" ] && [ "$VPN_IP" != "$EC2_IP" ]; then - echo "WireGuard VPN is active." -else - echo "WARNING: VPN may not be working. Check 'sudo wg show'." -fi diff --git a/docs/EC2_INSTANCES.md b/docs/EC2_INSTANCES.md index 9ea02b1..0d1dc23 100644 --- a/docs/EC2_INSTANCES.md +++ b/docs/EC2_INSTANCES.md @@ -50,8 +50,8 @@ curl -SL https://github.com/docker/compose/releases/latest/download/docker-compo -o /usr/local/lib/docker/cli-plugins/docker-compose chmod +x /usr/local/lib/docker/cli-plugins/docker-compose -# Install git and wireguard-tools -yum install -y git wireguard-tools +# Install git +yum install -y git # Create 2GB swap (prevents OOM during deployments) dd if=/dev/zero of=/swapfile bs=128M count=16 @@ -79,10 +79,9 @@ echo '/swapfile swap swap defaults 0 0' >> /etc/fstab sudo chown ec2-user:ec2-user /opt/crypto-sentiment ``` 6. **Create `.env` file** at `/opt/crypto-sentiment/.env` with API keys (see `.env.docker.example` for template) -7. **Set up WireGuard VPN for normal crawler egress** (see below) -8. **Set up the Reddit Unbrowser solver** (see `REDDIT_UNBROWSER_SETUP.md`) -9. **Deploy the application** using `deploy/push-to-ec2.sh` or CI/CD -10. **Set up daily S3 backup cron:** +7. **Set up the Reddit Unbrowser solver** (see `REDDIT_UNBROWSER_SETUP.md`) +8. **Deploy the application** using `deploy/push-to-ec2.sh` or CI/CD +9. **Set up daily S3 backup cron:** ```bash crontab -e # Add: 0 0 * * * /home/ec2-user/crypto_sentiment_crawler/deploy/backup-db.sh @@ -90,47 +89,28 @@ echo '/swapfile swap swap defaults 0 0' >> /etc/fstab --- -## WireGuard VPN Setup +## Production Egress -Reddit blocks AWS/datacenter IPs. WireGuard runs on the EC2 host and routes -outbound crawler traffic through the VPN. Docker containers inherit that route -through host NAT. +The host and Docker containers use direct EC2 egress for CoinGecko, Telegram, +macro data, on-chain APIs, and Unbrowser navigation. Reddit cookies come from +the supervised solver described in `REDDIT_UNBROWSER_SETUP.md`; no host-wide +VPN is required. -### Setup +Existing hosts previously configured with WireGuard must retire the full +tunnel before running collectors: -1. **Add WireGuard config to `/opt/crypto-sentiment/.env`** on EC2: - ```bash - WG_PRIVATE_KEY=your_wireguard_private_key - WG_ADDRESS=10.x.x.x/32 - WG_DNS=10.64.0.1 - WG_PEER_PUBKEY=server_public_key - WG_ENDPOINT=server_ip:51820 - ``` - -2. **Run the setup script:** - ```bash - cd /home/ec2-user/crypto_sentiment_crawler - bash deploy/setup-wireguard.sh /opt/crypto-sentiment/.env - ``` - -3. **Deploy the application.** Configure WireGuard once and when its settings - change; release deploys preserve the existing `wg0` service. - -4. **Verify:** - ```bash - # Should show the VPN egress IP, not the EC2 IP - curl https://ipinfo.io/ip - - sudo wg show - ``` - -### Behavior +```bash +sudo systemctl disable --now wg-quick@wg0 2>/dev/null || true +if ip link show wg0 >/dev/null 2>&1; then + sudo wg-quick down wg0 +fi +sudo rm -f /etc/wireguard/wg0.conf + +getent ahostsv4 api.coingecko.com +curl -fsS https://api.coingecko.com/api/v3/ping +``` -WireGuard carries ordinary crawler traffic through the host VPN. When the -release's cookie-backed Unbrowser canary succeeds, the crawler uses that -transport for Reddit HTML; otherwise only Reddit collection falls back to the -standard fetcher. See `REDDIT_UNBROWSER_SETUP.md` for the solver verification. -`PROXY_URL` remains available as an optional generic fallback transport. +Release and legacy deployment scripts perform this cleanup automatically. --- @@ -159,20 +139,34 @@ swapon --show ### Reddit Collection Issues ```bash -# Check VPN status -sudo wg show -sudo systemctl status wg-quick@wg0 - -# Check VPN egress IP -curl https://ipinfo.io/ip - # Check the solver socket without requesting cookie values curl --unix-socket /opt/crypto-sentiment/run/reddit-cookie-solver.sock \ http://localhost/healthz + +# Confirm the crawler is using Unbrowser +docker exec crypto-crawler sh -c 'test "$REDDIT_FETCH_MODE" = unbrowser' +``` + +Anonymous Reddit HTML can return `403`. Follow `REDDIT_UNBROWSER_SETUP.md` to +restore the supervised solver and socket forward. + +### All Outbound Feeds Stop Updating + +Check DNS from both the host and crawler. A legacy `wg0` interface must not be +present: + +```bash +getent ahostsv4 api.coingecko.com +docker exec crypto-crawler getent ahostsv4 api.coingecko.com +ip link show wg0 # expected: device does not exist ``` -Anonymous Reddit HTML can return `403` even when WireGuard is healthy. Follow -`REDDIT_UNBROWSER_SETUP.md` to restore the supervised solver and socket forward. +If `wg0` exists, disable it using the Production Egress commands above, then +restart only the outbound workers so Docker refreshes their resolver state: + +```bash +docker restart -t 120 crypto-crawler crypto-signals +``` ### S3 Backup Failing ("Unable to locate credentials") Verify the instance can reach EC2 metadata: @@ -180,16 +174,12 @@ Verify the instance can reach EC2 metadata: # Should return IAM role info — if it hangs, the route is missing curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ ``` -If it fails, re-run the WireGuard setup script to restore the metadata bypass -route: - -```bash -bash deploy/setup-wireguard.sh /opt/crypto-sentiment/.env -``` +With direct EC2 egress, a failure indicates an IAM role, metadata option, or +instance networking problem. ### Dashboard Not Accessible (but SSH works) -If Docker response packets route through the VPN, the WireGuard connmark rules -are missing. Re-run setup: +Check the frontend container and local HTTPS endpoint: ```bash -bash deploy/setup-wireguard.sh /opt/crypto-sentiment/.env +docker ps --filter name=crypto-frontend +curl -kfsS https://localhost/ >/dev/null ``` diff --git a/docs/PROXY_SETUP.md b/docs/PROXY_SETUP.md index 47df4ad..b407d56 100644 --- a/docs/PROXY_SETUP.md +++ b/docs/PROXY_SETUP.md @@ -7,16 +7,15 @@ without relying on a residential proxy credential. See [REDDIT_UNBROWSER_SETUP.md](REDDIT_UNBROWSER_SETUP.md) for the required local solver and SSH Unix-socket forward. -WireGuard remains enabled for the rest of production traffic. The integration -refreshes its in-memory Reddit cookies after an unusable Reddit response and -does not intentionally write them to GitHub secrets, the database, logs, or +The host and containers use direct EC2 egress. The integration refreshes its +in-memory Reddit cookies after an unusable Reddit response and does not +intentionally write them to GitHub secrets, the database, logs, or `/opt/crypto-sentiment/.env`. See the runtime-storage caveat in `REDDIT_UNBROWSER_SETUP.md`. ## Verification ```bash -sudo wg show curl --unix-socket /opt/crypto-sentiment/run/reddit-cookie-solver.sock \ http://localhost/healthz docker exec crypto-crawler sh -c 'test "$REDDIT_FETCH_MODE" = unbrowser' diff --git a/docs/REDDIT_UNBROWSER_SETUP.md b/docs/REDDIT_UNBROWSER_SETUP.md index 5175156..4aada4a 100644 --- a/docs/REDDIT_UNBROWSER_SETUP.md +++ b/docs/REDDIT_UNBROWSER_SETUP.md @@ -1,9 +1,9 @@ # Reddit Unbrowser Cookie Transport -Reddit blocks the crawler's normal HTTP client, including through the production -WireGuard exit. The crawler can use Unbrowser for existing `old.reddit.com` -listing, thread, and comment parsing. It asks a Mac-only solver for fresh -cookies only after Reddit returns a `403` or an HTTP-200 blocked/welcome page. +Reddit blocks the crawler's normal HTTP client from EC2. The crawler uses +Unbrowser over direct EC2 egress for existing `old.reddit.com` listing, thread, +and comment parsing. It asks a Mac-only solver for fresh cookies only after +Reddit returns a `403` or an HTTP-200 blocked/welcome page. The integration does not intentionally log cookie values or write them to its database. The Unbrowser client clears its cookies on orderly shutdown, but its @@ -144,7 +144,7 @@ stat -c '%a:%U:%G' /opt/crypto-sentiment/run/reddit-cookie-solver.sock The expected metadata is mode `600` and the deployment SSH user's owner/group. The release workflow verifies the socket, token-authenticated cookie solver, and a crawlable `old.reddit.com` listing before enabling Unbrowser for the new -crawler. WireGuard stays enabled. +crawler. The host remains on direct EC2 networking; WireGuard is not required. If the solver is unavailable or canary fails, deployment continues with standard Reddit fetching so unrelated API, frontend, and security releases remain