Smart Mobile Proxy Farm

A production-grade, self-healing mobile proxy farm running on a Raspberry Pi 4 with Huawei 4G LTE USB modems. Written in Go. Provides a rotating pool of real mobile IPs for large-scale web scraping — each request can exit through a different carrier IP.

What This Is
Hardware Setup
System Architecture
How a Request Flows
How Rotation Works
Scheduler Logic
Project File Structure
Configuration
Environment Variables
API Reference
System Files on the Pi
Deploying to the Pi
First-Time Pi Setup
Adding a New Modem
Removing a Modem
Troubleshooting
Planned Features
Key Design Decisions

What This Is

A Raspberry Pi 4 hosts several Huawei 4G USB modems. Each modem has a SIM card (Banglalink) and provides an independent LTE connection with a different public IP.
Clients connect to a single proxy endpoint (HAProxy on port 41234). HAProxy round-robins across 3proxy instances — one per modem.
When a modem finishes serving a request, the Go daemon detects the traffic and reboots the modem via Huawei's internal XML API to trigger an IP change. The new IP is live in ~43 seconds.
The result: every request (or every N requests) gets a fresh, real mobile IP from a different carrier session — ideal for scraping, ad verification, and SERP checking.

Hardware Setup

Component	Details
Raspberry Pi 4	4 GB RAM, running Raspberry Pi OS (64-bit)
Modems	Huawei E3372h (HiLink mode, appear as USB ethernet `eth1`/`eth2`/`eth3`)
SIM Cards	Banglalink 4G (Bangladesh)
Power	Each modem draws ~0.9A — do not exceed Pi's 1.2A USB budget

The modems appear to the Pi as USB ethernet adapters, not as USB serial devices. They expose a web UI at their gateway IP (e.g. 192.168.69.1) and an XML API the daemon uses.

System Architecture

 Internet / Scraper Client
          │
          ▼
  HAProxy :41234 (TCP, round-robin)
  /etc/haproxy/haproxy.cfg
          │
    ┌─────┼─────┐
    ▼     ▼     ▼
  3proxy :8081  :8082  :8083
  /etc/3proxy/3proxy.cfg
    │     │     │
    ▼     ▼     ▼
  modem1 modem2 modem3
  eth1   eth2   eth3
  (192.168.69.x) (192.168.79.x) (192.168.59.x)
    │     │     │
    ▼     ▼     ▼
  LTE → Banglalink carrier → Public IPs

          ▲
          │ manages everything
  Go Daemon :5000
  (proxy-rotator binary)
  polls HAProxy stats socket
  calls Huawei XML API
  writes 3proxy.cfg
  writes haproxy.cfg
  manages Linux routing tables

The Three Layers

1. HAProxy (The Router)

Listens on port 41234 — the single endpoint all clients use.
TCP mode (protocol-agnostic, passes HTTP/HTTPS/SOCKS traffic without inspection).
Round-robin load balances across 3proxy instances.
Active health-checks: if a 3proxy port goes down, HAProxy silently skips it.
During rotation: the daemon puts the rotating modem into maintenance mode via HAProxy's Unix socket so zero requests hit a modem that's mid-reboot.

2. 3proxy (The Tunnels)

Lightweight open-source proxy server.
Each modem gets its own port (8081, 8082, 8083) and is bound to that modem's local IP.
Binding to the specific local IP (-e<modem_local_ip>) forces traffic out through that modem's routing table — prevents IP leaks.
Config at /etc/3proxy/3proxy.cfg is regenerated by the daemon after each rotation (only if local IPs change, which is rare).

3. Go Daemon (The Brain)

Runs as a systemd service (proxy-rotator.service).
Manages all orchestration: modem state, routing tables, HAProxy maintenance mode, 3proxy config, rotation timing.
Exposes a REST API on port 5000.
Polls HAProxy's stats socket every 3 seconds to detect which modems served requests.

Linux Routing Tables

Each modem needs its own routing table to prevent IP leaks. Without this, outbound traffic might go out through the Pi's home interface instead of the modem.

# Routing table entries created by the daemon for each modem:
ip route add default via 192.168.69.1 dev eth1 table modem1
ip rule add from 192.168.69.100 table modem1

The daemon rebuilds these after each rotation (recoverNetwork() in modem/network.go).

How a Request Flows

Client connects to <pi_ip>:41234.
HAProxy picks the next available backend (e.g., modem2 on 127.0.0.1:8082).
3proxy on port 8082 receives the connection and opens an outbound connection via 192.168.79.100 (modem2's local IP on the Pi).
Linux routing table modem2 forces that outbound connection through eth2.
Traffic exits Banglalink's LTE network with modem2's current public IP.
HAProxy records the session in its stats counter for modem2.

How Rotation Works

Rotation is triggered either by the scheduler (automatic) or the API (manual). Full sequence:

Rotate(modem) called
    │
    ├─ Check if already rotating → abort if yes
    ├─ Check if other backends online → if yes, set HAProxy maintenance
    │
    ├─ fetchTokens()
    │     GET http://192.168.69.1/api/webserver/SesTokInfo
    │     Returns: SessionID + __RequestVerificationToken
    │
    ├─ sendRebootCommand()
    │     POST http://192.168.69.1/api/device/control
    │     Body: <request><Control>1</Control></request>
    │     Auth: Cookie + Token headers
    │
    ├─ waitForBoot()  [~25-35s total]
    │     Sleep 15s (modem physically powers down)
    │     Loop until modem web server returns HTTP 200:
    │       - Bring up eth interfaces
    │       - Run dhclient -1 on any interface without an IP
    │       - GET http://192.168.69.1/ → wait for 200
    │       - Poll every 3s, timeout 60s
    │
    ├─ waitForLTE()  [~3-9s]
    │     Loop until LTE is connected:
    │       - GET http://192.168.69.1/api/monitoring/status
    │       - Auth: fresh SessionID + Token
    │       - Wait for ConnectionStatus == 901 (connected)
    │       - Poll every 3s, timeout 60s
    │
    ├─ recoverNetwork()
    │     Discover modem's new local IP via: ip -4 -o addr show | grep 192.168.69.
    │     Store localIP in Modem struct
    │     Rebuild routing table:
    │       ip route add default via 192.168.69.1 dev eth1 table modem1
    │       ip rule add from <localIP> table modem1
    │
    ├─ Regenerate3Proxy()
    │     Generate new /etc/3proxy/3proxy.cfg from all modems' current localIPs
    │     Skip restart if file content is unchanged (Pi's DHCP is stable)
    │     If changed: systemctl restart 3proxy
    │
    ├─ Remove HAProxy maintenance mode (SetMaintenance false)
    │
    └─ Set status = ONLINE, record lastRotatedAt

Total rotation time: ~43 seconds

15s power-down (fixed, hardware requirement)
~15-20s for web server to come up
~3-6s for LTE to connect
~2-3s for routing recovery
3proxy restart: skipped in most cases (IPs are stable)

HAProxy Maintenance Mode

When a modem rotates, it's put into HAProxy maintenance mode:

set server proxy_nodes/modem1 state maint

This tells HAProxy to stop routing any traffic to that backend immediately — no health check delay. After rotation completes:

set server proxy_nodes/modem1 state ready

Important safety rule: Maintenance is only set if at least one other modem is ONLINE. During startup all modems rotate simultaneously. If all were put in maintenance there would be zero backends and every request would fail. The daemon checks for other online backends first.

Scheduler Logic

The scheduler (scheduler/scheduler.go) drives automatic rotation. It runs every 3 seconds.

Philosophy: Only rotate a modem that actually served requests. Don't rotate idle modems.

Algorithm:

Call haproxy.GetStats() — reads HAProxy's Unix socket (show stat CSV).
Compare each modem's stot (total sessions) to the last known value.
If stot increased → modem served new requests → add to FIFO rotation queue.
If any modem is currently StatusRotating → skip this tick entirely (one rotation at a time). Do NOT update lastStatsTot — this preserves the pending-request signal so it's not lost.
Pop the front of the FIFO queue and trigger m.Rotate(pool) in a goroutine.
Idle drain: if a queued modem hasn't seen traffic in >5 minutes, drop it from the queue.

Why FIFO: Without a fair queue, the first modem in the pool would always rotate because it's always checked first. FIFO ensures modem1, modem2, and modem3 all rotate in turn.

Why one at a time: Each rotation takes ~43s and involves HAProxy maintenance mode. If two modems rotate simultaneously and one is already in maintenance, the other being set to maintenance leaves zero backends.

Project File Structure

proxy-rotator/
├── main.go                          # Entry point: loads config, builds pool, starts scheduler + API
├── config.json                      # Modem hardware map + tuning parameters
├── .env                             # Secrets: API_PORT, API_SECRET_KEY
├── go.mod                           # Go module (only dep: godotenv)
│
├── config/
│   └── config.go                    # Config struct, Load(), Save(), IdleTimeout(), RotationInterval()
│
├── modem/
│   ├── modem.go                     # Modem struct, Snapshot, status constants, getters/setters
│   ├── pool.go                      # Pool (slice of Modems), Add/Remove/Get/All, proxy3Mu serialiser
│   ├── rotate.go                    # Rotate(), Recover(), waitForBoot(), waitForLTE()
│   ├── network.go                   # recoverNetwork(), discoverInterface(), gatewaySubnet(), runBash()
│   ├── huawei.go                    # fetchTokens(), sendRebootCommand(), checkLTEConnected()
│   └── proxy3.go                    # Regenerate3ProxyConfig() — writes /etc/3proxy/3proxy.cfg
│
├── haproxy/
│   └── haproxy.go                   # sendCommand(), GetStats(), SetMaintenance(), AddServer(),
│                                    # RemoveServer(), UpdateConfig() — haproxy.cfg template
│
├── scheduler/
│   └── scheduler.go                 # FIFO traffic-aware scheduler, polls every 3s
│
├── api/
│   ├── server.go                    # HTTP route wiring, authOK() bearer token check
│   └── handlers.go                  # All request handlers + saveConfig()
│
└── system-configs/
    ├── haproxy.cfg                  # Initial HAProxy config (copied to /etc/haproxy/ on setup)
    ├── 3proxy.cfg                   # Initial 3proxy config (copied to /etc/3proxy/ on setup)
    └── proxy-rotator.service        # systemd unit file

What each file does

File	Responsibility
`main.go`	Orchestrates startup: loads `.env`, reads `config.json`, builds modem pool, runs pre-flight interface wake, triggers staggered startup rotations, starts scheduler goroutine, starts HTTP server
`config/config.go`	JSON marshal/unmarshal for `config.json`. `IdleTimeout()` defaults to 5 min if not set. `RotationInterval()` defaults to 30 min if not set
`modem/modem.go`	Core data structure. All fields except `ID/GatewayIP/TableName/LocalPort` are mutex-protected. `Snapshot` is a lock-safe copy for JSON serialisation
`modem/pool.go`	Thread-safe slice of modems. `proxy3Mu` ensures concurrent rotations don't write `/etc/3proxy/3proxy.cfg` simultaneously
`modem/rotate.go`	The main rotation state machine. Manages modem status transitions (ONLINE → ROTATING → ONLINE/OFFLINE). Calls Huawei API, then boot/LTE wait, then network recovery
`modem/network.go`	Linux network operations: `dhclient`, `ip addr`, `ip route`, `ip rule`. `discoverInterface()` retries 5× with 2s sleep to handle DHCP latency
`modem/huawei.go`	Huawei HiLink XML API: session token fetch, reboot command, LTE status poll. All authenticated with Cookie + `__RequestVerificationToken` headers
`modem/proxy3.go`	Generates `3proxy.cfg` from current pool state. Skips `systemctl restart 3proxy` if config bytes are identical (Pi's DHCP renews same lease, so config rarely changes)
`haproxy/haproxy.go`	Talks to HAProxy via Unix socket at `/run/haproxy/admin.sock`. `GetStats()` parses CSV `show stat` output. Runtime add/remove servers without reload
`scheduler/scheduler.go`	The escalator: only rotates modems that earned it by serving requests. FIFO fairness. 5-minute idle drain
`api/server.go`	Registers all HTTP routes, defines `authOK()` (checks `Authorization: Bearer <key>`)
`api/handlers.go`	Handler implementations. `saveConfig()` persists pool state to `config.json` and regenerates `haproxy.cfg`

Configuration

`config.json`

{
  "rotation_interval_minutes": 30,
  "idle_timeout_minutes": 5,
  "modems": [
    {
      "id": "modem1",
      "gateway_ip": "192.168.69.1",
      "table_name": "modem1",
      "local_port": 8081
    },
    {
      "id": "modem2",
      "gateway_ip": "192.168.79.1",
      "table_name": "modem2",
      "local_port": 8082
    },
    {
      "id": "modem3",
      "gateway_ip": "192.168.59.1",
      "table_name": "modem3",
      "local_port": 8083
    }
  ]
}

Field	Type	Description
`rotation_interval_minutes`	int	Reserved for future time-based rotation. Currently unused by the scheduler (traffic-driven). Default: 30
`idle_timeout_minutes`	int	How long a modem can sit in the rotation queue without new traffic before being dropped. Default: 5
`modems[].id`	string	Unique identifier used everywhere (API paths, HAProxy server name, log prefix)
`modems[].gateway_ip`	string	The Huawei modem's internal web UI IP. Typically `192.168.x.1`
`modems[].table_name`	string	Linux routing table name. Must match `/etc/iproute2/rt_tables`. Usually same as `id`
`modems[].local_port`	int	The 3proxy port for this modem. Must match HAProxy backend and 3proxy.cfg

This file is auto-saved when modems are added or removed via the API.

Environment Variables

Stored in .env at the project working directory. Loaded at startup via godotenv.

Variable	Default	Description
`API_PORT`	`5000`	Port the Go daemon's HTTP API listens on
`API_SECRET_KEY`	(empty)	If set, all write endpoints require `Authorization: Bearer <key>`. If empty, no auth

Example .env:

API_PORT=5000
API_SECRET_KEY=my_super_secret_key_0321

API Reference

Base URL: http://<pi_ip>:5000

Auth header for protected endpoints: Authorization: Bearer <API_SECRET_KEY>

`GET /api/modems`

Also available as GET /api/status (legacy alias).

Returns real-time state of all modems in the pool.

Auth required: No

Response: 200 OK, application/json

[
  {
    "id": "modem1",
    "gateway_ip": "192.168.69.1",
    "table_name": "modem1",
    "local_port": 8081,
    "status": "ONLINE",
    "local_ip": "192.168.69.100",
    "auto_rotate": true,
    "last_rotated_at": "2025-04-24T15:58:07Z"
  },
  {
    "id": "modem2",
    "gateway_ip": "192.168.79.1",
    "table_name": "modem2",
    "local_port": 8082,
    "status": "ROTATING",
    "local_ip": "192.168.79.100",
    "auto_rotate": true,
    "last_rotated_at": "2025-04-24T15:56:20Z"
  }
]

Field	Values	Description
`status`	`ONLINE`, `OFFLINE`, `ROTATING`	Current modem state
`local_ip`	e.g. `192.168.69.100`	Pi's IP on the modem's subnet (discovered via DHCP after boot)
`auto_rotate`	`true`/`false`	Whether the scheduler will auto-rotate this modem
`last_rotated_at`	RFC3339 timestamp	When the last rotation completed successfully

`POST /api/modems`

Dynamically adds a new modem to the pool at runtime. No daemon restart needed.

Auth required: Yes

Request body (application/json):

{
  "id": "modem4",
  "gateway_ip": "192.168.49.1",
  "table_name": "modem4",
  "local_port": 8084
}

All four fields are required. id must be unique.

What happens internally:

Creates Modem struct and adds to pool.
Calls haproxy.AddServer() — registers the new backend in HAProxy at runtime (zero downtime, no reload).
Saves updated config.json.
Regenerates haproxy.cfg so the modem survives HAProxy restarts.
Triggers m.Recover() in a goroutine — sets up routing tables and adds modem to 3proxy config.

Response: 201 Created with the new modem's Snapshot JSON.

Error responses:

400 Bad Request — missing required fields
409 Conflict — id already exists
401 Unauthorized — bad/missing auth

`DELETE /api/modems/{id}`

Removes a modem from the pool at runtime.

Auth required: Yes

What happens internally:

Drains the backend in HAProxy (set server ... state drain), then disables it.
Removes from pool.
Saves updated config.json.
Regenerates 3proxy.cfg without this modem (restarts 3proxy if config changed).

Response: 204 No Content

Error responses:

404 Not Found — modem ID doesn't exist
401 Unauthorized

`POST /api/modems/{id}/rotate`

Also available as GET /api/rotate?id=<id> (legacy alias).

Manually triggers an IP rotation for one modem. Full reboot cycle.

Auth required: Yes

What happens: Runs m.Rotate(pool) in a background goroutine. Returns immediately.

Response: 202 Accepted, body: rotation triggered

Use case: When a scraper hits a CAPTCHA or ban and needs a new IP immediately rather than waiting for the scheduler.

`POST /api/modems/{id}/recover`

Also available as GET /api/recover?id=<id> (legacy alias).

Triggers network recovery without rebooting the modem hardware.

Auth required: Yes

What happens: Skips the Huawei reboot command. Runs:

recoverNetwork() — rediscovers local IP, rebuilds routing tables.
Regenerate3Proxy() — updates 3proxy config if needed.
Sets status to ONLINE.

Response: 202 Accepted, body: recovery triggered

Use cases:

Modem was physically unplugged and re-plugged (hot-plug event).
Modem is in a "zombie" state where it has an IP but routing is broken.
Pi rebooted and routing tables were lost (they don't survive reboots without this).
After manually bringing up a new interface.

`POST /api/modems/{id}/rotation/enable`

Enables automatic rotation for one modem (sets auto_rotate = true).

Auth required: Yes

Response: 200 OK, body: auto-rotation enabled for modem1

`POST /api/modems/{id}/rotation/disable`

Disables automatic rotation for one modem (sets auto_rotate = false). The modem still serves traffic; it just won't be added to the rotation queue by the scheduler. Manual rotation via the API still works.

Auth required: Yes

Response: 200 OK, body: auto-rotation disabled for modem1

`POST /api/rotation/enable`

Enables automatic rotation for all modems in the pool.

Auth required: Yes

Response: 200 OK, body: auto-rotation enabled for all modems

`POST /api/rotation/disable`

Disables automatic rotation for all modems. Useful for maintenance windows or when you need IP stability for a period.

Auth required: Yes

Response: 200 OK, body: auto-rotation disabled for all modems

System Files on the Pi

Path	Managed by	Description
`/etc/3proxy/3proxy.cfg`	Go daemon	Auto-regenerated after rotation if IPs change
`/etc/haproxy/haproxy.cfg`	Go daemon	Regenerated on modem add/remove
`/run/haproxy/admin.sock`	HAProxy	Unix socket for runtime control
`/etc/systemd/system/proxy-rotator.service`	Manual	systemd unit
`/etc/iproute2/rt_tables`	Manual	Must have entries for each modem's table name
`/home/tohfaakib/proxy-rotator/proxy-rotator`	Deployed binary	The compiled Go daemon
`/home/tohfaakib/proxy-rotator/config.json`	Daemon + API	Runtime state
`/home/tohfaakib/proxy-rotator/.env`	Manual	Secrets

`/etc/3proxy/3proxy.cfg` format

nserver 8.8.8.8
nserver 1.1.1.1
timeouts 1 5 30 60 180 1800 15 60
flush

auth none
allow *

proxy -n -p8081 -i127.0.0.1 -e192.168.69.100
proxy -n -p8082 -i127.0.0.1 -e192.168.79.100
proxy -n -p8083 -i127.0.0.1 -e192.168.59.100

-p<port> — listen port for this modem's proxy
-i127.0.0.1 — bind listener to localhost (HAProxy connects here)
-e<local_ip> — bind outbound connections to this modem's local IP, forcing traffic through its routing table

`/etc/iproute2/rt_tables`

Must have a numbered entry for each modem's table name:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
100     modem1
101     modem2
102     modem3

HAProxy config (`/etc/haproxy/haproxy.cfg`)

frontend proxy_pool
    bind *:41234
    mode tcp
    default_backend proxy_nodes

backend proxy_nodes
    mode tcp
    balance roundrobin
    server modem1 127.0.0.1:8081 check
    server modem2 127.0.0.1:8082 check
    server modem3 127.0.0.1:8083 check

TCP mode is critical — HAProxy must not interpret or re-encode HTTP traffic.

Deploying to the Pi

The Go daemon runs on the Pi (ARM64). Code is developed on a Mac (amd64). Cross-compile and SCP:

# Cross-compile for ARM64 Linux (Raspberry Pi 4)
cd /Users/tohfaakib/GolandProjects/proxy-rotator
GOOS=linux GOARCH=arm64 go build -o proxy-rotator .

# Copy binary to Pi
scp proxy-rotator tohfaakib@<pi_ip>:/home/tohfaakib/proxy-rotator/proxy-rotator

# Restart the service on Pi
ssh tohfaakib@<pi_ip> "sudo systemctl restart proxy-rotator"

# Watch logs
ssh tohfaakib@<pi_ip> "sudo journalctl -u proxy-rotator -f"

First-Time Pi Setup

1. Install dependencies

sudo apt update
sudo apt install -y haproxy 3proxy dhclient iproute2

2. Prevent NetworkManager from managing modem interfaces

NetworkManager can interfere with the manual DHCP management the daemon does. Tell it to ignore the modem ethernet interfaces:

sudo bash -c 'cat > /etc/NetworkManager/conf.d/99-unmanaged-modems.conf << EOF
[keyfile]
unmanaged-devices=interface-name:eth1;interface-name:eth2;interface-name:eth3;interface-name:eth4;interface-name:eth5
EOF'
sudo systemctl restart NetworkManager

3. Configure routing tables

Add entries to /etc/iproute2/rt_tables for each modem (see format above).

4. Copy system configs

# Copy from this repo's system-configs/
sudo cp system-configs/haproxy.cfg /etc/haproxy/haproxy.cfg
sudo cp system-configs/3proxy.cfg /etc/3proxy/3proxy.cfg
sudo cp system-configs/proxy-rotator.service /etc/systemd/system/

sudo systemctl daemon-reload
sudo systemctl enable haproxy 3proxy proxy-rotator
sudo systemctl start haproxy 3proxy

5. Deploy the binary

mkdir -p /home/tohfaakib/proxy-rotator
# Copy binary, config.json, and .env (see Deploying section)

6. Start the daemon

sudo systemctl start proxy-rotator
sudo journalctl -u proxy-rotator -f

The daemon waits 30 seconds on startup (ExecStartPre=/bin/sleep 30) for the USB bus to enumerate all modems before running.

7. Verify

# Check all modems are ONLINE
curl http://localhost:5000/api/modems

# Test a request through the proxy
curl -x http://<pi_ip>:41234 https://httpbin.org/ip

Adding a New Modem

Option A: API (hot-add, no restart)

curl -X POST http://<pi_ip>:5000/api/modems \
  -H "Authorization: Bearer my_super_secret_key_0321" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "modem4",
    "gateway_ip": "192.168.49.1",
    "table_name": "modem4",
    "local_port": 8084
  }'

The API call:

Adds modem to the live pool.
Registers the new backend in HAProxy at runtime (no reload needed).
Saves config.json so it survives restarts.
Triggers Recover() to set up routing tables and update 3proxy.

Prerequisites before calling the API:

a. Add routing table entry on the Pi:

echo "103     modem4" | sudo tee -a /etc/iproute2/rt_tables

b. Plug in the new modem and verify it gets an interface:

ip -o link show | grep eth

c. Verify the modem's gateway IP is reachable:

curl http://192.168.49.1/api/webserver/SesTokInfo

Option B: Edit `config.json` + restart

Edit config.json to add the new modem entry, then:

sudo systemctl restart proxy-rotator

This works but requires a service restart (brief downtime during startup rotations).

Removing a Modem

Option A: API (hot-remove, no restart)

curl -X DELETE http://<pi_ip>:5000/api/modems/modem4 \
  -H "Authorization: Bearer my_super_secret_key_0321"

The API call:

Drains the backend in HAProxy (waits for active connections to finish).
Disables the server in HAProxy.
Removes from pool and saves config.json.
Updates 3proxy config.

Option B: Edit `config.json` + restart

Remove the modem entry from config.json, then:

sudo systemctl restart proxy-rotator

Troubleshooting

Rotation times out: "modem did not connect to LTE within 60s"

Check: Is the modem getting a session token?

curl http://192.168.69.1/api/webserver/SesTokInfo

Should return XML with SesInfo and TokInfo. If it fails, the modem web server isn't up yet.

Check: What is the actual LTE status?

# Get a token first
SESSION=$(curl -s http://192.168.69.1/api/webserver/SesTokInfo | grep -oP '(?<=<SesInfo>).*(?=</SesInfo>)')
TOKEN=$(curl -s http://192.168.69.1/api/webserver/SesTokInfo | grep -oP '(?<=<TokInfo>).*(?=</TokInfo>)')

# Check LTE status (901=connected, 902=disconnected)
curl -s http://192.168.69.1/api/monitoring/status \
  -H "Cookie: $SESSION" \
  -H "__RequestVerificationToken: $TOKEN"

3proxy down / no response through HAProxy

sudo systemctl status 3proxy
sudo systemctl restart 3proxy

Also check if 3proxy has the right local IPs:

cat /etc/3proxy/3proxy.cfg
ip -4 -o addr show | grep eth

The IPs in 3proxy.cfg must match what the Pi actually has on each eth interface.

Routing table broken (traffic going out wrong interface)

Trigger a recover for the affected modem:

curl -X POST http://localhost:5000/api/modems/modem1/recover \
  -H "Authorization: Bearer my_super_secret_key_0321"

Check routing tables manually:

ip route show table modem1
ip rule show

HAProxy not forwarding (502 errors)

Check HAProxy backend status:

echo "show stat" | sudo socat - UNIX-CONNECT:/run/haproxy/admin.sock | cut -d',' -f1,2,18

Should show all proxy_nodes servers as UP. If a server shows MAINT, the daemon may have crashed mid-rotation:

echo "set server proxy_nodes/modem1 state ready" | sudo socat - UNIX-CONNECT:/run/haproxy/admin.sock

dhclient processes accumulating

The daemon uses dhclient -1 (exit after one lease). If old processes accumulated:

sudo pkill dhclient

Then trigger recover on each modem to re-establish IPs:

curl -X POST http://localhost:5000/api/modems/modem1/recover -H "Authorization: Bearer ..."

Check daemon logs

sudo journalctl -u proxy-rotator -f --since "10 minutes ago"

Key log lines to look for:

[modem1] Rotation complete. — success
[modem1] LTE connect wait failed — Huawei API issue
[modem1] Boot wait failed — modem didn't come up within 60s
[scheduler] Rotating modem2 for fresh IP (queue len after: 1) — scheduler fired

Planned Features

Proxy Authentication

Currently 3proxy is open (auth none / allow *). The plan is to add per-user credentials once the client-facing website is built.

The 3proxy config template in modem/proxy3.go already has a placeholder:

# AUTH: replace the two lines below with per-user credentials once the website is integrated.
# Example:
#   auth strong
#   users alice:CL:s3cr3t bob:CL:p4ss
#   allow alice
#   allow bob
auth none
allow *

Auth blocks go above the proxy -n -p... lines. The website would POST credentials to an API endpoint that writes them to this section and restarts 3proxy.

Key Design Decisions

Decision	Why
Huawei HiLink XML API for reboot	No SSH on modems. HiLink exposes a documented XML REST API at the gateway IP
SIGHUP → `systemctl restart 3proxy`	3proxy has no SIGHUP reload handler. SIGHUP kills it without restart
Skip 3proxy restart when config unchanged	Pi's DHCP renews the same lease across reboots. Restart is disruptive (interrupts all proxied connections briefly)
HAProxy Unix socket (not config reload)	Runtime state changes (maintenance, add/remove) are instant via socket. Config reload takes ~100ms and closes active connections
HAProxy maintenance before rotation	Health checks have a 2-3s polling interval. Without maintenance mode, HAProxy would still route requests to a rebooting modem for up to 3s, causing connection failures
`dhclient -1` flag	Without `-1`, dhclient daemonizes and stays resident. Running it every 3s during the boot polling loop would accumulate dozens of zombie processes
15s sleep before polling	Huawei modems take 12-15s to physically power down. Polling before then gets false positives (web server still up from previous session)
`proxy3Mu` serialiser on Pool	Multiple modems can rotate concurrently during staggered startup. Each calls `Regenerate3Proxy()`. Without the mutex, concurrent writes corrupt the config file
TCP mode on HAProxy	HTTP mode would re-encode headers and break HTTPS tunnels (CONNECT method). TCP mode passes bytes through unchanged
Table-per-modem Linux routing	Without per-modem routing tables, the kernel routes outbound traffic through the default route (Pi's home interface), leaking the Pi's real IP
FIFO rotation queue	Simple round-robin check would always pick modem1 first. FIFO ensures fair turn-taking across all modems
One rotation at a time	Prevents two modems going into maintenance simultaneously, which would leave zero HAProxy backends
30s `ExecStartPre` sleep	USB bus takes 20-30s to enumerate all modems after Pi boot. Starting the daemon before that means `eth1`/`eth2`/`eth3` don't exist yet

Name		Name	Last commit message	Last commit date
.idea		.idea
api		api
config		config
haproxy		haproxy
modem		modem
scheduler		scheduler
system-configs		system-configs
.gitignore		.gitignore
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Latest commit

History

Folders and files

Repository files navigation

Smart Mobile Proxy Farm

Table of Contents

What This Is

Hardware Setup

System Architecture

The Three Layers

Linux Routing Tables

How a Request Flows

How Rotation Works

HAProxy Maintenance Mode

Scheduler Logic

Project File Structure

What each file does

Configuration

config.json

Environment Variables

API Reference

GET /api/modems

POST /api/modems

DELETE /api/modems/{id}

POST /api/modems/{id}/rotate

POST /api/modems/{id}/recover

POST /api/modems/{id}/rotation/enable

POST /api/modems/{id}/rotation/disable

POST /api/rotation/enable

POST /api/rotation/disable

System Files on the Pi

/etc/3proxy/3proxy.cfg format

/etc/iproute2/rt_tables

HAProxy config (/etc/haproxy/haproxy.cfg)

Deploying to the Pi

First-Time Pi Setup

1. Install dependencies

2. Prevent NetworkManager from managing modem interfaces

3. Configure routing tables

4. Copy system configs

5. Deploy the binary

6. Start the daemon

7. Verify

Adding a New Modem

Option A: API (hot-add, no restart)

Option B: Edit config.json + restart

Removing a Modem

Option A: API (hot-remove, no restart)

Option B: Edit config.json + restart

Troubleshooting

Rotation times out: "modem did not connect to LTE within 60s"

3proxy down / no response through HAProxy

Routing table broken (traffic going out wrong interface)

HAProxy not forwarding (502 errors)

dhclient processes accumulating

Check daemon logs

Planned Features

Proxy Authentication

Key Design Decisions

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`config.json`

`GET /api/modems`

`POST /api/modems`

`DELETE /api/modems/{id}`

`POST /api/modems/{id}/rotate`

`POST /api/modems/{id}/recover`

`POST /api/modems/{id}/rotation/enable`

`POST /api/modems/{id}/rotation/disable`

`POST /api/rotation/enable`

`POST /api/rotation/disable`

`/etc/3proxy/3proxy.cfg` format

`/etc/iproute2/rt_tables`

HAProxy config (`/etc/haproxy/haproxy.cfg`)

Option B: Edit `config.json` + restart

Option B: Edit `config.json` + restart