Skip to content

Keep device runtime state out of PostgreSQL #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yellowman merged 10 commits into from
Sep 19, 2026
Merged

Keep device runtime state out of PostgreSQL #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
75d4b80
Keep device runtime status out of PostgreSQL
yellowman Sep 19, 2026
b000bf2
Stop STA association polls from rewriting inventory
yellowman Sep 19, 2026
d894fda
Fix STA poller function boundary
yellowman Sep 19, 2026
e5f8a88
Stop serving stale device status from PostgreSQL
yellowman Sep 19, 2026
695d9d3
Use durable mismatch records instead of DB status
yellowman Sep 19, 2026
e607566
Build reports from live device status
yellowman Sep 19, 2026
05a9272
Document inventory-only device persistence
yellowman Sep 19, 2026
cfa79d4
Persist exact coarse last-seen timestamps
yellowman Sep 19, 2026
0af2104
Clarify coarse last-seen semantics
yellowman Sep 19, 2026
f2b5383
Fix runtime-state review findings
yellowman Sep 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 51 additions & 52 deletions SPEC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -804,64 +804,63 @@ func (p *Poller) discoverSTAs(apIP string, creds Credentials) ([]STAInfo, error)
// 4. Sleep until next interval
```

### State-Transition Database Pattern
### Inventory-Only Device Database Pattern

**Principle**: Database writes only occur on state changes, not on every poll cycle.
**Principle**: the `devices` table is durable inventory/configuration, not a telemetry store.

This pattern dramatically reduces database I/O - with 5000 devices at 30-second polls, this eliminates ~600,000 unnecessary writes per hour.
Live availability, status reasons, uptime, CPU/RAM/temperature, radio metrics, peer
association state, counters, and other polling results live in the in-memory stats
store and are broadcast over WebSocket. A process restart intentionally starts device
status at `unknown` until fresh polls arrive; stale database status must never be
used as a fallback.

#### What Triggers a Database Write
#### What Triggers a `devices` Row Write

| Event | Database Action |
|-------|-----------------|
| Device comes online (was offline/unknown) | `UPDATE status='online', last_seen=NOW()` |
| Device goes offline (was online) | `UPDATE status='offline', last_seen=NOW()` |
| Hostname changes | `UPDATE hostname=?` |
| Firmware changes | `UPDATE firmware=?, firmware_version=?` |
| Other static info changes | Update via `IS DISTINCT FROM` check |
| Device/IP/MAC identity changes | Update the changed inventory field |
| Parent AP / SSID / role changes | Update durable hierarchy/configuration |
| Hostname/model/platform/flavor/firmware changes | Update only when `IS DISTINCT FROM` the stored value |
| Site / operator configuration changes | Persist the operator-owned value |
| Device first discovered | Insert its inventory row |
| Coarse availability checkpoint | Batch-update `last_seen` at most about once per hour |

#### What Does NOT Trigger a Database Write
#### What Does NOT Trigger a `devices` Row Write

| Event | Where Data Lives |
|-------|------------------|
| Successful poll (device already online) | Memory store only |
| Stats refresh (CPU, memory, signal, rates) | Memory store only |
| Peer list update | Memory store only |
| Real-time counters | Memory store only |

#### Implementation Pattern

```go
// Update memory store - returns true if state changed (offline->online)
becameOnline := store.Update(ip, deviceStats)

// Only write to DB on state transition
if becameOnline {
db.Exec(`UPDATE devices SET status = 'online', last_seen = NOW() WHERE id = $1`, deviceID)
}

// For failures - SetOffline returns true if state changed (online->offline)
becameOffline := store.SetOffline(ip, errorMessage)
if becameOffline {
db.Exec(`UPDATE devices SET status = 'offline', last_seen = NOW() WHERE id = $1`, deviceID)
}
```

#### Periodic Batch Sync

For crash recovery, a periodic batch sync runs every ~10 minutes:
- Syncs `last_seen` timestamps to database
- Ensures `status` column matches memory state
- Single bulk query instead of per-device writes

```go
// Every 20 poll cycles (~10 min at 30s interval)
if cleanupCounter%20 == 0 {
p.batchSyncToDB()
}
```

This pattern applies uniformly to all device types: Wave, LTU, airMAX AC/M, AirFiber.
| Online/offline/unknown transition | In-memory stats store + WebSocket |
| Status reason / poll error | In-memory stats store + WebSocket |
| Successful poll | In-memory stats store |
| CPU, RAM, temperature, uptime | In-memory stats store |
| Signal, rates, capacity, MCS, airtime | In-memory stats store |
| Peer association/disassociation | In-memory stats store + WebSocket |
| Firmware upgrade runtime state | Job records + in-memory/WebSocket state |

The legacy `devices.status` and `devices.status_reason` columns remain only for schema
compatibility. Polling code must not read or write them.

#### Coarse Last-Available Persistence

`last_seen` is the one polling-derived value retained in `devices`. It is deliberately
low-frequency: an hourly wall-clock task takes the in-memory last-seen snapshot and
performs one batch update using each device's actual in-memory timestamp. Rows whose
durable timestamp was updated within roughly the previous hour are skipped; an offline
device can therefore eventually persist its exact final availability time without any
per-poll writes.

This keeps a useful "last available" marker across restarts without turning PostgreSQL
or its WAL into a 30-second telemetry sink. The live API may return a newer in-memory
`last_seen` while the process is running without persisting it immediately.

This rule applies uniformly to Wave, LTU, airMAX AC/M, and AirFiber.

**GPS persistence:** poller-learned AirMAX GPS coordinates are fill-once inventory.
Once durable `gps_lat/gps_lon` are populated, ordinary polling does not rewrite
them for GPS jitter or physical relocation. While the process is running, the API
overlays current live GPS from memory. After a restart, the durable coordinates
are shown until a successful poll provides the live overlay. Durable location
changes should be explicit inventory actions rather than telemetry side effects.

### Device Identification

Expand All @@ -887,7 +886,7 @@ To prevent this, the poller enforces a **canonical MAC** per poll job:
- Other observed MACs are informational and are included in logs/context.
- If the device returns a MAC candidate set and **none match the expected/job MAC**, wavecontrol treats this as a data quality issue:
- Do **not** apply the stats/peers update to the expected device
- Mark the expected device `status=unknown`, `status_reason=mac_mismatch`
- Mark the expected device unknown with reason `mac_mismatch` **in memory/WebSocket state only**
- Do **not** advance `last_seen` for the expected device (we did not see it)
- Persist `device_identity_mismatches` with expected MAC, observed MAC candidates, observed IP, source, timestamp, and last error
- Broadcast a WebSocket patch containing `identity_mismatch` context for the selected detail pane
Expand All @@ -900,12 +899,12 @@ A MAC mismatch is never auto-healed. AP or directly managed STA replacement must
The operator may resolve a confirmed AP or managed STA swap through `POST /api/wavecontrol/devices/{id}/learn-mac`. The server must:

1. Require editor/administrator permission.
2. Verify the device is still in `status_reason=mac_mismatch`.
3. Verify a persisted identity-mismatch row exists for the device.
2. Verify a persisted `device_identity_mismatches` row exists for the device.
3. Verify its expected MAC still matches the current inventory MAC.
4. Verify the requested new MAC is one of the observed MAC candidates.
5. Verify the observed IP still matches the device row IP.
6. Reject the request if the observed MAC already exists on another device row.
7. Update the device MAC, clear mismatch state, remove stale in-memory stats, write changelog, broadcast a WebSocket patch, and queue a refresh. For AP rows only, rewrite child `parent_mac` references from the old AP MAC to the new AP MAC; for STA rows, keep the AP association unchanged.
7. Update the device MAC, clear the mismatch record and stale in-memory stats, write changelog, broadcast a WebSocket patch, and queue a refresh. For AP rows only, rewrite child `parent_mac` references from the old AP MAC to the new AP MAC; for STA rows, keep the AP association unchanged.

#### Identification Flow

Expand Down
82 changes: 41 additions & 41 deletions cmd/server/api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,7 +421,7 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {
d.hostname, d.product, d.model, d.platform, d.flavor, d.firmware, d.firmware_version,
d.parent_id,
lower(d.parent_mac) AS parent_mac,
d.status, d.status_reason, d.last_seen, d.role, d.managed, d.alertable, d.alert_silenced_until, d.alert_notes, d.ssid, d.frequency, d.channel_width, d.gps_lat, d.gps_lon,
d.last_seen, d.role, d.managed, d.alertable, d.alert_silenced_until, d.alert_notes, d.ssid, d.frequency, d.channel_width, d.gps_lat, d.gps_lon,
d.antenna_model, d.antenna_override, d.antenna_azimuth_deg, d.antenna_downtilt_deg, d.antenna_electrical_downtilt_deg, d.antenna_beamwidth_h_deg, d.antenna_beamwidth_v_deg,
d.radius_m, d.tech, d.down_mbps, d.up_mbps, d.latency_ms, d.bizres,
d.site_id, s.name as site_name, r.name as region_name
Expand All @@ -441,7 +441,7 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {

for rows.Next() {
var id int64
var mac, ipAddr, hostname, product, model, platform, flavor, fw, fwVer, parentMAC, dbStatus, dbStatusReason, role, ssid sql.NullString
var mac, ipAddr, hostname, product, model, platform, flavor, fw, fwVer, parentMAC, role, ssid sql.NullString
var managed, alertable bool
var alertSilencedUntil sql.NullTime
var alertNotes sql.NullString
Expand All @@ -457,7 +457,7 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {
var tech sql.NullInt64
var bizres sql.NullString
if rows.Scan(&id, &mac, &ipAddr, &hostname, &product, &model, &platform, &flavor, &fw, &fwVer,
&parentID, &parentMAC, &dbStatus, &dbStatusReason, &lastSeen, &role, &managed, &alertable, &alertSilencedUntil, &alertNotes, &ssid, &frequency, &channelWidth, &gpsLat, &gpsLon,
&parentID, &parentMAC, &lastSeen, &role, &managed, &alertable, &alertSilencedUntil, &alertNotes, &ssid, &frequency, &channelWidth, &gpsLat, &gpsLon,
&antennaModel, &antennaOverride, &antennaAzimuthDeg, &antennaDowntiltDeg, &antennaElectricalDowntiltDeg, &antennaBeamH, &antennaBeamV,
&radiusM, &tech, &downMbps, &upMbps, &latencyMS, &bizres,
&siteID, &siteName, &regionName) != nil {
Expand All @@ -469,12 +469,12 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {
d := map[string]any{"id": id, "mac": mac.String, "ip_address": ipHost, "hostname": hostname.String,
"product": product.String, "model": model.String, "platform": platform.String, "flavor": flavor.String,
"firmware": fw.String, "firmware_version": fwVer.String,
// Persisted DB status (source of truth when we have no live stats)
"db_status": dbStatus.String,
"db_status_reason": dbStatusReason.String,
// Live/computed status defaults to DB status and may be overridden below
"status": dbStatus.String,
"status_reason": dbStatusReason.String,
// Runtime status is memory-only. db_status remains a browser
// compatibility alias and is never loaded from PostgreSQL.
"db_status": "unknown",
"db_status_reason": "",
"status": "unknown",
"status_reason": "",
"role": role.String,
"managed": managed,
"alertable": alertable,
Expand Down Expand Up @@ -572,15 +572,12 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {

if liveStats != nil {
d["online"] = liveStats.Online
// Prefer live tri-state status over DB status
d["status"] = string(liveStats.Status)
if liveStats.StatusReason != "" {
d["status_reason"] = liveStats.StatusReason
} else {
// Clear stale DB reasons when device is currently healthy
if liveStats.Online {
d["status_reason"] = ""
}
d["db_status"] = string(liveStats.Status)
d["status_reason"] = liveStats.StatusReason
d["db_status_reason"] = liveStats.StatusReason
if !liveStats.LastSeen.IsZero() {
d["last_seen"] = liveStats.LastSeen
}
d["uptime"] = liveStats.Uptime
d["peer_count"] = liveStats.PeerCount
Expand Down Expand Up @@ -639,9 +636,6 @@ func (a *API) ListDevices(w http.ResponseWriter, r *http.Request) {
if liveStats.Config != nil {
d["config"] = liveStats.Config
}
} else {
// No live stats - set online based on db_status for consistent client-side checking
d["online"] = dbStatus.String == "online"
}
devices = append(devices, d)
}
Expand Down Expand Up @@ -1118,7 +1112,6 @@ func (a *API) upsertDiscoveredDevice(device *DeviceInfo, ip, username, password
site_id = COALESCE($11, devices.site_id),
managed = TRUE,
alertable = TRUE,
status = 'online',
last_seen = NOW(),
username = $12,
password = $13
Expand All @@ -1130,8 +1123,8 @@ func (a *API) upsertDiscoveredDevice(device *DeviceInfo, ip, username, password
}
case selErr == sql.ErrNoRows:
err := a.DB.QueryRow(`
INSERT INTO devices (mac, ip_address, hostname, product, model, platform, flavor, firmware, firmware_version, site_id, managed, alertable, status, last_seen, username, password)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, TRUE, TRUE, 'online', NOW(), $11, $12)
INSERT INTO devices (mac, ip_address, hostname, product, model, platform, flavor, firmware, firmware_version, site_id, managed, alertable, last_seen, username, password)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, TRUE, TRUE, NOW(), $11, $12)
RETURNING id
`, mac, ip, hostname, product, model, platform, flavor, firmware, firmwareVersion, siteID, username, storedPassword).Scan(&id)
if err != nil {
Expand Down Expand Up @@ -1404,7 +1397,7 @@ func (a *API) loadCredentials() (apCreds, staCreds []Credential) {

func (a *API) GetDevice(w http.ResponseWriter, r *http.Request) {
id, _ := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
var mac, ipAddr, hostname, product, model, platform, flavor, fw, fwVer, parentMAC, dbStatus, dbStatusReason sql.NullString
var mac, ipAddr, hostname, product, model, platform, flavor, fw, fwVer, parentMAC sql.NullString
var managed, alertable bool
var alertSilencedUntil sql.NullTime
var alertNotes sql.NullString
Expand All @@ -1418,18 +1411,18 @@ func (a *API) GetDevice(w http.ResponseWriter, r *http.Request) {
var lastSeen sql.NullTime
err := a.DB.QueryRow(`SELECT mac, host(ip_address), hostname, product, model, platform, flavor, firmware, firmware_version,
managed, alertable, alert_silenced_until, alert_notes,
parent_id, parent_mac, status, status_reason, last_seen,
parent_id, parent_mac, last_seen,
antenna_model, antenna_override, antenna_azimuth_deg, antenna_downtilt_deg, antenna_electrical_downtilt_deg, antenna_beamwidth_h_deg, antenna_beamwidth_v_deg,
radius_m, tech, down_mbps, up_mbps, latency_ms, bizres
FROM devices WHERE id = $1`, id).
Scan(&mac, &ipAddr, &hostname, &product, &model, &platform, &flavor, &fw, &fwVer, &managed, &alertable, &alertSilencedUntil, &alertNotes, &parentID, &parentMAC, &dbStatus, &dbStatusReason, &lastSeen,
Scan(&mac, &ipAddr, &hostname, &product, &model, &platform, &flavor, &fw, &fwVer, &managed, &alertable, &alertSilencedUntil, &alertNotes, &parentID, &parentMAC, &lastSeen,
&antennaModel, &antennaOverride, &antennaAzimuthDeg, &antennaDowntiltDeg, &antennaElectricalDowntiltDeg, &antennaBeamH, &antennaBeamV,
&radiusM, &tech, &downMbps, &upMbps, &latencyMS, &bizres)
if err == sql.ErrNoRows {
http.Error(w, "not found", 404)
return
}
d := map[string]any{"id": id, "mac": mac.String, "ip_address": ipAddr.String, "hostname": hostname.String, "product": product.String, "model": model.String, "platform": platform.String, "flavor": flavor.String, "firmware": fw.String, "firmware_version": fwVer.String, "db_status": dbStatus.String, "db_status_reason": dbStatusReason.String, "status": dbStatus.String, "status_reason": dbStatusReason.String, "managed": managed, "alertable": alertable, "alert_notes": alertNotes.String}
d := map[string]any{"id": id, "mac": mac.String, "ip_address": ipAddr.String, "hostname": hostname.String, "product": product.String, "model": model.String, "platform": platform.String, "flavor": flavor.String, "firmware": fw.String, "firmware_version": fwVer.String, "db_status": "unknown", "db_status_reason": "", "status": "unknown", "status_reason": "", "managed": managed, "alertable": alertable, "alert_notes": alertNotes.String}

// Optional antenna modeling fields
d["antenna_model"] = antennaModel.String
Expand Down Expand Up @@ -1494,6 +1487,14 @@ func (a *API) GetDevice(w http.ResponseWriter, r *http.Request) {
}
if liveStats != nil {
d["live_stats"] = liveStats
d["online"] = liveStats.Online
d["status"] = string(liveStats.Status)
d["db_status"] = string(liveStats.Status)
d["status_reason"] = liveStats.StatusReason
d["db_status_reason"] = liveStats.StatusReason
if !liveStats.LastSeen.IsZero() {
d["last_seen"] = liveStats.LastSeen
}
}
a.attachIdentityMismatch(d, id)
writeJSON(w, d)
Expand Down Expand Up @@ -6092,13 +6093,14 @@ func (a *API) DryRunOperation(w http.ResponseWriter, r *http.Request) {
Compatible: true,
}

// Get device info from database
var ip, hostname, product, firmware, flavor, status string
// Get durable inventory only. Runtime eligibility comes exclusively
// from the in-memory stats store.
var ip, mac, hostname, product, firmware, flavor string
err := a.DB.QueryRow(`
SELECT host(ip_address), COALESCE(hostname, ''), COALESCE(product, ''),
COALESCE(firmware, ''), COALESCE(flavor, ''), COALESCE(status, 'unknown')
SELECT host(ip_address), lower(mac), COALESCE(hostname, ''), COALESCE(product, ''),
COALESCE(firmware, ''), COALESCE(flavor, '')
FROM devices WHERE id = $1
`, deviceID).Scan(&ip, &hostname, &product, &firmware, &flavor, &status)
`, deviceID).Scan(&ip, &mac, &hostname, &product, &firmware, &flavor)

if err != nil {
result.Compatible = false
Expand All @@ -6112,15 +6114,13 @@ func (a *API) DryRunOperation(w http.ResponseWriter, r *http.Request) {
result.CurrentVer = firmware
result.Flavor = flavor

// Check device is online using stats store (real-time) or database status (fallback)
online := false
if stats := a.Stats.Get(ip); stats != nil {
online = stats.Online
} else {
online = status == "online"
}

if !online {
// No live sample means the device is not eligible yet; never revive
// stale status from the inventory row after a restart.
live := a.Stats.GetByMAC(mac)
if live == nil {
result.Compatible = false
result.Issues = append(result.Issues, "Device has no live status")
} else if !live.Online {
result.Compatible = false
result.Issues = append(result.Issues, "Device is offline")
}
Expand Down
16 changes: 4 additions & 12 deletions cmd/server/identity_mismatch_api.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,13 +130,13 @@ func (a *API) LearnDeviceMAC(w http.ResponseWriter, r *http.Request) {
}
defer tx.Rollback()

var oldMAC, ipAddr, statusReason, role, hostname sql.NullString
var oldMAC, ipAddr, role, hostname sql.NullString
var parentID sql.NullInt64
err = tx.QueryRowContext(r.Context(), `
SELECT lower(mac), host(ip_address), status_reason, role, hostname, parent_id
SELECT lower(mac), host(ip_address), role, hostname, parent_id
FROM devices
WHERE id = $1
FOR UPDATE`, id).Scan(&oldMAC, &ipAddr, &statusReason, &role, &hostname, &parentID)
FOR UPDATE`, id).Scan(&oldMAC, &ipAddr, &role, &hostname, &parentID)
if err == sql.ErrNoRows {
http.Error(w, "device not found", http.StatusNotFound)
return
Expand Down Expand Up @@ -167,14 +167,6 @@ func (a *API) LearnDeviceMAC(w http.ResponseWriter, r *http.Request) {
return
}

if !strings.EqualFold(statusReason.String, "mac_mismatch") {
writeJSONStatus(w, http.StatusConflict, map[string]any{
"error": "device_not_in_mac_mismatch",
"status_reason": statusReason.String,
})
return
}

oldCanon, err := normalizeCanonicalMAC(oldMAC.String)
if err != nil {
writeJSONStatus(w, http.StatusConflict, map[string]any{"error": "current_device_mac_invalid"})
Expand Down Expand Up @@ -252,7 +244,7 @@ func (a *API) LearnDeviceMAC(w http.ResponseWriter, r *http.Request) {

if _, err := tx.ExecContext(r.Context(), `
UPDATE devices
SET mac = $2, status = 'unknown', status_reason = NULL, updated_at = NOW()
SET mac = $2, updated_at = NOW()
WHERE id = $1`, id, newCanon); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
Expand Down
Loading
Loading