perf(server): stop the device heartbeat from blocking HOT updates

[otavio]

Every device presence heartbeat writes last_seen, and PostgreSQL disqualifies HOT whenever an
indexed column changes. devices_last_seen was a btree on exactly that column, so no heartbeat
could ever be a HOT update — each one rewrote the heap tuple and inserted into every index on
the table.

Measured on a 58,670-device deployment: ~2,244 row updates/second, 107.7 GB of WAL per day, a
heap bloated ~12× past the width of its rows, and autovacuum running continuously without keeping
up.

What this does

Migration	Change
017	drops devices_last_seen and devices_disconnected_at
018	ALTER TABLE devices SET (fillfactor = 85)
019	VACUUM (FULL, ANALYZE) devices

Plus wal_compression=lz4 in docker-compose.postgres.yml.

Order is load-bearing and follows from the numbering: the drops run before the rewrite so
VACUUM FULL never rebuilds indexes that are about to disappear, and 018 runs before 019
because VACUUM FULL honours fillfactor as it rewrites (the same 58,670 rows rebuild into
2,257 pages at the default and 2,667 at 85).

Verified in production

Each half was applied and measured independently, then reverted:

	baseline	+ wal_compression=lz4	+ indexes dropped
HOT ratio	0.0000%	0.0000%	97.2521%
WAL	107.7 GB/day	97.0 GB/day	31.7 GB/day
wal_records/s	12,203	12,204	3,044
postgres CPU (core-s/s)	0.326	0.264	0.205

The win shows up in wal_records, not wal_fpi: a non-HOT update emitted a heap record plus
index-tuple inserts into two btrees, where a HOT update emits one. wal_compression is separate
and additive at ~10% — full-page images are only ~36% of WAL volume here — and it lowered
postgres CPU by 19%, because compressing a page costs less than writing the extra bytes.

The trade

devices_last_seen did serve the device list's default ORDER BY last_seen DESC, which now sorts
over a sequential scan: ~121 ms on the bloated heap, ~24 ms once compacted — which is why
019 is part of this PR rather than a follow-up. No index on last_seen can coexist with HOT
(composite and partial alike), so this is structural: either the write side pays or the read side
does, and one endpoint at ~24 ms is much cheaper than 76 GB/day of WAL.

devices_disconnected_at goes along for free. Nothing filters or orders on it alone, only inside
the unselective online predicate; 0 scans in 66 days of counters.

Why 018 is needed

Production hit 97.25% HOT with fillfactor at the default, because a 12×-bloated heap already
holds all the free space HOT needs — so fillfactor is not required for HOT to work, contrary
to the original issue. It is required to keep HOT working once 019 compacts that space away.
Measured locally from a freshly compacted heap, six full passes over a 58,670-row table:

	HOT ratio	heap growth
default fillfactor	40.4%	4.7×
fillfactor = 85	85.1%	3.0×

Better HOT and a smaller table — reserving 15% up front costs less than letting the heap
rediscover the same slack by bloating. The autovacuum scale-factor knobs proposed alongside it
showed no material effect once fillfactor is set and are not included.

019 is the repo's first non-transactional migration

VACUUM cannot run inside a transaction block, so the file omits the .tx. suffix — and because
the pool runs in pgx QueryExecModeSimpleProtocol, where a multi-statement Exec is itself an
implicit transaction, every statement must sit alone between --bun:split markers. Neither
requirement is visible in the SQL, so TestNonTransactionalMigrations now enforces both for every
migration from here on.

Two further edges are handled rather than ignored:

• Timeouts bound the boot. Migrations run inline in Server.Setup before the listener binds.
  lock_timeout makes 019 fail fast rather than queue behind a long snapshot (a nightly logical
  backup, say). bun marks a migration applied before running it, so a failure there costs one
  crash-restart and leaves the table merely still bloated — degraded, not broken. Recover with
  psql -c 'VACUUM (FULL, ANALYZE) devices;'.
• The GUCs are reset explicitly. A non-transactional migration runs on one pooled bun.Conn
  that returns to the pool without a session reset, so SET lock_timeout would otherwise leak
  into application queries.

VACUUM FULL needs free space for a full copy of the table (~320 MB at the reference scale) and
holds ACCESS EXCLUSIVE for its duration, which is acceptable inside the upgrade's own restart
window. Worth a line in the release notes.

Testing

• TestNonTransactionalMigrations — scans every embedded migration for a statement PostgreSQL
  refuses inside a transaction and asserts it neither carries .tx. nor shares its
  --bun:split chunk. TestNonTransactionalDetection covers the guard itself, including prose
  that merely names a VACUUM.
• Full pg store suite green against a schema built from 001 through 019.
• End-to-end through bun's runner on the dev stack: 018 applies, reloptions becomes
  {fillfactor=85}, 019 compacts, three indexes remain, clean boot.

Fixes shellhub-io/team#197.

[github-actions]

Code Review

• Gathered PR context
• Reviewing with specialized agents
• Posting feedback

[otavio]

Follow-up #6884 is stacked on this branch — it drops three unused sessions indexes as migration 020, and is based here rather than on master so the migration numbering cannot collide. Merge this one first.

[github-actions]

Claude finished @otavio's task in 2m 22s —— View job

---

Code Review Complete

Reviewed 8 files across code quality, security, testing, language patterns, and architecture — no new issues found. The code looks good as-is.

A few things I checked closely and found correct:

• TestNonTransactionalMigrations scans every migration, including 001–016 and 018. stripSQLComments runs first, so the VACUUM FULL prose in the comment blocks of 018 and 019 is removed before keyword detection — no false-positive failures. Only the real VACUUM (FULL, ANALYZE) devices; statement in the non-.tx. file is matched.
• Session-GUC lifetime in 019. SET (not SET LOCAL) is the right choice for the non-transactional file, and the explicit RESET chunks prevent the timeouts leaking back into the pooled connection — consistent with the SET LOCAL used in the transactional 017.
• Test compiles — io/fs, strings, and the sqlMigrations embed var are all in scope.
• countStatements / findNonTransactionalStatement handle the split-across-lines and shared-chunk cases correctly, matching the table-driven expectations in TestNonTransactionalDetection.

The migration ordering (drops → fillfactor → vacuum), the lock_timeout/statement_timeout bounding of boot, and the wal_compression=lz4 pin are all well-reasoned and documented inline. Production + local measurements are included in the PR body.

To request another review round, comment /review.

--- · perf/device-heartbeat-hot-updates