feat(postgres): enable diagnostic extensions and a deployment tuning seam - #6886
Merged
Conversation
This was referenced Aug 11, 2026
gustavosbarreto
force-pushed
the
perf/drop-unused-session-indexes
branch
from
August 12, 2026 13:42
f9be886 to
84ff059
Compare
gustavosbarreto
force-pushed
the
perf/enable-pg-stat-statements
branch
from
August 12, 2026 13:42
2a1352d to
a1841fe
Compare
This was referenced Aug 12, 2026
otavio
force-pushed
the
perf/drop-unused-session-indexes
branch
from
August 12, 2026 21:39
84ff059 to
c12baa5
Compare
otavio
force-pushed
the
perf/enable-pg-stat-statements
branch
from
August 12, 2026 21:46
a1841fe to
43dd80a
Compare
otavio
changed the base branch from
master
to
perf/device-heartbeat-hot-updates
August 12, 2026 21:48
otavio
force-pushed
the
perf/enable-pg-stat-statements
branch
from
August 12, 2026 23:27
43dd80a to
c9ce2ba
Compare
otavio
changed the base branch from
perf/device-heartbeat-hot-updates
to
master
August 12, 2026 23:29
…seam Table-level counters can say which table is being scanned but never which query is scanning it, which is why the sequential-scan investigation could report 1.37 billion scans on a two-row `namespaces` table without naming a single statement. pg_stat_statements closes that gap; pgstattuple and pg_buffercache close the neighbouring ones, reporting bloat and in-page free space directly instead of inferring them from heap size over row count, and showing what actually occupies shared_buffers. All three already ship in postgres:18.0, so this is a preload flag plus a CREATE EXTENSION. Only pg_stat_statements needs shared_preload_libraries, which takes effect on restart rather than reload -- an upgrade recreates the stack, so it arrives with the version. compute_query_id is already at its "auto" default and activates itself once the library loads. Each CREATE is wrapped in a DO block that can only warn. Migrations run inline in Server.Setup before the listener binds and an error there is fatal, and none of these three is a trusted extension, so an unguarded CREATE EXTENSION would turn a database role without superuser into a server that cannot boot. Verified both halves: the bare statement fails with "Must be superuser to create this extension", the guarded one emits a notice and continues. Creating pg_stat_statements without the preload is also safe -- its install script only defines functions and a view, and the library loads lazily -- so the extension can be created anywhere and merely errors on read until postgres restarts with it preloaded. Verified that too: 018 applied cleanly against a postgres with an empty shared_preload_libraries, the view reported "must be loaded via shared_preload_libraries", and the API served normally throughout. The cost is roughly 1 MB of shared memory and about 1% CPU. track_planning stays off, since that is the expensive one. Sizing does not follow it into this file. PostgreSQL runs on stock defaults here -- 128 MB of shared_buffers regardless of the machine underneath -- and the right values differ per host, so they belong to the deployment that knows the host. What the deployment lacked is a way to say so: Compose replaces `command:` rather than merging it, so an override file has to restate every flag and silently drops whatever the list grows next. SHELLHUB_POSTGRES_EXTRA_ARGS is that seam, appended after the flags the product itself requires. postgres takes the final occurrence of a setting, so a deployment can raise shared_buffers or outrank io_method without naming the flags it does not care about, and adding a flag here later does not break an existing override. Empty by default, so nothing changes out of the box: .env documents it with an example, and .env.override -- which bin/docker-compose already loads last -- is where a host sets it. Docker-level limits deliberately stay out of it: shm_size and mem_limit merge from an override file the ordinary way, and only `command:` has the wholesale-replacement problem. Refs: shellhub-io/team#198 Refs: shellhub-io/team#200
otavio
force-pushed
the
perf/enable-pg-stat-statements
branch
from
August 13, 2026 11:30
c9ce2ba to
381eb43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Table-level counters can say which table is being scanned but never which query is scanning
it. That is why the sequential-scan investigation (shellhub-io/team#200) could report 1.37 billion
sequential scans against a two-row
namespacestable without being able to name a singlestatement responsible. It lists enabling
pg_stat_statementsas its own first action.This enables it, plus two extensions that close the neighbouring measurement gaps:
pg_stat_statementscallsandtotal_exec_timepgstattuplepg_buffercacheshared_buffersAll three already ship in
postgres:18.0, so there is no image build here — a preload flag and aCREATE EXTENSION.The two safety properties, both verified
1. A diagnostic must never be able to fail a boot. Migrations run inline in
Server.Setupbefore the listener binds and an error there is
log.Fatal, and none of these three is atrusted extension — so an unguarded
CREATE EXTENSIONwould turn a database role withoutsuperuser into a server that cannot start. Each statement is therefore wrapped in a
DOblockthat can only warn. Both halves tested against a non-superuser role:
The trade-off is that a swallowed failure is not retried, since bun marks a migration applied
before running it. That is deliberate for diagnostics, and
IF NOT EXISTSkeeps the manual fixidempotent.
2. The migration does not depend on the compose flag.
CREATE EXTENSION pg_stat_statementssucceeds withoutshared_preload_libraries— its install script only definesfunctions and a view, and the library loads lazily. So the migration is safe on a deployment whose
postgres was started without the flag; only reading the view fails there. Verified by applying
018against a postgres with an emptyshared_preload_libraries:…while migrations completed successfully,
pgstattupleworked normally, and the API kept serving.Nothing in ShellHub queries the view, so this degraded state has no application impact.
Configuration
Restart-only rather than reload, which an upgrade's stack recreation already provides.
compute_query_idis already at itsautodefault and activates itself once the library loads,so there is no second flag.
No other GUCs, deliberately:
pg_stat_statements.max(5000) is far above ShellHub'sparameterised statement count,
track = topavoids nested-statement cost we have no question for,and
track_planningstays off because it is the expensive one — a contended spinlock perplan. Cost is roughly 1 MB of shared memory and ~1% CPU.
docker-compose.postgres.test.ymlis left alone: it is a standalone replacement overlay used onlyby the integration-test harness, which gains nothing from statement collection.
Sizing belongs to the deployment (was #6892)
PostgreSQL otherwise runs on stock defaults everywhere — 128 MB of
shared_buffersregardless ofthe machine underneath, which on the largest managed instance is an 84.76% buffer cache hit ratio
against a 5.9 GB database on a 15.6 GB host (shellhub-io/team#198). The right values differ per
host, so they belong to whatever knows the host, not to this file.
What the deployment lacked was a way to say so. Compose replaces
command:rather than mergingit, so an override file has to restate every flag and silently drops whatever the list grows next.
SHELLHUB_POSTGRES_EXTRA_ARGSis that seam: appended after the flags the product itself requires,empty by default, documented in
.envand set from.env.overrideor the managed deploy.Appending is enough because postgres takes the final occurrence of a setting — a deployment can
raise
shared_buffersor even outrankio_methodwithout naming the flags it does not care about,and adding a flag here later cannot break an existing override. Verified on
postgres:18.0:Docker-level limits deliberately stay out of it:
shm_sizeandmem_limitmerge from an overridefile the ordinary way (confirmed:
64m→1g), and onlycommand:has the replacement problem.This carries no behaviour of its own. With the variable empty, the rendered argv is exactly
master's plus the preload flag — no warning on stderr, no stray empty argument.It already earned its keep
Five authenticated
GET /api/devicesrequests against the dev stack, with counters reset first:SELECT … FROM namespaces …SELECT … FROM memberships …SELECT … FROM users …SELECT device.* …(the device list)SELECT count(*) FROM devices …(its pagination count)Five statements of identity and tenancy resolution per request, against two that do the work.
The namespace is fetched twice per request — once by
GetUserRolein the authenticator, once byListDevicespurely to read the device-limit fields — and each fetch drags inRelation("Memberships.User"), whichListDevicesnever reads. That is the mechanism behind#200's 1.37 B
namespacesand ~472 Musers/membershipsscans, now confirmed from both ends.Details posted on the issue; the fix is a separate change.
Testing
pgstore suite green against a schema built from001through
018.018applied and reverted against a stockpostgres:18.0: all three extensions created,pg_stat_statementsreadable with the preload, all three dropped cleanly on the way down.docker compose configrendered with the variable empty, populated, and carrying a quoted valuecontaining a space (
shared_preload_libraries='pg_stat_statements, pg_prewarm'survives as oneargument).
preload), plus the non-superuser privilege path shown above.
Refs shellhub-io/team#198, shellhub-io/team#200.