perf(server): stop re-resolving the namespace on every authenticated request - #6890
Merged
Conversation
otavio
force-pushed
the
perf/session-retention-policy
branch
from
August 12, 2026 16:33
7b8205d to
6e33420
Compare
otavio
force-pushed
the
perf/identity-resolution-statements
branch
from
August 12, 2026 16:33
a46e644 to
ac00c50
Compare
otavio
force-pushed
the
perf/session-retention-policy
branch
from
August 12, 2026 21:46
6e33420 to
1848aeb
Compare
otavio
force-pushed
the
perf/identity-resolution-statements
branch
from
August 12, 2026 21:46
ac00c50 to
a5652ac
Compare
otavio
force-pushed
the
perf/session-retention-policy
branch
from
August 13, 2026 12:34
1848aeb to
3df2e69
Compare
otavio
force-pushed
the
perf/identity-resolution-statements
branch
from
August 13, 2026 12:34
a5652ac to
f1212e4
Compare
otavio
force-pushed
the
perf/session-retention-policy
branch
from
August 13, 2026 13:28
3df2e69 to
7f5a963
Compare
otavio
force-pushed
the
perf/identity-resolution-statements
branch
from
August 13, 2026 13:28
f1212e4 to
9f27458
Compare
otavio
force-pushed
the
perf/session-retention-policy
branch
from
August 17, 2026 17:10
7f5a963 to
a0656a4
Compare
otavio
force-pushed
the
perf/identity-resolution-statements
branch
2 times, most recently
from
August 17, 2026 18:35
96e8dcc to
75b6c77
Compare
…request
Establishing who is calling costs five statements against the two that answer a device-list
request. The authenticator resolves the namespace to read the caller's role, and
NamespaceResolve carries Relation("Memberships.User"), which bun issues as a second query;
GetUserAdmin adds a third on users; then ListDevices resolves the same namespace all over
again, memberships included, purely to read two integers off it.
On a 58,670-device production deployment that shows up as 1,371,078,783 sequential scans on
a 2-row namespaces table in 66 days, with users and memberships at ~472 million each. The
plans are correct — Postgres should seq-scan a 2-row table — so the cost is the call volume,
not the query.
NamespaceGetDeviceLimit reads the two columns the device-limit check actually needs. It is
additive: NamespaceResolve keeps its relation, so the ~40 sites in the cloud repo that read
memberships off a resolved namespace are untouched. The rule itself moves to
NamespaceDeviceLimit, and Namespace.HasMaxDevices/HasMaxDevicesReached delegate to it so it
stays defined once.
The authenticator hands the limit it read to the rest of the request through authctx, paired
with the tenant it belongs to so a request targeting another namespace cannot be answered
from it, and ListDevices uses it instead of reading the store again. It falls back to the
store when the request carries no limit — API keys, internal callers, the admin surface. The
limit is a snapshot, which is what the handler it replaces already worked from: that one
re-read the namespace within the same request.
Re-fetching per request stays deliberate: role and admin are dynamic, and a cached role is
what let a demoted admin keep access for up to 72 hours. GetUserRole is gone, its one
remaining caller now going straight to ResolveNamespaceRole.
GetStats bounded its five counts with namespace_id = (SELECT id FROM namespaces WHERE id =
?), one namespaces scan per stat, where every other scoped query in the store uses the
shared scope option. They now do too, with the active-session count naming the devices table
it reaches the namespace through. One error code changes with it: a bounded scope carrying a
malformed tenant ID now returns ErrNoDocuments from the shared guard, where the raw subquery
sent it to Postgres and got SQLSTATE 22P02 back. That matches NamespaceResolve and the rest
of the store.
Per authenticated device-list request this takes identity resolution from five statements to
three, and /stats loses five namespaces scans per call.
The systems singleton — one row, 10,401,522 sequential scans and zero index scans in the
same window — is cached behind the UI-polled info endpoint. Nothing is cached before setup
completes, so the false-to-true transition the UI is polling for cannot be served stale by
the admin CLI, which has no cache to invalidate; afterwards the only mutable field this
endpoint exposes is the local-authentication flag, whose writers evict the key.
That key gets a single definition in pkg/cache. It is read here, read again by the cloud
service that serves the same endpoint, and deleted by the cloud store on an authentication
change — three packages across two repositories, where a second literal spelling of it would
silently stop invalidating. It was already spelled twice, and nothing populated it.
buildOnlineDevicesQuery moved in this change and took a time.Now() with it, now clock.Now()
so the two-minute window is pinnable in tests. Two store tests pin it -- one device seen a
minute ago, one seen three minutes ago, then the clock moved past the window to take the first
offline -- and they fail if the query goes back to time.Now(). The existing stats tests create
devices at the current time, so they pass either way and assert nothing about which clock is
read. WithDeviceLastSeen comes along because DeviceCreate writes last_seen straight from the
model and the suite had no way to set it, and pinClock joins it in helpers.go because pinning
the clock is no longer specific to one test file.
Issue: shellhub-io/team#200
otavio
force-pushed
the
perf/identity-resolution-statements
branch
2 times, most recently
from
August 17, 2026 18:55
75b6c77 to
01a2e56
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Cuts identity resolution on an authenticated device-list request from five statements to
three, removes five
namespacesscans per/statscall, and puts thesystemssingletonbehind the cache.
This branch applies directly on top of
master— it is no longer stacked on #6888. It is asingle commit; the clock-pinning store tests are part of it, since they assert this change's
own switch from
time.Now()toclock.Now().Why
Part of shellhub-io/team#200. On the 58,670-device production instance the review measured,
a 2-row
namespacestable took 1,371,078,783 sequential scans in 66 days, withusersandmembershipsat ~472 million each. The plans are correct — PostgreSQL shouldseq-scan a 2-row table — so the cost is call volume, not query shape, and no index would
help.
Traced to source, one authenticated
GET /api/devicesspends five statements establishingwho is calling against two that answer the request:
Authenticator.Resolve→GetUserRole→NamespaceResolve, which carriesRelation("Memberships.User")— bun issues the has-many as a second statement.GetUserAdmin→UserResolve, a third, onusers.ListDevicesthen resolves the same namespace again, memberships included, purely toread two integers off it (
HasMaxDevices/HasMaxDevicesReached).That mechanism is why
usersandmembershipssit at near-identical counts, and whynamespacesruns ahead of both — it is fetched twice per request.Changes
NamespaceGetDeviceLimit: reads the two columns the device-limit check needs. Keptadditive on purpose —
NamespaceResolvekeeps its relation, so the ~40 sites in thecloudrepo that read memberships off a resolved namespace are untouched. Dropping therelation from
NamespaceResolveitself would have broken cloud at runtime, not at compiletime.
models.NamespaceDeviceLimit: the ceiling rule moves here;Namespace.HasMaxDevices/HasMaxDevicesReacheddelegate, so it stays defined once.authctx: the authenticator forwards the limit it already read, paired with the tenantit belongs to so a request targeting another namespace cannot be answered from it. The
guard lives inside the accessor rather than at the call site, so a future caller cannot
skip it. Falls back to the store when the request carries no limit — API keys, internal
callers, the admin surface.
GetUserRoledeleted: it had become a one-line wrapper overResolveNamespaceRolewith a single caller.
GetStats: its five counts bounded the namespace withnamespace_id = (SELECT id FROM namespaces WHERE id = ?), onenamespacesscan per stat,where every other scoped query in the store uses the shared scope option. They now do too.
systemscache: one row, 10,401,522 sequential scans and zero index scans inthe same window. The key gets a single definition in
pkg/cache— it is spelled in threepackages across two repositories, and a second literal would silently stop invalidating. It
was already spelled twice, with nothing populating it.
Reviewer notes
Re-fetching per request stays deliberate.
authn.gore-reads role and admin on everyrequest because shellhub-io/team#193 found a demoted admin keeping API access for up to 72
hours. This PR does not cache them; it removes the duplicate resolve in the handler. Any
future identity cache needs explicit invalidation on membership/role/user writes.
One error code changes.
scope.NewBoundedrejects only empty tenant IDs, so a boundedscope can carry a malformed UUID. It used to reach PostgreSQL as SQLSTATE 22P02; the shared
scope option's guard now returns
ErrNoDocuments, matchingNamespaceResolveand the rest ofthe store.
pinClocknow lives instoretest/helpers.go. #6888 introduced the helper next to thesession-retention tests it was written for. This PR needs it and no longer sits on that
branch, so the helper moves to the shared helpers file. If #6888 rebases onto a
masterthatcarries this PR, it must drop its own copy.
The
systemscache does not help Enterprise or Cloud on its own — they serve/infofrom their own service. shellhub-io/cloud#2491 covers that side, and must land
with this PR anyway:
NamespaceGetDeviceLimiton the store interface requires cloud'sgenerated mock to be regenerated, or cloud's test build fails in eight packages.
Not in this PR, and deliberately so: caching identity resolution (needs its own design
against the #193 constraint), the unattributed
sessionsscans (blocked onpg_stat_statementsin #6886), thedevicesonline count (moved toshellhub-io/team#209), and the enterprise license evaluator's unbounded
GetStats— splitout to shellhub-io/team#212 after it turned out to account for ~43% of the
devicessequential scans on that host.
Testing
server: full suite andgolangci-lintclean; rootpkg/modelsandpkg/cacheclean.Worth probing:
TestGetStatsalready covers cross-tenant isolation on the rewritten counts, including theactive-session count that reaches the namespace through
devices— that one needed a tablealias, and a wrong alias would silently widen the bound rather than error.
TestSystemGetcovers the subtle path:Cache.Getreports no error on a miss andleaves the destination zeroed, so the
Setup && Authentication.Local != nilguard is theonly thing standing between a miss and a zero-value response.
TestGetStatsOnlineBoundaryfails ifbuildOnlineDevicesQuerygoes back totime.Now();verified by reverting it.