Skip to content

perf(server): stop re-resolving the namespace on every authenticated request - #6890

Merged
otavio merged 1 commit into
masterfrom
perf/identity-resolution-statements
Aug 17, 2026
Merged

perf(server): stop re-resolving the namespace on every authenticated request#6890
otavio merged 1 commit into
masterfrom
perf/identity-resolution-statements

Conversation

@otavio

@otavio otavio commented Aug 12, 2026

Copy link
Copy Markdown
Member

What

Cuts identity resolution on an authenticated device-list request from five statements to
three, removes five namespaces scans per /stats call, and puts the systems singleton
behind the cache.

This branch applies directly on top of master — it is no longer stacked on #6888. It is a
single commit; the clock-pinning store tests are part of it, since they assert this change's
own switch from time.Now() to clock.Now().

Why

Part of shellhub-io/team#200. On the 58,670-device production instance the review measured,
a 2-row namespaces table took 1,371,078,783 sequential scans in 66 days, with
users and memberships at ~472 million each. The plans are correct — PostgreSQL should
seq-scan a 2-row table — so the cost is call volume, not query shape, and no index would
help.

Traced to source, one authenticated GET /api/devices spends five statements establishing
who is calling against two that answer the request:

  1. Authenticator.ResolveGetUserRoleNamespaceResolve, which carries
    Relation("Memberships.User") — bun issues the has-many as a second statement.
  2. GetUserAdminUserResolve, a third, on users.
  3. ListDevices then resolves the same namespace again, memberships included, purely to
    read two integers off it (HasMaxDevices / HasMaxDevicesReached).

That mechanism is why users and memberships sit at near-identical counts, and why
namespaces runs ahead of both — it is fetched twice per request.

Changes

  • NamespaceGetDeviceLimit: reads the two columns the device-limit check needs. Kept
    additive on purpose — NamespaceResolve keeps its relation, so the ~40 sites in the
    cloud repo that read memberships off a resolved namespace are untouched. Dropping the
    relation from NamespaceResolve itself would have broken cloud at runtime, not at compile
    time.
  • models.NamespaceDeviceLimit: the ceiling rule moves here;
    Namespace.HasMaxDevices/HasMaxDevicesReached delegate, so it stays defined once.
  • authctx: the authenticator forwards the limit it already read, paired with the tenant
    it belongs to so a request targeting another namespace cannot be answered from it. The
    guard lives inside the accessor rather than at the call site, so a future caller cannot
    skip it. Falls back to the store when the request carries no limit — API keys, internal
    callers, the admin surface.
  • GetUserRole deleted: it had become a one-line wrapper over ResolveNamespaceRole
    with a single caller.
  • GetStats: its five counts bounded the namespace with
    namespace_id = (SELECT id FROM namespaces WHERE id = ?), one namespaces scan per stat,
    where every other scoped query in the store uses the shared scope option. They now do too.
  • systems cache: one row, 10,401,522 sequential scans and zero index scans in
    the same window. The key gets a single definition in pkg/cache — it is spelled in three
    packages across two repositories, and a second literal would silently stop invalidating. It
    was already spelled twice, with nothing populating it.

Reviewer notes

Re-fetching per request stays deliberate. authn.go re-reads role and admin on every
request because shellhub-io/team#193 found a demoted admin keeping API access for up to 72
hours. This PR does not cache them; it removes the duplicate resolve in the handler. Any
future identity cache needs explicit invalidation on membership/role/user writes.

One error code changes. scope.NewBounded rejects only empty tenant IDs, so a bounded
scope can carry a malformed UUID. It used to reach PostgreSQL as SQLSTATE 22P02; the shared
scope option's guard now returns ErrNoDocuments, matching NamespaceResolve and the rest of
the store.

pinClock now lives in storetest/helpers.go. #6888 introduced the helper next to the
session-retention tests it was written for. This PR needs it and no longer sits on that
branch, so the helper moves to the shared helpers file. If #6888 rebases onto a master that
carries this PR, it must drop its own copy.

The systems cache does not help Enterprise or Cloud on its own — they serve /info
from their own service. shellhub-io/cloud#2491 covers that side, and must land
with this PR anyway: NamespaceGetDeviceLimit on the store interface requires cloud's
generated mock to be regenerated, or cloud's test build fails in eight packages.

Not in this PR, and deliberately so: caching identity resolution (needs its own design
against the #193 constraint), the unattributed sessions scans (blocked on
pg_stat_statements in #6886), the devices online count (moved to
shellhub-io/team#209), and the enterprise license evaluator's unbounded GetStats — split
out to shellhub-io/team#212 after it turned out to account for ~43% of the devices
sequential scans on that host.

Testing

server: full suite and golangci-lint clean; root pkg/models and pkg/cache clean.

Worth probing:

  • TestGetStats already covers cross-tenant isolation on the rewritten counts, including the
    active-session count that reaches the namespace through devices — that one needed a table
    alias, and a wrong alias would silently widen the bound rather than error.
  • TestSystemGet covers the subtle path: Cache.Get reports no error on a miss and
    leaves the destination zeroed, so the Setup && Authentication.Local != nil guard is the
    only thing standing between a miss and a zero-value response.
  • TestGetStatsOnlineBoundary fails if buildOnlineDevicesQuery goes back to time.Now();
    verified by reverting it.

@otavio
otavio requested review from a team as code owners August 12, 2026 16:12
@otavio
otavio force-pushed the perf/session-retention-policy branch from 7b8205d to 6e33420 Compare August 12, 2026 16:33
@otavio
otavio force-pushed the perf/identity-resolution-statements branch from a46e644 to ac00c50 Compare August 12, 2026 16:33
@otavio
otavio force-pushed the perf/session-retention-policy branch from 6e33420 to 1848aeb Compare August 12, 2026 21:46
@otavio
otavio force-pushed the perf/identity-resolution-statements branch from ac00c50 to a5652ac Compare August 12, 2026 21:46
@otavio
otavio force-pushed the perf/session-retention-policy branch from 1848aeb to 3df2e69 Compare August 13, 2026 12:34
@otavio
otavio force-pushed the perf/identity-resolution-statements branch from a5652ac to f1212e4 Compare August 13, 2026 12:34
@otavio
otavio force-pushed the perf/session-retention-policy branch from 3df2e69 to 7f5a963 Compare August 13, 2026 13:28
@otavio
otavio force-pushed the perf/identity-resolution-statements branch from f1212e4 to 9f27458 Compare August 13, 2026 13:28
@otavio
otavio force-pushed the perf/session-retention-policy branch from 7f5a963 to a0656a4 Compare August 17, 2026 17:10
@otavio
otavio force-pushed the perf/identity-resolution-statements branch 2 times, most recently from 96e8dcc to 75b6c77 Compare August 17, 2026 18:35
@otavio
otavio changed the base branch from perf/session-retention-policy to master August 17, 2026 18:35
…request

Establishing who is calling costs five statements against the two that answer a device-list
request. The authenticator resolves the namespace to read the caller's role, and
NamespaceResolve carries Relation("Memberships.User"), which bun issues as a second query;
GetUserAdmin adds a third on users; then ListDevices resolves the same namespace all over
again, memberships included, purely to read two integers off it.

On a 58,670-device production deployment that shows up as 1,371,078,783 sequential scans on
a 2-row namespaces table in 66 days, with users and memberships at ~472 million each. The
plans are correct — Postgres should seq-scan a 2-row table — so the cost is the call volume,
not the query.

NamespaceGetDeviceLimit reads the two columns the device-limit check actually needs. It is
additive: NamespaceResolve keeps its relation, so the ~40 sites in the cloud repo that read
memberships off a resolved namespace are untouched. The rule itself moves to
NamespaceDeviceLimit, and Namespace.HasMaxDevices/HasMaxDevicesReached delegate to it so it
stays defined once.

The authenticator hands the limit it read to the rest of the request through authctx, paired
with the tenant it belongs to so a request targeting another namespace cannot be answered
from it, and ListDevices uses it instead of reading the store again. It falls back to the
store when the request carries no limit — API keys, internal callers, the admin surface. The
limit is a snapshot, which is what the handler it replaces already worked from: that one
re-read the namespace within the same request.

Re-fetching per request stays deliberate: role and admin are dynamic, and a cached role is
what let a demoted admin keep access for up to 72 hours. GetUserRole is gone, its one
remaining caller now going straight to ResolveNamespaceRole.

GetStats bounded its five counts with namespace_id = (SELECT id FROM namespaces WHERE id =
?), one namespaces scan per stat, where every other scoped query in the store uses the
shared scope option. They now do too, with the active-session count naming the devices table
it reaches the namespace through. One error code changes with it: a bounded scope carrying a
malformed tenant ID now returns ErrNoDocuments from the shared guard, where the raw subquery
sent it to Postgres and got SQLSTATE 22P02 back. That matches NamespaceResolve and the rest
of the store.

Per authenticated device-list request this takes identity resolution from five statements to
three, and /stats loses five namespaces scans per call.

The systems singleton — one row, 10,401,522 sequential scans and zero index scans in the
same window — is cached behind the UI-polled info endpoint. Nothing is cached before setup
completes, so the false-to-true transition the UI is polling for cannot be served stale by
the admin CLI, which has no cache to invalidate; afterwards the only mutable field this
endpoint exposes is the local-authentication flag, whose writers evict the key.

That key gets a single definition in pkg/cache. It is read here, read again by the cloud
service that serves the same endpoint, and deleted by the cloud store on an authentication
change — three packages across two repositories, where a second literal spelling of it would
silently stop invalidating. It was already spelled twice, and nothing populated it.

buildOnlineDevicesQuery moved in this change and took a time.Now() with it, now clock.Now()
so the two-minute window is pinnable in tests. Two store tests pin it -- one device seen a
minute ago, one seen three minutes ago, then the clock moved past the window to take the first
offline -- and they fail if the query goes back to time.Now(). The existing stats tests create
devices at the current time, so they pass either way and assert nothing about which clock is
read. WithDeviceLastSeen comes along because DeviceCreate writes last_seen straight from the
model and the suite had no way to set it, and pinClock joins it in helpers.go because pinning
the clock is no longer specific to one test file.

Issue: shellhub-io/team#200
@otavio
otavio force-pushed the perf/identity-resolution-statements branch 2 times, most recently from 75b6c77 to 01a2e56 Compare August 17, 2026 18:55
@otavio
otavio merged commit 9183608 into master Aug 17, 2026
38 of 39 checks passed
@otavio
otavio deleted the perf/identity-resolution-statements branch August 17, 2026 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant