Skip to content

fix(blockchain-api): stop tracing middleware, it exhausts the org Sentry quota - #1229

Merged
bryzettler merged 1 commit into
developfrom
sentry-trace-volume
Jul 27, 2026
Merged

fix(blockchain-api): stop tracing middleware, it exhausts the org Sentry quota#1229
bryzettler merged 1 commit into
developfrom
sentry-trace-volume

Conversation

@allenan

@allenan allenan commented Jul 24, 2026

Copy link
Copy Markdown
Member

Needs an engineer from this repo to review before merge. I found this while investigating why Sentry was dropping all performance data org-wide, and I don't own this service. Please sanity-check the assumption in "What I'm assuming" below.

The problem

sentry.edge.config.ts sets tracesSampleRate: 1. In this service the edge runtime only traces middleware, so every single request produces a bare middleware GET or middleware POST transaction on top of the route transaction that request already emits.

Volume for the 2026-07-12 → 08-11 billing period:

81,753  middleware POST
74,010  middleware GET
───────
155,763  = 40% of this service's 393,645 transactions

This service is 79% of the whole org's transaction volume. The org's 500k transaction quota was exhausted on 2026-07-21 — nine days into a thirty-day period — with on-demand spend at $0.

Sentry's quota is org-wide, not per-project. Past that point Sentry rate-limits performance data for every project in the org. Concretely, a newly instrumented service elsewhere in the org is currently getting 100% of its spans and transactions dropped:

24  rate_limited  span
 6  rate_limited  transaction
 0  accepted

The change

tracesSampleRate: 10 in the edge config only.

Effect: this service drops to ~238k transactions per period, the org to ~344k of 500k. Back under quota with headroom.

Tracing is the only thing affected. Errors thrown in middleware still report, unsampled. Sampling and error capture are independent in the SDK.

What I'm assuming

That nobody relies on middleware GET / middleware POST transactions as a signal. They carry only middleware execution timing, and the route transaction for the same request already records the request. If someone is actively using them, say so and I'll switch this to a tracesSampler that keeps a small sample instead of zero.

Deliberately not in this PR

sentry.server.config.ts also runs tracesSampleRate: 1. Lowering it (say to 0.25, env-overridable) would take this service to ~60k per period instead of ~238k. I left it alone because the edge change is sufficient to get back under quota, and dialling route-level tracing down is a judgement call for whoever owns this service.

Also noticed while in here: sentry.base.config.ts exports commonSentryOptions with tracesSampleRate: 1, but neither the server nor edge config spreads it — it looks like dead config.

Verification

The numbers above come from Sentry's stats_v2 API for org helium-su, queried per-project. Worth knowing if you go checking: groupBy=project silently returns only projects you're a member of, which under-reports by ~4x. Per-project queries with an explicit project=<id> filter give the real attribution.

…try quota

The edge runtime's Sentry config traced every request at 1.0, and the only
thing it traces in this service is middleware. That produced 155,763 bare
`middleware GET` / `middleware POST` transactions in the 2026-07-12 billing
period, each duplicating timing the route transaction for the same request
already records.

That is 40% of this service's 393,645 transactions for the period, and this
service is 79% of the org's total. The org's 500k transaction quota was
exhausted on 2026-07-21, nine days into a thirty-day period, with on-demand
spend at zero. Sentry's quota is org-wide, so past that point performance data
is rate-limited for every project in the org, not just this one.

Dropping middleware tracing alone takes this service to ~238k for a period and
the org to ~344k of 500k, back under quota with headroom.

Tracing is the only thing affected. Errors thrown in middleware still report
and are still unsampled; the org's error quota runs at about 3.5%.

Worth a follow-up: sentry.server.config.ts also traces at 1.0, and
sentry.base.config.ts exports a commonSentryOptions with tracesSampleRate 1
that neither the server nor edge config actually spreads.
@bryzettler

Copy link
Copy Markdown
Contributor

Seems fine to me

@bryzettler
bryzettler merged commit f64a0ea into develop Jul 27, 2026
110 of 111 checks passed
@bryzettler
bryzettler deleted the sentry-trace-volume branch July 27, 2026 14:32
@github-actions github-actions Bot mentioned this pull request Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants