fix(blockchain-api): stop tracing middleware, it exhausts the org Sentry quota - #1229
Merged
Conversation
…try quota The edge runtime's Sentry config traced every request at 1.0, and the only thing it traces in this service is middleware. That produced 155,763 bare `middleware GET` / `middleware POST` transactions in the 2026-07-12 billing period, each duplicating timing the route transaction for the same request already records. That is 40% of this service's 393,645 transactions for the period, and this service is 79% of the org's total. The org's 500k transaction quota was exhausted on 2026-07-21, nine days into a thirty-day period, with on-demand spend at zero. Sentry's quota is org-wide, so past that point performance data is rate-limited for every project in the org, not just this one. Dropping middleware tracing alone takes this service to ~238k for a period and the org to ~344k of 500k, back under quota with headroom. Tracing is the only thing affected. Errors thrown in middleware still report and are still unsampled; the org's error quota runs at about 3.5%. Worth a follow-up: sentry.server.config.ts also traces at 1.0, and sentry.base.config.ts exports a commonSentryOptions with tracesSampleRate 1 that neither the server nor edge config actually spreads.
Contributor
|
Seems fine to me |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
sentry.edge.config.tssetstracesSampleRate: 1. In this service the edge runtime only traces middleware, so every single request produces a baremiddleware GETormiddleware POSTtransaction on top of the route transaction that request already emits.Volume for the 2026-07-12 → 08-11 billing period:
This service is 79% of the whole org's transaction volume. The org's 500k transaction quota was exhausted on 2026-07-21 — nine days into a thirty-day period — with on-demand spend at $0.
Sentry's quota is org-wide, not per-project. Past that point Sentry rate-limits performance data for every project in the org. Concretely, a newly instrumented service elsewhere in the org is currently getting 100% of its spans and transactions dropped:
The change
tracesSampleRate: 1→0in the edge config only.Effect: this service drops to ~238k transactions per period, the org to ~344k of 500k. Back under quota with headroom.
Tracing is the only thing affected. Errors thrown in middleware still report, unsampled. Sampling and error capture are independent in the SDK.
What I'm assuming
That nobody relies on
middleware GET/middleware POSTtransactions as a signal. They carry only middleware execution timing, and the route transaction for the same request already records the request. If someone is actively using them, say so and I'll switch this to atracesSamplerthat keeps a small sample instead of zero.Deliberately not in this PR
sentry.server.config.tsalso runstracesSampleRate: 1. Lowering it (say to0.25, env-overridable) would take this service to ~60k per period instead of ~238k. I left it alone because the edge change is sufficient to get back under quota, and dialling route-level tracing down is a judgement call for whoever owns this service.Also noticed while in here:
sentry.base.config.tsexportscommonSentryOptionswithtracesSampleRate: 1, but neither the server nor edge config spreads it — it looks like dead config.Verification
The numbers above come from Sentry's
stats_v2API for orghelium-su, queried per-project. Worth knowing if you go checking:groupBy=projectsilently returns only projects you're a member of, which under-reports by ~4x. Per-project queries with an explicitproject=<id>filter give the real attribution.