Skip to content

fix(server): bound OTLP trace export backlog - #40710

Closed
Cujuju wants to merge 2 commits into
pingdotgg:mainfrom
Cujuju:t3/fix-serve-memory-cpu-saturation
Closed

Cujuju wants to merge 2 commits into
pingdotgg:mainfrom
Cujuju:t3/fix-serve-memory-cpu-saturation

Conversation

@Cujuju

@Cujuju Cujuju commented Nov 22, 2026

Copy link
Copy Markdown

A stalled OTLP collector could make t3 serve accumulate overlapping trace exports. The Effect exporter launched another request whenever a batch filled, even while earlier requests were still waiting. The encoded bodies and HTTP client copies stayed live, and serializing each batch added CPU load during the outage.

The server now allows one trace export in flight, gives each flush a 10-second deadline, and keeps up to 1,000 pending spans. Overflow is counted in a warning. The local NDJSON sink still receives the spans.

Evidence

On Windows, I ran the affected nightly source with an isolated SQLite backup and sent the same 6,000 read requests to each version while a loopback collector withheld its response. All 6,000 completed in both runs.

Measurement Stock nightly With the fix
Peak concurrent OTLP requests 273 1
Peak outstanding OTLP request bodies 117.5 MiB 0.72 MiB
Peak process RSS 1,205 MiB 866 MiB
Peak Node ArrayBuffer allocation 368 MiB 54 MiB
API latency p95 150 ms 64 ms

CPU profiles showed OTLP encoding and Undici buffer copies during the backlog. The live service logged OTLP trace connection timeouts. The isolated runs establish the exporter failure mode. The cause of the remaining gap to the historical multi-gigabyte JS heap is unverified.

The 11.45 GiB SQLite backup held 4.22 GiB of free pages, and turn-item projections occupied 3.86 GiB. Offline vacuum reduced a copy to 6.97 GiB, passed SQLite's quick check, and preserved the counted projection and event rows. T3 already has offline compaction and event pruning.

Two regression cases fail with the stock exporter and pass with this change. The tests cover congestion, timeout and recovery, and shutdown flush. vp run check:changed passed.

gpt-6.2-sol via Codex.

Cujuju and others added 2 commits November 14, 2026 19:29
Adds a fork section, marked with `<!-- fork:start -->` / `<!-- fork:end
-->`, at the top of `CONTRIBUTING.md`. Upstream's guide below it is
unchanged (diff: 14 insertions, 0 deletions), which keeps upstream syncs
conflict-free here.

The section says:
- this is a fork of pingdotgg/t3code, used to develop the CLIProxyAPI
usage-hub integration and send it upstream (e.g. pingdotgg#27901)
- changes that aren't fork-specific belong upstream
- PRs here target `main`, and the maintainer picks the merge method
- upstream syncs must be merged with a merge commit, never squashed or
rebased

The description comes from the README and the fork-only branches
(`fix/hub-routes-clients`, `feat/hub-account-names`,
`fix/limit-recovery-on-usage`). The fork's `main` itself has no
fork-only commits: it is 3593 commits behind upstream `main`.

Verified: docs-only. I didn't run `vp run check:changed` (no install in
this clone).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Closing: Cujuju already has 3 or more open PRs here, which is the limit for contributors who are not vouched.

Once one of them is merged or closed, open this again. See the open PR limit.

@github-actions github-actions Bot closed this Nov 22, 2026
@github-actions github-actions Bot added vouch:unvouched Author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:M 30-99 changed lines (additions + deletions). vouch:unvouched Author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant