Add separate rate limit for graph launches (default 20/s) - #22
Merged
Conversation
Graph launches previously bypassed the callback rate limiter entirely, so a high graph-replay rate could flood the cupti_events ringbuf: each graph launch fans out to many kernel activity records, and graphs dominate GPU event volume on graph-heavy workloads (e.g. TRT-LLM decode). Give graph launches their own token bucket (default 20/sec, PARCAGPU_GRAPH_RATE_LIMIT) instead of the bypass. This preserves the original intent -- eager bursts can't starve graph sampling, since the budgets are now separate -- while bounding graph bursts. A rate-limited graph launch is never inserted into the graph correlation map, so its kernel activities are filtered before they reach the ringbuf, throttling the dominant traffic at the source. Apply both configured rates (eager and graph) from process-global atomics that each thread copies into its thread_local bucket on first use. Previously init_debug called setRate on the thread_local bucket directly, which only affected the single thread that ran init_debug -- so PARCAGPU_RATE_LIMIT (and the new graph knob) silently had no effect on other launching threads.
brancz
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Graph launches previously bypassed the callback rate limiter entirely. On graph-heavy workloads (e.g. TRT-LLM decode) a high graph-replay rate floods the
cupti_eventsringbuf: each graph launch fans out to many kernel activity records, and graphs dominate GPU event volume. Observed ~330 graph launches/s aggregate (peaks 737/s) on a 4×B200 node, with ringbuf drops tracking the bursts.Give graph launches their own thread-local token bucket (default 20/s,
PARCAGPU_GRAPH_RATE_LIMIT) instead of the bypass. A rate-limited graph launch is never inserted into the graph correlation map, so its kernel activities are filtered before reaching the ringbuf — throttling the dominant traffic at the source. Separate budgets preserve the original intent: eager bursts can't starve graph sampling.Also fixes rate-limit application across threads: env-configured rates now live in process-global atomics that each thread copies into its
thread_localbucket on first use. Previouslyinit_debugcalledsetRateon the thread_local bucket directly, soPARCAGPU_RATE_LIMIT(and the new graph knob) only affected the single thread that raninit_debug.