CUDA Graph + kernel-only activity: GPU memory growth and invalid timestamps after ~25M records (v0.1.2)

[123liuziming]

CUDA Graph + kernel-only activity: GPU memory grows ~1 GiB, then kernel timestamps become invalid (v0.1.2)

We reproduced invalid CUPTI kernel timestamps with the unmodified upstream v0.1.2 CUDA 13 amd64 release asset (libparcagpucupti.so.13-amd64, SHA256 cc516a1903c6249242869ce42669618594e2ad74bd7b94a4bcd5d3bb966f0185). This may cause GPU profiling to silently lose valid kernel durations under sustained CUDA Graph workloads. I have not tested current main or other CUPTI versions.

Environment and workload

• NVIDIA A10, driver 580.126.09; CUDA 13.0, CUPTI 13.0.85.
• vLLM 0.24.0 precompiled, Qwen2.5-Coder-1.5B-Instruct, TP=1, CUDA Graph enabled, --max-model-len 2048 --gpu-memory-utilization 0.55 --no-enable-prefix-caching.
• 32 concurrent streaming chat requests, max_tokens=256, sustained across the threshold. The test returned 7,722 successful HTTP 200 chat responses.
• CUDA_INJECTION64_PATH explicitly pointed to the above upstream release asset at process startup. Activity recording began before vLLM captured its CUDA Graphs. Only CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL was enabled by parcagpu; no memory activity kinds or PC sampling were enabled.
• The vLLM process loaded only the upstream parcagpu library and one libcupti.so.13. A system bootstrap library was mapped but did not override the explicit injection path. The eBPF consumer was not needed for the timestamp reproduction.

The upstream asset requires GLIBCXX_3.4.29, which the host's default libstdc++ lacks. We supplied an existing newer libstdc++ and the existing CUPTI through a private per-process LD_LIBRARY_PATH; no system libraries or GPU driver were changed.

Direct observations

We interposed cuptiActivityGetNextRecord only to inspect each raw record before parcagpu sampling/USDT output. The wrapper resolves the actual loaded CUPTI function, calls it exactly once, and returns its result unchanged. It does not change activity kinds, buffer limits, the flush policy, or injection timing. The wrapper logged a non-null real function address; this run had zero CUPTI_ERROR_UNKNOWN errors.

forwarded=25028431 kernels=24987198 zeros=0 invalid=0 first=0
FIRST_INVALID kernel #25099665: start=1789389883393552621 end=1789389883373107821 (end < start)
forwarded=25145582 kernels=25104156 zeros=4268 invalid=4269 first=25099665
forwarded=25237846 kernels=25196268 zeros=96380 invalid=96381 first=25099665

Here zeros means both start and end are zero; invalid additionally includes either timestamp zero or end < start. GPU total memory for this process rose from approximately 12,742 MiB after startup to 13,739 MiB near the first invalid record. The data show growth to a capacity boundary, not an unbounded memory leak.

For comparison, our locally modified ABI-compatible CUDA 13 library, which also enables only concurrent-kernel activity, produced its first invalid record at #25,099,889 under the same A10/vLLM workload—only 224 records after the upstream result. This rules out our backport as a necessary cause.

An independent native CUDA/CUPTI reproducer, with no parcagpu or eBPF agent, showed the same capacity dependence on both A10 and H20: a Graph replay containing a 64 KiB device-to-device copy followed by a tiny kernel, with early CUPTI initialization and device buffer pool limit 8, stopped delivering kernel records at approximately 400,000. Enabling CUPTI_ACTIVITY_KIND_MEMCPY before Graph creation allowed 1,000,000 replays to complete; enabling MEMSET instead did not. The analogous Graph-memset case required MEMSET. Non-Graph copies, or Graph HtoD/DtoH copies, did not fail in that run. Increasing the pool limit to 16/32 delayed the stop to approximately 800,000/1,600,000 kernel records. This is a separate native symptom (records stop, rather than zero timestamps), so we do not equate their surface behavior.

Separately, when no CUPTI API is called until after cudaGraphInstantiate, a kernel-only collector observed 30 million valid kernel records on H20 with stable GPU memory. Delaying only cuptiActivityEnable after earlier CUPTI calls did not have that effect. This is a timing-dependent workaround, not a reliable general fix if new Graphs are created after attachment.

Questions

1. Can the upstream collector handle CUDA Graph memcpy/memset nodes without accumulating unusable device activity buffers when it subscribes before Graph creation but enables only kernel activity?
2. Is enabling the matching memory activity kinds (and discarding their records downstream) an acceptable mitigation, or is there a lower-overhead way to release/reuse the device buffers?
3. Are there known CUPTI version or CUDA Graph API conditions under which this is already fixed? The observed device-buffer behavior may require NVIDIA's input; we have not identified a specific line in the closed-source CUPTI implementation.

The raw log excerpt, exact release-asset hash, and a standalone CUDA reproducer are available if useful. We can share a smaller sanitized reproducer directly in this issue.

[123liuziming]

Follow-up for the latest release, v0.3.2 (caad3a3): I built the exact release tag with its pinned Proton submodule (bfc06a7) because the release asset CDN was unreachable from this test environment. I added only raw-record counters before completeBuffer's correlation filter, and attached the cuda_correlation USDT before starting vLLM so the normal activity-buffer path was active before CUDA Graph capture.

With the default CUPTI device buffer pool, the A10/vLLM 32-concurrent streaming workload reached 22 million raw kernel records in 15 minutes, all timestamps valid. This stopped below the prior ~25.1-million failure point and does not establish that the default-pool case is fixed.

For a quick reproduction, I set only CUPTI_ACTIVITY_ATTR_DEVICE_BUFFER_POOL_LIMIT=8 before registering activity callbacks; cuptiActivitySetAttribute returned success. The first invalid raw kernel timestamp occurred at record 899,477 (end < start). At record 1,000,000, 100,196 records had start=end=0 and 100,197 had invalid timestamps. This is the same failure mode at a proportionally lower record threshold. GPU memory was about 12,771 MiB at failure, versus ~13,739 MiB in the previous default-pool v0.1.2 run.

v0.3.2 still enables only CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL. PR #22 rate-limits graph-launch correlation/USDT emission, but raw CUPTI activities are still generated and read before the filter. The reduced-pool result shows this rate limit does not address the underlying invalid raw timestamp behavior. The pool-8 build includes diagnostic counters and this one attribute setting; it is not the official prebuilt asset. The earlier v0.1.2 reproduction did use the unmodified official asset and default pool.

[123liuziming]

Follow-up with a candidate fix, using the same v0.3.2 source-tag diagnostic build and the same A10/vLLM workload as above. Both runs set CUPTI_ACTIVITY_ATTR_DEVICE_BUFFER_POOL_LIMIT=8 before CUDA context creation and attached the cuda_correlation USDT before vLLM startup; no late attach was used.

• CONCURRENT_KERNEL only: first invalid raw kernel timestamp at record 899,477; by 1 million kernel records, 100,196 had start=end=0. GPU memory at failure was about 12,771 MiB.
• CONCURRENT_KERNEL plus MEMCPY and MEMSET: both additional cuptiActivityEnable calls returned CUPTI_SUCCESS. Over 6 million raw kernel records were read with zero invalid/zero timestamps. GPU memory stayed about 12,751 MiB at the 3M and 6M checkpoints. The activity-buffer callback traversed the memory records but did not emit them as kernel events.

This points to a graph-node activity-kind dependency, not an inherent problem with attaching CUPTI before graph creation. Our standalone CUDA Graph reproducer previously showed that a D2D-copy graph needs MEMCPY enabled, a memset graph needs MEMSET, and enabling an unrelated kind does not help. Please consider enabling the relevant memory activity kinds when collecting per-node CUDA Graph kernel timing, even if the consumer only exports kernel records. A production fix still needs performance and longer-run validation. NVIDIA's CUPTI documentation describes kernel, memcpy and memset as the per-node graph trace activity kinds, but I have not found a documented requirement that all kinds be enabled to avoid timestamp loss.

[123liuziming]

Resolved after upgrading CUPTI to 13.4.x