Repository navigation
Gate the first CUDA call on checked capture readiness (Linux) - #171
Merged
Merged
Conversation
On an L4 a whole Trace session's kernel and memcpy records arrived with start == end == 0. They were dropped, as they must be, but the session then claimed no kernel activity record had arrived. Count the records that reach the timestamp check and the ones it rejects (zero-zero separately from other invalid timing), and report kernel_timestamps_unavailable, _invalid or _partial (memcpy alike) as the capability reason with the counts. Statuses are unchanged, so the server's classification is unaffected.
The injected library started CUPTI from a detached worker while the application thread went on into context creation and module loading; on an L4 whole Trace sessions lost every kernel timestamp that way, and nothing checked whether capture was ready. CaptureGate separates the initializer lifecycle from a latched readiness result and from coverage notes. The first supported runtime call triggers driver init through the real cudaGetDeviceCount (driver calls through a checked cuInit) and waits, within one CLOCK_MONOTONIC deadline, until the trigger and the worker have both returned and the Trace subscription and every required activity kind are on. A timeout, a late success, a failed trigger, a missing callback or an unowned first touch is latched and never becomes ready. Manual mode keeps one synchronous initializer in the handshake; --warmup is reported as an unqualified capture. The library writes a per-pid bootstrap status file without the logger, and the session gets capture_readiness and capture_coverage rows. The launcher sets the protocol, warmup and status variables, polls in every mode, and kills a child whose initialization outlives the published deadline. At exit an initializer still live after the deadline ends the process with status 86 instead of running teardown under it. Design: agent_comm l4-qualification-20261015 v3 + v3.1 (Codex CLEAR).
A stand-in libgpufl_inject.so built from the real gate, Linux glue and typed CUDA wrappers, with only gpufl::init faked, runs a target linked against a fake libcuda.so.1 and libcudart.so.13 that call InitializeInjection from inside cuInit like the driver does. The scenarios cover the runtime and cudaSetDevice routes, an unwrapped and a Driver API first touch, the Rust bootstrap order in manual mode, trigger failure after the worker succeeded, a missing callback, a timeout, exit with a wedged initializer (status 86, no teardown) and with a late one (teardown after it), a process without CUDA and an accidental preload, plus the real `gpufl trace` watchdog in natural-exit and windowed runs and for a trigger blocked after its worker succeeded. They found that the worker published Ready after waking the application, so a process could exit with a stale pending record and the worker still used gate state after Finished. Snapshots are now published under the gate mutex.
Warmup: the initialization deadline now includes the requested delay once, so a healthy warmup longer than the init allowance is not killed, and an exit that starts during the delay stops the worker before CUPTI (exit_before_capture) instead of waiting it out. The delay hook is gone. Facilities: every required activity kind's enable result is recorded, at start and at the re-enable after engine start. A non-success latches, including the results CUPTI logs as benign, and a required kind lost after readiness adds trace_facility_lost at shutdown. The facility query no longer depends on the monitor's initialized flag, which shutdown clears before the final capability report. Launcher: the read after exit is combined with the last accepted record; a missing, malformed, foreign or older final record is reported as such next to the accepted one instead of being replaced by it. The status parser requires every field the library writes and refuses states the library cannot be in. Tests: every first-touch runtime wrapper and the copy and sync wrappers end to end, the driver wrappers after an unwrapped cuInit, exit during a warmup, a healthy long warmup and a wedge after it under the real launcher, the required-kind tally, and the final-read combination. The comments no longer state the attach overlap as the established cause.
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On an L4 (2026-10-16), whole
gpufl traceTrace sessions received every kernel and memcpy record withstart == end == 0, so they had no kernel timeline. The injected library started CUPTI on a detached worker while the application went on into context and module setup, and nothing checked whether capture was ready. The loss is consistent with that overlap. The CUPTI-internal cause is not established, and real-GPU requalification is what validates this change.What changes
kernel_timestamps_unavailable,_invalidor_partial(and the memcpy equivalents) with the counts, instead of "no kernel activity record arrived".include/gpufl/inject/capture_gate*.{hpp,cpp}), Linux only.cudaGetDeviceCount(driver calls through a checkedcuInit).CLOCK_MONOTONICdeadline, until:CUPTI_SUCCESS.GPUFL_INJECT_MANUAL_INIT=1: Numba, and the backend's Rust and Python hosts) keeps one synchronous initializer in the handshake.--warmupis budgeted once and reported aswarmup_unqualified. An exit during the warmup stops the worker before CUPTI.capture_readinessandcapture_coveragerows.no_known_lossis explicitly weaker than completeness.status-<pid>.jsonwithout the logger.Limits (also in the CHANGELOG)
unsupported_first_touch.Tests
CUDA 13.1 devel container, CPU only, CI recipe, serial
ctest: 668/668.CaptureGateEndToEnd: 21 scenarios. They use a fakelibcuda.so.1that callsInitializeInjectionfrom insidecuInit, a fakelibcudart.so.13, and a stand-in injection library built from the real gate, glue and typed wrappers. They cover every first-touch runtime wrapper, copy and sync, the driver wrappers, manual and Rust bootstrap order, failures, timeout, exit wedged, late and during-warmup, and the realgpufl tracewatchdog in natural-exit, windowed and warmup runs._Exit(86)or the warmup budget makes tests fail.Not tested here: any GPU, the Windows and macOS builds, and the wheel. Fakes do not establish real driver lock ordering or the CUPTI cause. Before claiming the issue resolved, an L4 requalification is required: delays of 1 ms and 100 ms, the manual route ordering log, a no-CUDA target, and the July 1 ms crash as a regression. The backend's 100 ms default stays unchanged.
The design (v3 and v3.1) and this implementation were reviewed and cleared before this PR.
🤖 Generated with Claude Code