Gate the first CUDA call on checked capture readiness (Linux)

CHANGELOG.md

@@ -168,6 +168,43 @@ types are additive.
   `--agent-drain-ms`, `--agent-cursor` and `--all-sessions` control it, and the
   launcher sends upload-complete only after a clean agent drain. `--upload` is
   scoped to a per-run folder rather than the shared `--output` directory.
+- **Linux injection lifecycle (`gpufl trace`).**
+  - **Warmup.** `--warmup` now sets `GPUFL_INJECT_WARMUP_MS` instead of
+    `GPUFL_INJECT_INIT_DELAY_MS`. Nothing waits for a warmup, so a warmup run
+    is reported as an unqualified Trace capture. The initialization deadline
+    includes the warmup once, so a long healthy warmup is not killed. A
+    process that exits during the warmup stops the worker before CUPTI starts
+    (`exit_before_capture`), and normal teardown runs. A new library launched with
+    a long `GPUFL_INJECT_INIT_DELAY_MS` and without the launcher's
+    `GPUFL_INJECT_PROTOCOL` (an older launcher, or a direct injection) records
+    `legacy_launcher_delay`. The launcher and the library ship together in one
+    wheel. `GPUFL_INJECT_SYNC_WAIT_MS` is no longer read: one deadline covers
+    every wait.
+  - **Status file.** The launcher gives each child a status directory
+    (`GPUFL_INJECT_STATUS_DIR`, `GPUFL_INJECT_STATUS_NONCE`). The library
+    writes `status-<pid>.json` there without the logger, and the launcher
+    prints the result as `capture readiness: ...`. If the read after exit
+    does not confirm the last accepted record (missing, malformed, foreign or
+    older), the summary says so and shows that record as history. The status
+    is diagnostics, not qualification authority. Writing it can hold the gate
+    on a slow filesystem past the nominal deadline.
+  - **Watchdog.** In every mode, including a natural exit, the launcher kills
+    (SIGKILL) a child whose capture initialization outlives the published
+    deadline plus `GPUFL_INJECT_WATCHDOG_GRACE_MS` (default 5000). It does not
+    bound the application's own run time.
+  - **Watchdog limits.**
+    - It covers only a paired library after its first status record. It does
+      not cover an older library, a manual host's own `cuInit` before its
+      handshake, a failed status write, or a wedge before the first record.
+    - SIGKILL plus a blocking reap is not a bound on a process stuck in the
+      kernel.
+  - **Exit status 86.** If an initializer is still inside CUDA/CUPTI at exit
+    after the deadline, the process now ends with status 86. Without this, the
+    remaining exit handlers and destructors would run under that initializer.
+    Application exit handlers registered after the first CUDA call still run
+    before that check: that overlap is not controlled.
+  - **Driver API hosts.** A host that calls `cuInit` itself, without the
+    manual handshake, is reported as `unsupported_first_touch`.
 
 ### Deprecated
 
@@ -176,6 +213,45 @@ types are additive.
 
 ### Fixed
 
+- **Linux `gpufl trace` capture readiness.** The injected library started
+  CUPTI on a worker thread while the application went on into context and
+  module setup. On an L4, whole Trace sessions received every kernel and
+  memcpy record with zero timestamps. Nothing checked whether capture was
+  ready. Whether this race is the CUPTI-internal cause is not established,
+  and real-GPU requalification validates the fix.
+  - **The first call waits.** The application's first supported CUDA runtime
+    call now triggers driver initialization itself (through the real
+    `cudaGetDeviceCount`) and waits, within one deadline, until the driver,
+    the Trace subscription and every required activity kind are ready. A kind
+    counts only when its enable, and its re-enable after the engine starts,
+    returned success: a fallback result CUPTI logs as benign is not an enable.
+    A required kind lost after readiness adds `trace_facility_lost`.
+    `GPUFL_INJECT_LAUNCH_WAIT_MS` (1-600000, default 15000) sets that
+    deadline. The first-touch calls covered are:
+    - `cudaMalloc`, `cudaMallocManaged`, `cudaMallocHost`, `cudaHostAlloc`,
+      `cudaFree`;
+    - `cudaSetDevice`, `cudaGetDeviceCount`, `cudaGetDeviceProperties`,
+      `cudaMemGetInfo`;
+    - `cudaEventCreate*`, `cudaStreamCreate*`;
+    - the driver's `cuMemAlloc_v2` and `cuModuleLoadData*`;
+    - the existing launch, sync and copy wrappers.
+  - **New session rows.** Each session gets a `capture_readiness` row
+    (`ready`, `failed` or `pending`, with the reason) and a `capture_coverage`
+    row. Coverage is `no_known_loss` or `incomplete` with reasons, such as
+    `unsupported_first_touch` for driver init through an unwrapped route,
+    `manual_protocol_violation`, `warmup_unqualified` and
+    `kernel_timestamps_unavailable`. `no_known_loss` is not a completeness
+    proof: a kernel that produced no record cannot be counted.
+  - **Timeouts.** A timeout lets the application continue. The run then stays
+    `failed` even when initialization finishes later.
+  - **Manual handshake.** Hosts that use the manual handshake
+    (`GPUFL_INJECT_MANUAL_INIT=1`: Numba, and the backend's Rust and Python
+    hosts) get one synchronous initializer. `GpuFlightInitializeInjectionAfterCuda`
+    returns 1 only when capture is ready.
+- **Activity records without timing are reported as such.** Kernel and memcpy
+  records that arrive with `start == end == 0` are counted, and the capability
+  reason says `kernel_timestamps_unavailable` (or `_invalid`, or `_partial`),
+  with the counts, instead of "no kernel activity record arrived".
 - **PM/PC sample timestamps under `gpufl trace`.** The deferred engine start
   (the path Windows injection always takes, since gpufl initializes before
   the target creates a CUDA context) built its `EngineContext` without the

CMakeLists.txt

@@ -804,6 +804,12 @@ if(BUILD_GPUFL_INJECT AND ((UNIX AND NOT APPLE) OR WIN32))
     add_library(gpufl_inject SHARED
         include/gpufl/inject/inject_entry.cpp
     )
+    if(UNIX AND NOT APPLE)
+        target_sources(gpufl_inject PRIVATE
+            include/gpufl/inject/capture_gate.cpp
+            include/gpufl/inject/capture_gate_linux.cpp
+        )
+    endif()
     if(UNIX AND NOT APPLE AND GPUFL_HAS_CUDA)
         # Compile CUDA boundary wrappers against the toolkit's official ABI.
         # inject_entry.cpp keeps its header-free fallback for no-CUDA builds.

daemon/launcher/CMakeLists.txt

@@ -21,6 +21,7 @@ endif()
 add_executable(gpufl_launcher
     main.cpp
     agent_launcher.cpp
+    bootstrap_status.cpp
     cli_help.cpp
     cli_parse.cpp
     cli_parse_internal.cpp

daemon/launcher/bootstrap_status.cpp

@@ -0,0 +1,220 @@
+#include "bootstrap_status.hpp"
+
+#include <chrono>
+#include <fstream>
+#include <limits>
+#include <random>
+
+#include "gpufl/core/json/json.hpp"
+
+#if !defined(_WIN32)
+#include <time.h>
+#endif
+
+namespace gpufl::launcher {
+namespace {
+
+constexpr std::size_t kMaxStatusBytes = 4096;
+
+BootstrapStatus Invalid(std::string why) {
+    BootstrapStatus status;
+    status.error = std::move(why);
+    return status;
+}
+
+}  // namespace
+
+std::filesystem::path BootstrapStatusPath(const std::filesystem::path& dir,
+                                          const long pid) {
+    return dir / ("status-" + std::to_string(pid) + ".json");
+}
+
+BootstrapStatus ReadBootstrapStatus(const std::filesystem::path& dir,
+                                    const long pid, const std::string& nonce) {
+    std::ifstream in(BootstrapStatusPath(dir, pid), std::ios::binary);
+    if (!in) return Invalid("missing");
+    std::string text(kMaxStatusBytes + 1, '\0');
+    in.read(text.data(), static_cast<std::streamsize>(text.size()));
+    text.resize(static_cast<std::size_t>(in.gcount()));
+    if (text.size() > kMaxStatusBytes) return Invalid("oversized");
+
+    const json::JsonValue doc = json::parseJson(text);
+    if (!doc.is_object()) return Invalid("malformed");
+    if (doc.value<int64_t>("schema", 0) != 1) return Invalid("schema");
+    if (doc.value<std::string>("nonce", "") != nonce || nonce.empty()) {
+        return Invalid("foreign_nonce");
+    }
+    if (doc.value<int64_t>("pid", 0) != pid) return Invalid("foreign_pid");
+
+    // Every field of schema 1 must be there with its type, and the record
+    // must describe a state the gate can publish (the checks below). This is
+    // an operational diagnostic: a valid record is a consistent report, not
+    // evidence that capture worked.
+    const auto has = [&doc](const char* key, const json::JsonValue::Type type) {
+        return doc.contains(key) && doc[key].type() == type;
+    };
+    using Type = json::JsonValue::Type;
+    for (const char* key : {"seq", "deadline_monotonic_ns", "callbacks_seen",
+                            "handshake_reentries"}) {
+        if (!has(key, Type::Int)) return Invalid("malformed");
+    }
+    if (!has("wait_config_invalid", Type::Bool)) return Invalid("malformed");
+    for (const char* key : {"lifecycle", "owner", "readiness", "reason", "protocol"}) {
+        if (!has(key, Type::String)) return Invalid("malformed");
+    }
+    for (const char* key : {"trigger", "worker", "handshake"}) {
+        if (!has(key, Type::Object)) return Invalid("malformed");
+        for (const char* field : {"expected", "returned", "ok"}) {
+            if (!doc[key].contains(field) || !doc[key][field].is_bool()) {
+                return Invalid("malformed");
+            }
+        }
+    }
+    if (!has("coverage_notes", Type::Array)) return Invalid("malformed");
+
+    BootstrapStatus status;
+    status.seq = doc["seq"].get_int();
+    status.lifecycle = doc["lifecycle"].get_string();
+    status.owner = doc["owner"].get_string();
+    status.readiness = doc["readiness"].get_string();
+    status.reason = doc["reason"].get_string();
+    status.deadline_monotonic_ns = doc["deadline_monotonic_ns"].get_int();
+    status.protocol = doc["protocol"].get_string();
+    for (const json::JsonValue& note : doc["coverage_notes"].get_array()) {
+        if (!note.is_string()) return Invalid("malformed");
+        status.coverage_notes.insert(note.get_string());
+    }
+
+    const std::set<std::string> lifecycles = {"not_requested", "driver_initializing",
+                                              "driver_ready", "capture_initializing",
+                                              "finished"};
+    const std::set<std::string> owners = {"none", "gate", "callback", "manual"};
+    const std::set<std::string> readiness = {"pending", "ready", "failed"};
+    if (status.seq < 1 || !lifecycles.count(status.lifecycle) ||
+        !owners.count(status.owner) || !readiness.count(status.readiness)) {
+        return Invalid("inconsistent");
+    }
+    const bool requested = status.lifecycle != "not_requested";
+    if (requested != (status.owner != "none") ||
+        requested != (status.deadline_monotonic_ns > 0) ||
+        (status.lifecycle == "finished" && status.readiness == "pending") ||
+        (status.readiness == "ready" && status.lifecycle != "finished") ||
+        (status.readiness == "failed") == status.reason.empty()) {
+        return Invalid("inconsistent");
+    }
+
+    // Participants: returned needs expected, ok needs returned; the owner
+    // decides which may be expected; Finished needs every expected one
+    // returned, and Ready needs every expected one ok.
+    struct Participant { bool expected, returned, ok; };
+    const auto participant = [&doc](const char* key) {
+        const json::JsonValue& p = doc[key];
+        return Participant{p["expected"].get_bool(), p["returned"].get_bool(),
+                           p["ok"].get_bool()};
+    };
+    const Participant trigger = participant("trigger");
+    const Participant worker = participant("worker");
+    const Participant handshake = participant("handshake");
+    bool any_expected = false;
+    for (const Participant& p : {trigger, worker, handshake}) {
+        if ((p.returned && !p.expected) || (p.ok && !p.returned)) {
+            return Invalid("inconsistent");
+        }
+        any_expected = any_expected || p.expected;
+        if (status.lifecycle == "finished" && p.expected && !p.returned) {
+            return Invalid("inconsistent");
+        }
+        if (status.readiness == "ready" && p.expected && !p.ok) {
+            return Invalid("inconsistent");
+        }
+    }
+    const bool owner_fits =
+        status.owner == "none"     ? !any_expected
+        : status.owner == "gate"   ? trigger.expected && !handshake.expected
+        : status.owner == "callback" ? worker.expected && !trigger.expected &&
+                                         !handshake.expected
+                                   : handshake.expected && !trigger.expected &&
+                                         !worker.expected;
+    if (!owner_fits) return Invalid("inconsistent");
+    status.valid = true;
+    return status;
+}
+
+int64_t MonotonicNowNs() {
+#if defined(_WIN32)
+    return std::chrono::duration_cast<std::chrono::nanoseconds>(
+               std::chrono::steady_clock::now().time_since_epoch())
+        .count();
+#else
+    timespec ts{};
+    clock_gettime(CLOCK_MONOTONIC, &ts);
+    return static_cast<int64_t>(ts.tv_sec) * 1'000'000'000LL + ts.tv_nsec;
+#endif
+}
+
+bool InitWatchdog::Observe(const BootstrapStatus& status, const int64_t now_ns) {
+    if (status.valid) {
+        if (status.seq >= last_seq_) {
+            last_seq_ = status.seq;
+            last_ = status;
+            seen_valid_ = true;
+            if (status.lifecycle == "finished") disarmed_ = true;
+            if (!armed_ && status.lifecycle != "not_requested" &&
+                status.deadline_monotonic_ns > 0) {
+                armed_ = true;
+                deadline_ns_ = status.deadline_monotonic_ns;
+            }
+        }
+    }
+    if (!armed_ || disarmed_) return false;
+    const int64_t max = std::numeric_limits<int64_t>::max();
+    const int64_t limit = deadline_ns_ > max - grace_ns_ ? max : deadline_ns_ + grace_ns_;
+    return now_ns > limit;
+}
+
+FinalBootstrapObservation CombineFinalBootstrap(const InitWatchdog& watchdog,
+                                                const BootstrapStatus& final_read) {
+    FinalBootstrapObservation out;
+    if (!watchdog.seen_valid()) {
+        out.accepted = final_read;  // invalid; its error says why
+        out.final_error = final_read.error;
+        return out;
+    }
+    out.accepted = watchdog.last();
+    if (!final_read.valid) {
+        out.final_error = final_read.error;
+    } else if (final_read.seq < watchdog.last().seq) {
+        out.final_error = "stale_final_seq";
+    }
+    return out;
+}
+
+std::string DescribeBootstrapStatus(const BootstrapStatus& status) {
+    if (!status.valid) return "unavailable (" + status.error + ")";
+    std::string out = status.readiness + " owner=" + status.owner +
+                      " lifecycle=" + status.lifecycle;
+    if (!status.reason.empty()) out += " reason=" + status.reason;
+    if (!status.coverage_notes.empty()) {
+        out += " coverage=";
+        bool first = true;
+        for (const std::string& note : status.coverage_notes) {
+            if (!first) out += ',';
+            first = false;
+            out += note;
+        }
+    }
+    return out;
+}
+
+std::string MakeBootstrapNonce() {
+    std::random_device device;
+    static const char* kHex = "0123456789abcdef";
+    std::string out;
+    for (int i = 0; i < 4; ++i) {
+        const uint32_t word = device();
+        for (int shift = 28; shift >= 0; shift -= 4) out += kHex[(word >> shift) & 0xf];
+    }
+    return out;
+}
+
+}  // namespace gpufl::launcher