Add thapi_start() and thapi_stop()

[thilinarmtb]

I created lttng_ust_toggle event category and implemented thapi_start() and thapi_stop()
user APIs (in a similar spirit to cudaProfilerStart() etc.). The library which the user has to link
against is named libThapi.so.

Usage of the API is as follows:

#include <thapi.h>
//
// code
//
thapi_start();
// code to be profiled
thapi_stop();
//
// more code
//

If libThapi.so is linked, thapi_auto_stop() (which is not part of the user API) is called automatically
during the library load time using __attribute__((constructor)). I added a check in configure.ac to
check if the compiler supports __attribute__((constructor)).

If the user runs iprof with --toggle-on, the traces generated by thapi_auto_stop(), thapi_start()
and thapi_stop() will be respected by babeltrace (otherwise there is no change to iprof/babeltrace
behavior). Since thapi_auto_stop() is called automatically, babeltrace will filter out all of the traces
till it see a thapi_start(). Trace filtering out will start again once the user call thapi_stop(). This API
only affects the trace collection of the calling process/thread.

P.S: This feature requires lttng-ust >= 2.12.8. Specifically, this fix.

TODO

• Respect lttng_ust_profiler_* lttng_ust_toggle_* events when calculating statistics
• Fix failing checks
• Add the filter.toggle.toggle to babeltrace graph

[thilinarmtb]

@TApplencourt : Two tests fail out of the three tests used to check the toggle API. Those two fail
because we have not integrated the toggle plugin to babeltrace_thapi. Once we do that, I think
those two will pass as well.

[TApplencourt]

I can show you how to do it tomorrow : )

[thilinarmtb]

I can show you how to do it tomorrow : )

Works for me.

[TApplencourt]

@thapi-local-reviewer review

[thapi-local-reviewer]

Defects

1. push_downstream uses a global hostname instead of the per-event hostname

File: utils/thapi_toggle_callbacks.cpp, function push_downstream

auto key = ToggleKey{std::string(hostname_s), vpid_v};

hostname_s is a file-scope static that is only written by thapi_start_callback. In a multi-host MPI run (the test toggle_plugin_mpi_np_2 uses mpirun -n 2), processes on different hosts produce events with different hostnames. push_downstream is invoked for every downstream message, but it always looks up the toggle map with the hostname of the last thapi_start it saw, not the hostname of the event currently being filtered.

Trigger: Two MPI ranks on different hosts. Rank 0 on hostA calls thapi_start() (sets hostname_s = "hostA"). A regular API-call event from rank 1 on hostB then arrives at push_downstream; the key is constructed as ("hostA", vpid_rank1) instead of ("hostB", vpid_rank1), so the lookup either misses (event incorrectly dropped) or hits the wrong entry (event incorrectly passed).

The fix is to extract the hostname from the current event/stream context inside push_downstream rather than relying on the global.

---

2. No null-check on the vpid field before dereferencing

File: utils/thapi_toggle_callbacks.cpp, function push_downstream

const bt_field *ccf = bt_event_borrow_common_context_field_const(event);
const bt_field *vpid = bt_field_structure_borrow_member_field_by_name_const(ccf, "vpid");
uint64_t vpid_v = bt_field_integer_signed_get_value(vpid);

If the event's common context is absent (ccf == NULL) or does not contain a member named "vpid" (e.g., a stream whose context layout differs, or a malformed/legacy trace), vpid will be NULL and bt_field_integer_signed_get_value(NULL) is undefined behaviour (in practice a segfault in babeltrace2).

Trigger: Any event that flows through the filter whose common-context structure does not include a vpid member. The YAML in btx_thapi_toggle.yaml declares vpid in the context, but the filter sits after the muxer and sees events from all backends; a backend whose stream class omits vpid from its common context will hit this path.

---

3. Toggle granularity is per-process, not per-thread, despite the stated intent

File: utils/thapi_toggle_callbacks.cpp (all three callbacks and push_downstream)

The PR description says:

This API only affects the trace collection of the calling process/thread.

But every key is (hostname, vpid) — the virtual process id. vtid is received by the callbacks but never used in the key. If two threads in the same process call thapi_start() / thapi_stop() concurrently (or in sequence), the second thread's thapi_stop() will turn off tracing for the first thread as well, because they share the same vpid key.

Trigger: Thread A calls thapi_start(), thread B calls thapi_stop(). Thread A's subsequent API calls are silently dropped because the shared (hostname, vpid) entry is now false.

If per-process granularity is the intended design, the description should be corrected; if per-thread is intended, the key must include vtid.

Reviewed 2e63a99e — local model qwen3.8:27b. Verify before acting.

[TApplencourt]

sorry @thilinarmtb just playing with some bot :p

[thilinarmtb]

@thapi-local-reviewer review

[thapi-local-reviewer]

1. configure.ac — duplicate PKG_CHECK_MODULES for BABELTRACE2

File: configure.ac

The diff adds PKG_CHECK_MODULES([BABELTRACE2], [babeltrace2 >= 2.0]) at line ~128, but the existing line PKG_CHECK_MODULES([BABELTRACE2], [babeltrace2 >= 2.1]) (for blob support) is still present a few lines later. Autoconf's PKG_CHECK_MODULES re-runs pkg-config and overwrites the BABELTRACE2_CFLAGS/BABELTRACE2_LIBS variables each time, so the final values come from the second invocation (>= 2.1). This is not a hard break, but it makes the first check dead code and the version constraint confusing. More importantly, if the intent was to raise the minimum to 2.1, the first check is misleading; if the intent was 2.0, the second check silently overrides it. Pick one.

2. utils/thapi_toggle_callbacks.cpp — push_downstream drops toggle events themselves when tracing is off

File: utils/thapi_toggle_callbacks.cpp, function push_downstream

The filter gates all events (including lttng_ust_toggle:start / lttng_ust_toggle:stop) on the current toggle state. The initial state for a key is false (set by thapi_auto_stop_callback). Consider the sequence:

1. auto_stop arrives → map[key] = false.
2. start arrives → the callback sets map[key] = true, but then push_downstream is called for the same start event. At that point map[key] is true, so it passes. OK.
3. stop arrives → callback sets map[key] = false, then push_downstream sees map[key] == false and drops the stop event.
4. start arrives again → callback sets map[key] = true, push_downstream passes it.

So the stop event is never forwarded downstream. This means any downstream component that needs to see the stop event (e.g., for statistics, or for a second toggle filter in a pipeline) will never observe it. The start event is forwarded (because the callback runs before the filter check and sets the state to true first), but stop is not. This is an asymmetry that will produce incorrect trace output: the trace will show start events but no corresponding stop events, making the trace appear to never stop.

Trigger: any run where thapi_stop() is called. The stop tracepoint event is always dropped.

3. utils/thapi_toggle_callbacks.cpp — push_downstream uses map[key] which inserts a default entry

File: utils/thapi_toggle_callbacks.cpp, function push_downstream

push_msg = map[key]; uses std::map::operator[], which inserts a default-constructed false entry if the key is absent. This means every event from a thread that has never had a toggle event (e.g., a thread that never called thapi_start()/thapi_stop()/thapi_auto_stop()) will:

• Have a false entry inserted into the map (memory growth for every distinct (hostname, vpid, vtid) tuple seen).
• Be correctly filtered out (since false), but the map grows unboundedly.

More critically, if a start event is later received for that key, the callback does map[key] = true, which is fine. But the insertion in push_downstream means the map is no longer a pure record of toggle state — it's polluted with entries for threads that were never toggled. Use find() instead of operator[] to avoid the side effect.

4. utils/thapi_toggle_callbacks.cpp — thapi_auto_stop_callback ignores subsequent auto_stop events, but the dlopen test expects 2

File: utils/thapi_toggle_callbacks.cpp, function thapi_auto_stop_callback

The callback does:

if (auto_map.count(key))
    return;

This means the second auto_stop event for the same (hostname, vpid, vtid) is silently ignored — it does not reset map[key] to false. In the toggle_dlopen test, the library is dlopen/dlclose'd twice. Each dlopen triggers a new constructor call (thapi_auto_stop). If the vpid/vtid are the same (same process, same thread), the second auto_stop is ignored. The test expects auto_stop_count == 2 from babeltrace_thapi -c, which counts events in the raw trace, not after filtering. So the test may pass, but the filter behavior is wrong: after the second dlopen, the toggle state is not reset to "stopped," so events between the second start and stop may be incorrectly included or excluded depending on prior state.

5. utils/thapi_toggle.c — constructor priority arithmetic may overflow or be undefined

File: utils/thapi_toggle.c

void __attribute__((constructor(LTTNG_UST_CONSTRUCTOR_PRIO + 1))) thapi_auto_stop(void) {

LTTNG_UST_CONSTRUCTOR_PRIO is a macro defined by lttng-ust. If it is defined as a value near the maximum allowed constructor priority (GCC allows 0–65535), adding 1 could exceed the valid range, producing undefined behavior or a compiler error. The ax_gcc_func_attribute.m4 check tests constructor_priority with 65535/2, but the actual value of LTTNG_UST_CONSTRUCTOR_PRIO is not checked against the upper bound. If LTTNG_UST_CONSTRUCTOR_PRIO is 65535, +1 overflows.

6. utils/thapi_toggle_callbacks.cpp — push_downstream does not handle non-event messages

File: utils/thapi_toggle_callbacks.cpp, function push_downstream

If bt_message_get_type(msg) is not BT_MESSAGE_TYPE_EVENT (e.g., a stream begin/end, clock snapshot, or component log message), push_msg remains true and the message is forwarded. This is correct behavior, but the code path for non-event messages does not call bt_message_put_ref on the drop path (which is fine since it's not dropped). However, if a future change adds a condition that could set push_msg = false for non-event messages, the bt_message_put_ref call would be incorrect for non-event message types. This is a latent issue, not a current bug.

7. integration_tests/toggle_mpi.c — missing break in case 0 causes fall-through

File: integration_tests/toggle_mpi.c

switch (variant) {
case 0:
    thapi_start();
case 1:
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    if (rank == 0)
      thapi_start();
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    break;

case 0 falls through to case 1. This means when variant == 0, the code calls thapi_start(), then MPI_Comm_rank, then (if rank 0) thapi_start() again, then MPI_Comm_size. The test expects count_calls Total == 2 for variant 0, which would be MPI_Comm_rank + MPI_Comm_size = 2. But the double thapi_start() on rank 0 is unintended. This is a test bug, not a library bug, but it means the test is not testing what the comment says it's testing.

8. utils/Makefile.am — libThapiToggle_la_CFLAGS and CXXFLAGS both set, but the source is .cpp

File: utils/Makefile.am

libThapiToggle_la_CFLAGS = -fPIC -shared -Wall -Wextra -Wno-unused-parameter $(BABELTRACE2_CFLAGS) \
                           -I./btx_thapi_toggle -I$(top_srcdir)/utils/include
libThapiToggle_la_CXXFLAGS = -fPIC -shared -Wall -Wextra -Wno-unused-parameter $(BABELTRACE2_CFLAGS) \
                             -I./btx_thapi_toggle -I$(top_srcdir)/utils/include

Both CFLAGS and CXXFLAGS are set. For a .cpp file, libtool uses CXXFLAGS. The CFLAGS line is harmless but redundant. Not a bug, but worth noting.

9. utils/thapi_toggle_callbacks.cpp — init allocates ToggleMap[2] but finalize does delete[]

File: utils/thapi_toggle_callbacks.cpp

static void init(void **data) { *data = new ToggleMap[2]; }
static void finalize(void *data) { delete[] static_cast<ToggleMap *>(data); }

This is correct: new T[2] is matched by delete[]. No issue.

10. utils/btx_thapi_toggle.yaml — event classes have no fields, but the tracepoint definition also has no fields

File: utils/btx_thapi_toggle.yaml and utils/thapi_toggle_tracepoints.tp

Both define the events with no fields. This is consistent. No issue.

Summary of actionable defects:

1. push_downstream drops stop events (item 2) — the filter's own control events are asymmetrically filtered.
2. map[key] in push_downstream inserts entries (item 3) — use find() to avoid unbounded map growth and side effects.
3. thapi_auto_stop_callback ignores repeat auto_stop (item 4) — breaks the dlopen/dlclose re-initialization scenario.
4. Constructor priority +1 may overflow (item 5) — no bounds check.
5. toggle_mpi.c case 0 fall-through (item 7) — test logic bug.

Reviewed dd52c2f0 — local model qwen3.8:27b. Verify before acting.

[thilinarmtb]

Addressing the review from the bot:

1. Duplicate PKG_CHECK_MODULES: Valid, and LTTNG_UST was duplicated as well. Both came from rebasing onto Blob v2 (#543), which had already raised the checks to babeltrace2 >= 2.1 / lttng-ust >= 2.16. Removed the older checks and moved the constructor comment to the remaining lttng-ust check (2.16 includes the 2.12.8 fix). (dbfe991)

2. stop events dropped: Not the case. Metababel drops every event that has a registered callback right after running the callback (filter_message_iterator_next_call_dispatchers in filter.c.erb), so start, stop and auto_stop never reach push_downstream. All three are consumed alike, so there's no asymmetry. Added toggle_events_consumed, which checks that the raw trace has start/stop and that --toggle-on keeps the MPI calls but forwards no toggle event. (5498022)

3. map[key] inserts entries: Valid, though growth is bounded by one entry per thread, and the result was already correct since a missing key read as false. Switched to find(), treating a missing entry as not traced. (b86ef8a)

4. Repeated auto_stop ignored: Intentional. In the dlopen test, the state is off when the second auto_stop arrives (after the first stop), so ignoring it or resetting it gives the same result at every step, and start always turns tracing on regardless. Ignoring only matters if the library is unloaded while tracing is on. There, resetting would silently stop a trace the application explicitly started.

5. Constructor priority overflow: LTTNG_UST_CONSTRUCTOR_PRIO is 150 in lttng-ust (2.16 and master), so the priority is 151. An out-of-range priority is a compile error, not undefined behaviour. Still added an #error guard for a clearer message if it is ever overridden. (2837343)

6. Non-event messages: Agreed that it's not a current bug. Non-event messages are always forwarded.

7. case 0 fall-through: Intentional. Variant 0 turns tracing on for every rank and then makes the same calls as variant 1, which is why the tests expect Total == 2 on one rank and 2 processes on two ranks. The second thapi_start() on rank 0 just leaves tracing on. Added comments describing each variant, and a /* fall through */ marker. (54eea07)

8. CFLAGS and CXXFLAGS: Both are needed. The library is also built from metababel's generated C sources (btx_component.c, btx_upstream.c, btx_downstream.c, btx_main.c), which use CFLAGS for the babeltrace2 and btx_thapi_toggle include paths. The backend interval plugins set both for the same reason.

9, 10: No change, as noted.

[thilinarmtb]

@thapi-local-reviewer review

[thapi-local-reviewer]

I'll analyze the diff for defects.

Let me trace through the key logic carefully.

utils/thapi_toggle_callbacks.cpp — push_downstream

The filter gates events based on a per-thread toggle state. The state map (maps[1]) is only populated by the start/stop/auto_stop callbacks. The critical question: what happens to a thread that never emitted any toggle event?

In push_downstream, for an event with a valid vpid/vtid/hostname:

auto it = map.find(key);
push_msg = it != map.end() && it->second;

If the thread never called thapi_start(), map.find(key) returns end(), so push_msg = false — the event is dropped. That's the intended "not traced until start" behavior, and it's consistent with the toggle_threads test (worker thread's MPI_Comm_size is not traced). So that's fine.

But there's a real problem with the auto_stop callback and the dlopen test. Let me look at toggle_dlopen.c: it dlopens libThapi.so twice, calling start/stop each time, then dlclose. The test expects auto_stop_count == 2. The auto_stop constructor runs at library load. With RTLD_LOCAL and two separate dlopen calls, the constructor runs twice — that's the test's expectation, so it's consistent. Not a defect in the diff itself.

Now the actual defect I can point at:

utils/thapi_toggle_callbacks.cpp — thapi_auto_stop_callback ignores a second auto_stop for the same key, but the dlopen scenario produces two auto_stop events for the same (hostname, vpid, vtid) key.

In toggle_dlopen.c, both dlopen calls happen in the same process/thread, so both auto_stop events carry the same vpid/vtid/hostname. The first auto_stop sets auto_map[key] = true and map[key] = false. The second auto_stop hits if (auto_map.count(key)) return; and is ignored. That's fine for the filter's purpose (already stopped). Not a defect.

Let me look at the thapi_toggle.c constructor priority more carefully.

#if LTTNG_UST_CONSTRUCTOR_PRIO >= 65535
#error "LTTNG_UST_CONSTRUCTOR_PRIO + 1 is not a valid constructor priority."
#endif

This guards against LTTNG_UST_CONSTRUCTOR_PRIO + 1 overflowing the valid range. That's reasonable.

Now the configure.ac change: it adds AX_GCC_FUNC_ATTRIBUTE(constructor) which defines HAVE_FUNC_ATTRIBUTE_constructor, but nothing in the diff actually uses HAVE_FUNC_ATTRIBUTE_constructor. thapi_toggle.c unconditionally uses __attribute__((constructor(...))) with no #ifdef HAVE_FUNC_ATTRIBUTE_constructor guard. So the configure check is dead — if a compiler lacks the attribute, the build will fail with a hard error rather than being cleanly disabled. That's a broken error-handling / dead-check issue, but it's more of a "the check doesn't do what the comment implies" problem. I'll note it but it's minor.

The most concrete defect:

utils/thapi_toggle_callbacks.cpp — push_downstream drops events for threads that have no toggle state, but the auto_stop constructor only fires when libThapi.so is linked. For a process that links libThapi.so but the user never calls thapi_start(), every event is dropped — including the auto_stop event itself is consumed by the callback, but all backend events are suppressed. That's the documented intended behavior ("babeltrace will filter out all of the traces till it see a thapi_start()"), so not a defect.

Let me focus on what's genuinely broken.

Real defect — utils/thapi_toggle_callbacks.cpp, push_downstream: the hostname lookup can be NULL for non-LTTng streams, and the code only guards the toggle path, but the bt_value_string_get(hostname) is only reached when hostname is non-null. That's fine.

Let me re-examine the toggle_mpi.c fall-through logic against the test expectations, since that's where a logic error would be most visible.

Variant 0: thapi_start() (all ranks), then fall-through to case 1: MPI_Comm_rank, if rank==0 thapi_start() (rank 0 calls start again), MPI_Comm_size. Then thapi_stop().

• Rank 0: start, rank, start, size, stop → traced: rank, size (2 calls)
• Rank 1: start, rank, size, stop → traced: rank, size (2 calls)
• Test toggle_mpi_np_1 variant 0 expects Total == 2. With 1 rank, that's rank+size = 2. ✓
• Test toggle_mpi_np_2 variant 0 expects count_processes == 2. Both ranks traced. ✓

Variant 1: MPI_Comm_rank (not traced, no start yet), if rank==0 thapi_start(), MPI_Comm_size (traced on rank 0 only), thapi_stop().

• Rank 0: rank(not traced), start, size(traced), stop → 1 call
• Rank 1: rank(not traced), size(not traced), stop → 0 calls
• Test toggle_mpi_np_1 variant 1 expects Total == 1. ✓
• Test toggle_mpi_np_2 variant 1 expects count_processes == 1. ✓

OK the C logic matches.

Now the genuine defect I'm confident about:

utils/thapi_toggle_callbacks.cpp — push_downstream uses map.find(key) and drops the event if the key is absent. But the auto_stop callback sets map[key] = false only for the key it sees. Consider the toggle_dlopen case where the same thread does start/stop twice. After the first stop, map[key]=false. The second start sets map[key]=true, second stop sets map[key]=false. Fine.

I've gone back and forth. Let me settle on the defects I can actually point at with a triggering input:

1. Dead configure check (configure.ac + thapi_toggle.c): HAVE_FUNC_ATTRIBUTE_constructor is defined but never consulted; thapi_toggle.c unconditionally uses the attribute. A compiler without the attribute fails the build with no graceful fallback, contradicting the stated purpose of the check.

2. **thapi_toggle_callbacks.cpp push_downstream: events from a thread that has no toggle event are silently dropped, but the auto_stop constructor is the only thing that guarantees a stop-state entry exists. If libThapi.so is linked but the LTTng session was created before the constructor ran (or the constructor's tracepoint was not enabled), the thread has no entry and all its events are dropped with no way to recover — but more importantly, this means a process that links libThapi.so and does call thapi_start() will have its events traced, while a process that links it and relies on the default (no explicit start) is fully silenced. This is intended, so I'll drop it.

Given the instruction to only report defects I can point at with a triggering input/state, and to state uncertainty explicitly, here is my final list.

---

1. utils/thapi_toggle.c + configure.ac — the __attribute__((constructor)) support check is dead code.
configure.ac adds AX_GCC_FUNC_ATTRIBUTE(constructor), which defines HAVE_FUNC_ATTRIBUTE_constructor, and the comment says this is needed "for __attribute__((constructor)) to work properly." But thapi_toggle.c unconditionally uses __attribute__((constructor(LTTNG_UST_CONSTRUCTOR_PRIO + 1))) with no #ifdef HAVE_FUNC_ATTRIBUTE_constructor guard. Trigger: a toolchain whose compiler lacks the constructor attribute (or where the macro check fails) will produce a hard compile error in thapi_toggle.c instead of the build being cleanly disabled. The check the PR added does not actually protect the code that needs it.

2. utils/thapi_toggle_callbacks.cpp — push_downstream drops all events for a thread that has no toggle state, with no fallback, and the auto_stop callback's "ignore if seen" rule can leave a thread permanently un-traced after a dlopen/dlclose/re-dlopen cycle.
In thapi_auto_stop_callback, a second auto_stop for the same (hostname, vpid, vtid) key is ignored (if (auto_map.count(key)) return;). In toggle_dlopen.c the same thread emits auto_stop twice (two dlopens). The first sets map[key]=false; the second is ignored. That's fine within that process. But the broader issue: push_downstream treats "key not in map" as "drop," so any thread that never produced a start event is silently silenced. There is no code path that re-enables a thread once it has been stopped except an explicit thapi_start(). This is the intended semantics, but it means a mis-ordered or missing auto_stop (e.g., constructor not run because the library was loaded via a path that skips constructors, or the tracepoint not enabled) results in total loss of that thread's trace with no diagnostic. I'm flagging this as a robustness/error-handling gap rather than a definite bug; I'm not 100% certain it manifests in the shipped test matrix.

3. utils/thapi_toggle_callbacks.cpp — push_downstream reads vpid/vtid as uint64_t from a signed field and stores them into a key typed int64_t.

uint64_t vpid_v = bt_field_integer_signed_get_value(vpid);
...
auto key = ToggleKey{std::string(...), vpid_v, vtid_v};

ToggleKey is std::tuple<std::string, int64_t, int64_t>. The callbacks build the key with int64_t vpid, int64_t vtid (signed), but push_downstream builds it from uint64_t. For any negative vpid/vtid (which LTTng can emit, e.g. -1 for unknown), the uint64_t→int64_t conversion in the tuple construction is implementation-defined/round-trips to the same bit pattern on two's-complement, so it usually matches, but the type mismatch between the two key-construction sites is a latent correctness hazard: if the field value is read as unsigned and the stored key is signed, a value ≥ 2^63 will not match the callback-side key. Trigger: any event whose vpid or vtid is negative or otherwise outside the positive range. I'm not certain LTTng emits such values in practice, but the two sites disagree on signedness.

4. utils/Makefile.am — libThapiToggle_la_CFLAGS/CXXFLAGS pass -fPIC -shared as compiler flags.
-shared is a linker flag, not a compiler flag; placing it in CFLAGS/CXXFLAGS is ineffective (and -fPIC is redundant given it's already a shared libtool library). This won't break the build on GCC/Clang (they ignore -shared at compile time), so it's low severity, but it's a misplacement that could confuse a stricter toolchain. I'm flagging it as a minor error-handling/build-correctness issue, not a functional bug.

---

The highest-confidence, most concrete defect is #1 (the constructor-attribute check is added but never used, so it provides no protection). #3 is a real signedness inconsistency between the two key-construction sites. #2 and #4 are lower-confidence / lower-severity.

Reviewed 54eea072 — local model qwen3.8:27b. Verify before acting.