XRT Application Capture and Replay by stsoe · Pull Request #9828 · Xilinx/XRT

stsoe · 2026-05-25T01:28:01Z

Problem solved by the commit

Applications using XRT often involve complex, multi-layered software stacks (VAIML, PyTorch, TensorFlow) that make debugging, performance analysis, and regression testing difficult. There was no mechanism to capture the low-level XRT API interactions and replay them independently of the application stack. This commit introduces a capture and replay system that records XRT execution at the frame level (xrt::run::start() or xrt::runlist::execute() boundaries) and enables replay through a standalone executable.

How problem was solved, alternative solutions (if any) and why they were rejected

The solution uses a frame-based capture approach with buffer invalidation tracking. Key design decisions:

Capture Strategy: Uses polymorphic wrapper classes (run_impl_debug, runlist_impl_debug) to intercept XRT API calls with minimal performance impact when disabled. Macro XRT_REPLAY_CAPTURE with XRT_UNLIKELY branch hint ensures near-zero overhead when capture is disabled (capture_frames=0).

Buffer Management: Implements BO invalidation - buffer data is only dumped when xrt::bo::sync(XCL_BO_SYNC_BO_TO_DEVICE) is called. Run objects track which buffers are valid, avoiding redundant dumps when the same data is used across multiple frames.

Artifact Deduplication: Uses FNV-1a hash to deduplicate xclbins, ELF files, and buffer data, preventing duplicate storage of identical artifacts.

Wait Synchronization: Captures xrt::run::wait() and xrt::runlist::wait() calls and associates them with the current active frame. Replay executes waits in the same order, correctly handling asynchronous execution.

Risks (if any) associated the changes in the commit

Performance when enabled: Capture adds file I/O overhead proportional to buffer sync frequency. Measured impact is acceptable for debug/analysis workflows but should not be enabled in production.
Thread safety: The frames singleton uses std::mutex to protect shared state. All capture functions are thread-safe, but heavy contention under high-frequency API calls from multiple threads could cause slowdown.
Memory growth: Long-running captures with many unique ELF files will accumulate memory until process exit. Acceptable for typical debug sessions but could be problematic for extended captures.
Breaking changes: None. Capture is disabled by default and requires explicit xrt.ini configuration.

What has been tested and how, request additional testing if necessary

Tested:

Basic capture/replay with single run per frame
Runlist capture/replay with multiple runs per frame
Buffer invalidation with repeated xrt::bo::sync() calls
ELF flow (programs) and xclbin flow
Artifact deduplication (same xclbin/buffer used multiple times)
Wait synchronization across multiple queued frames
Stream seeking fix for elf_ctor (istream constructor)

Recommended additional testing:

Stress test: 1000+ frames, 100+ runs per frame
Large buffers (>1GB) to validate mmap efficiency
Multi-threaded applications with concurrent API calls
Edge cases: run started before set_arg, runlist with no runs
Performance regression test with capture disabled (should be <1% overhead)
Validation: compare output buffers from original vs replayed execution

Documentation impact (if any)

New documentation added in src/runtime_src/core/common/runner/replay.md covering:

Quick start guide with xrt.ini configuration
Replay executable usage and command-line options
JSON schema specification
Use cases (debugging, performance analysis, regression testing)
Troubleshooting guide
Advanced topics (artifact deduplication, ELF vs xclbin flow, module caching)

Configuration changes in xrt.ini:

[Runtime]
capture_frames=<num>          # Number of frames to capture (0=disabled, default)
capture_output_dir=<path>     # Output directory (default: ./)
New executable: xrt-replay for replaying captured execution.

stsoe · 2026-05-25T01:34:02Z

@sonals . This is a monster PR, but I think safe since capture is non-intrusive and default disabled. I need to do a lot more testing, but feel like getting these changes off my hand for now. You may want to browse to the replay.md file in this PR, the md file is written by Claude. The code was written by me, but reviewed by Claude.

github-actions