Add data-loading bottleneck diagnostic skill

[rostan-t]

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Create a skill allowing agent to detect and localize data loading bottlenecks in PyTorch training.

The workflow roughly contains two important phases:

• Detection of whether or not there's a data loading bottleneck using DALI's LoaderEvaluator
• Bottleneck localization with profiling

The complete workflow is more complex and looks as follows:

1. Identify the workload, environment and if it's at all possible to run.
2. Run (bounded) training, measure loader wait. If LoaderEvaluator is or can be made importable , continue to 3. Else, go to 4.
3. Replay the training with cached batches using LoaderEvaluator's replay mode. Establish if there's a measured bottleneck. If we this steps provdes that there's no bottleneck, go to 5. Otherwise, continue to 4. to localize.
4. Run profiling to localize the bottleneck if we come from 3., or to detect it if coming from 2.
5. Report the results.

This was tested on multiple models, with different environment requirements, including simple ResNet50, OpenCLIP, LeWorldModel, and Hugging Face timm.

In terms of the model used by the agent, I found that GPT 5.6 Terra or equivalent is a sound minimum requirement to run this reliably, although it can sometimes fail to accurately follow all instructions, like the shape of the report.

I unfortunately couldn't commit evaluations as the skill CI doesn't support GPU environments.

Additional information:

Affected modules and functionalities:

Skill created.

Key points relevant for the review:

• Is the skill's workflow sound?
• Can the skill miss an existing bottlenecks or detect one where there isn't?
• Is the skill too defensive on things like requiring equivalent work for replay?

Note that if you run locally, depending on the workload, it may take time to complete.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Skill CI doesn't support GPU workers yet

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other (skill)
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4796

[greptile-apps]

Greptile Summary

The PR adds an agent skill for diagnosing PyTorch data-loading bottlenecks through preflight checks, bounded Real/Replay comparisons, and Nsight Systems profiling.

• Adds workflow instructions, profiling and DALI replay references, and a structured report template.
• Adds environment/evaluation definitions for input-bound, compute-bound, and out-of-scope workloads.
• Adds scripts for collecting environment readiness and summarizing Nsight trace data.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
skills/data-loading-bottleneck/SKILL.md	Defines the end-to-end diagnostic routing, equivalence requirements, verdict thresholds, and reporting contract.
skills/data-loading-bottleneck/references/profiling.md	Documents production-path NVTX instrumentation, Nsight capture validation, and evidence requirements for localization.
skills/data-loading-bottleneck/references/pytorch-dali.md	Documents bounded LoaderEvaluator construction and Real/Replay work-equivalence checks.
skills/data-loading-bottleneck/scripts/collect_preflight.py	Collects workload, environment, GPU, dependency, filesystem, and Git readiness into a preflight artifact.
skills/data-loading-bottleneck/scripts/summarize_nsys.py	Exports and summarizes the selected NVTX window, CUDA activity, worker coverage, and loader-wait overlap.
skills/data-loading-bottleneck/evals/evals.json	Defines representative input-bound, compute-bound, and inference scenarios for evaluating skill behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Preflight workload and environment] --> B{Workload runnable?}
    B -- No --> I[Report INCONCLUSIVE]
    B -- Yes --> C[Run bounded Real workload]
    C --> D{LoaderEvaluator available?}
    D -- Yes --> E[Run equivalent Replay]
    E --> F{Replay speedup}
    F -- <= 1.10x --> N[Report NOT DETECTED]
    F -- > 1.10x --> P[Profile production path]
    D -- No --> P
    P --> G{Validated input-path delay?}
    G -- Yes --> H[Report DETECTED or POTENTIAL and localize]
    G -- No --> I

Loading

_{Reviews (3): Last reviewed commit: "Add eval suite for data-loading bottlene..." | Re-trigger Greptile}

[JanuszL]

/nvskills-ci

[JanuszL]

/nvskills-ci

[mdabek-nvidia]

Is this useful only for only pyTorch based training?

[mdabek-nvidia]

Is the intent to update version number manually? Can it re-use DALI versioning ?

[mdabek-nvidia]

The skill assumes that the platform that we run the preflight is configured so that it can run training.
Would it be possible to extend it to create virtual environment and execute pip install -f requiremnets.txt if such file exist?

[mdabek-nvidia]

Wouldn't it be better to create a venv?

[mdabek-nvidia]

Would it be possible to add a graph (run a python script), that would show how much improvement we get for specific batch sizes and number or workers?

[mdabek-nvidia]

I guess this could be a skill, but I understand that we need 100% reproducibility, correct?