DataTrace evaluates whether an AI agent can diagnose a database pipeline failure and support its diagnosis with evidence. Each task contains a faulty database object—such as a source column, transformation, constraint, or query—and an observable problem elsewhere in the pipeline. The agent must identify the faulty object, repair it, and report the artifacts it inspected.
DataTrace represents each pipeline as a graph whose nodes are database artifacts (schemas, data, queries, constraints, plans, tests, logs, and issue reports). Dependencies between those artifacts determine the reference answer and the reference evidence path: the artifacts that connect the observed problem to its root cause. This allows the benchmark to score both whether the agent found the right answer and whether its investigation covered the relevant evidence.
Many database benchmarks grade only the end result: whether a query is correct or a repair passes the tests. DataTrace also evaluates the diagnostic process:
- Scores evidence as well as the answer. Each task includes an artifact dependency graph and a reference evidence path. DataTrace measures which relevant artifacts the agent inspected alongside root-cause accuracy.
- Generates reference labels from known changes. DataTrace introduces a controlled fault, records its location, and uses pipeline dependencies to construct the reference path. This avoids writing a separate reasoning trace by hand for every task.
- Varies task difficulty systematically. The current corpus covers 83 runnable environments in 62 domains, seven fault types, and settings that vary path length, distracting evidence, and artifact types.
- Root cause: the single database object where the fault was introduced.
- Observed symptom: the failure or incorrect result visible to the agent.
- Reference evidence path (
π*): the relevant artifacts linking the observed symptom to the root cause. - Grounded answer: a correct root-cause answer for which the agent inspected at least half of the reference evidence path.
- Distractor (called a
decoyin task files): plausible but irrelevant evidence added to make a task harder. - Artifact type (called a
modalityin task files): a category of evidence, such as data, query code, a test, a log, or an issue report.
codebase/
├── src/pilot/ # the DataTrace harness (installable Python package)
│ ├── env.py # typed-action environment (inspect/profile/trace_lineage/run_sql/...)
│ ├── graph.py # typed object graph G=(V,E) + lineage recovery (sqlglot)
│ ├── faults.py # fault operators (null / dup-PK / join-key / rename / temporal / ...)
│ ├── project_*.py # buildable environments (ecom, retails/TPC-H, warehouse)
│ ├── agent.py # ReAct-DS scaffold (OpenAI / OpenRouter tool-calling)
│ ├── scoring.py # L1 outcome (RC@1, Fix) + L2 evidence (ER, EP, EF1, LG)
│ ├── rating.py # EG-Elo — evidence-grounded self-play rating
│ ├── run_leaderboard.py # run the frozen leaderboard corpus
│ ├── run_selfplay_eval.py # CoSE co-evolutionary self-play protocol
│ ├── run_llm_generator.py # generate controlled faults with an LLM
│ ├── repos/ # build tasks from incidents in public dbt repositories
│ └── adapters/ # cross-framework eval adapters (SWE-agent, Spider2, Alpha-SQL)
├── data/ # task specs + corpus manifest + corpus statistics
├── leaderboard/ # frozen leaderboard results + supplementary analyses
└── website/ # GitHub Pages site (static; leaderboard + overview)
python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
export PYTHONPATH=src # or: pip install -e .Model access uses an OpenAI-compatible endpoint. All proprietary models in the leaderboard are reached through a single OpenRouter key — no key is bundled; supply your own:
export OPENROUTER_KEY=sk-or-... # or OPENAI_API_KEY for the OpenAI endpointRun the benchmark on the built-in environments (introduce a fault → run an agent → score its answer and evidence):
python -m pilot.run_pilot \
--projects ecom,retails --scaffolds react_ds,swe,spider \
--models gpt-4o-mini --repeats 2Reproduce a leaderboard entry on the frozen corpus:
python -m pilot.run_leaderboard \
--models anthropic/claude-sonnet-5 \
--projects ecom,retails,warehouse --repeats 2 --workers 4 \
--out results_leaderboardRun the optional self-play experiment (the task generator proposes faults, the diagnostic agent solves them, and the evaluator updates the rating):
python -m pilot.run_selfplay_eval data/tasks.jsonl \
--model gpt-4o-mini --rounds 4 --gen-mode hybridOutputs land in results_*/: episodes.json (per-episode outcome + evidence
scores) and an aggregate leaderboard.{md,json,csv}.
| Metric | Meaning |
|---|---|
| RC@1 | Root-cause accuracy: the submitted object exactly matches the reference object. |
| RC@1ᵍ | Grounded accuracy: the root cause is correct and evidence recall is at least 0.5. This is the primary ranking metric. |
Evidence gap (infl) |
RC@1 − RC@1ᵍ: the share of all runs with a correct answer but insufficient evidence. |
| LG | Unsupported-correct rate: equivalent to the evidence gap above. |
| CUR | Conditional unsupported rate: the fraction of correct answers that have insufficient evidence. |
| ER, EP, EF1 | Recall, precision, and F1 for the artifacts inspected by the agent compared with the reference evidence path. |
| Fix | Repair success: the repaired outputs match the reference outputs and all tests pass. |
| EG-Elo | Rating used by the optional evidence-aware self-play experiment. |
See leaderboard/leaderboard.md for the current
standings and leaderboard/analysis/ for the
environment-clustered CIs, rank stability, and capability–grounding analyses.
data/corpus_manifest.jsonl is the full instance manifest; data/tasks.jsonl,
data/incident_tasks.jsonl, and data/birdwh_tasks.jsonl are the task specs.
Corpus composition (environments, fault types, dependency-path length, artifact
types, and difficulty tiers) is summarized in data/corpus_stats.md and
data/corpus_diversity.md.
@misc{datatrace,
title = {DataTrace: Evidence-Grounded Root-Cause Localization for Databases},
author = {The DataTrace Authors},
howpublished = {\url{https://anonymous.4open.science/r/datatrace}},
year = {2026}
}Repository code is released under the MIT License. Third-party datasets and external projects used with DataTrace may have separate licenses; review their terms before redistributing their data.