Bayesium Evaluation Queries

This repository contains the evaluation queries used in the paper "Bayesium: An Agentic RAG Interface for Traceable Exploration of Bayesian Network Literature" (PGM 2026).

Overview

The evaluation is split into four components:

1. Routing (100 queries): 25 BN, 24 MD, 26 CC, 25 OOC
2. Retrieval (384 gold chunks): Stratified sample of 384 paragraphs (96 per document-length quartile) with synthetically generated questions
3. Generation (50 BN queries): Conceptual questions about Bayesian networks
4. Text-to-SQL (18 queries): Bibliographic metadata queries

File Structure

routing_queries.json          # 100 hand-written routing queries with gold labels
retrieval_queries.json        # 384 (chunk_id, synthetic_question) pairs
generation_queries.json       # 50 BN conceptual questions
text_to_sql_queries.json      # 18 metadata filter queries

Each JSON file follows the format:

routing_queries.json

[
  {
    "id": 1,
    "query": "What is d-separation in Bayesian networks?",
    "gold_category": "BN"
  },
  ...
]

retrieval_queries.json

[
  {
    "id": 1,
    "chunk_id": 12345,
    "synthetic_question": "How does the K2 algorithm handle structure learning?",
    "doc_id": 789,
    "doc_title": "Learning Bayesian Network Structures"
  },
  ...
]

generation_queries.json

[
  {
    "id": 1,
    "query": "What is the K2 score and how is it used to learn Bayesian network structure?"
  },
  ...
]

text_to_sql_queries.json

[
  {
    "id": 1,
    "query": "papers by Pearl on causality after 1995",
    "gold_sql": "SELECT doc.id, doc.title, doc.year FROM doc JOIN doc_author ON doc.id = doc_author.doc_id JOIN author ON doc_author.author_id = author.id WHERE author.last_name = 'Pearl' AND doc.year > 1995 AND doc.title ILIKE '%causal%'",
    "gold_result_count": 12
  },
  ...
]

Citation

If you use these queries, please cite:

@inproceedings{paniagua2026bayesium,
  title     = {Bayesium: An Agentic RAG Interface for Traceable Exploration of Bayesian Network Literature},
  author    = {Paniagua, Kevin and Raposo, Adrián and Larrañaga, Pedro and Bielza, Concha},
  booktitle = {Proceedings of Machine Learning Research (Probabilistic Graphical Models)},
  year      = {2026}
}

License

These queries are released under CC BY 4.0.