This repo hold the dataset and evaluation code of the paper “Benchmarking Contextual Understanding for In-Car Conversational Systems”. The code is used to analyse LLM-based judgement for different large language models and various prompting techniques.
The dataset can be found here: dataset
The dataset is licensed under the accompanied license.
The evaluation code can be accessed via the following link: code
The code is licensed under the accompanied license.
The overall performance of LLM-based judgement techniques and models on the benchmarking dataset is provided below:
More results you can find in the paper.
When using the dataset our refernce to our results please cite the paper as follows.
@misc{habicht2025benchmarking,
title={Benchmarking Contextual Understanding for In-Car Conversational Systems},
author={Philipp Habicht and Lev Sorokin and Abdullah Saydemir and Ken E. Friedl and Andrea Stocco},
year={2025},
eprint={2512.12042},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.12042},
}