Can LLMs Match Folk Wisdom? A Contextual Evaluation of Proverb Selection in Romance Dialects of Italy

This repository contains materials associated to the paper:

Gabriele Errico, Alan Ramponi, Sara Tonelli, and Elisabetta Jezek. 2026. Can LLMs Match Folk Wisdom? A Contextual Evaluation of Proverb Selection in Romance Dialects of Italy. In Proceedings of the 12th Italian Conference on Computational Linguistics (CLiC-it 2026), Palermo, Italy. CEUR Workshop Proceedings. [cite] [paper]

• 📃 DiPRomIt dataset and benchmark
• 🚀 Code for the experiments
• 📖 Further information
• ✏️ Citation

DiPRomIt dataset and benchmark

Two language resources have been created and described in the paper:

• DiPRomIt: the first curated dataset of attested proverbs written in dialects of Romance languages spoken in Italy, accompanied by rich metadata and interlinking information;
• DiPRomIt-scenarios: a multi-choice, scenario-based benchmark for dialectal proverb selection that builds on top of DiPRomIt. It aims to test contextual cultural sensitivity of LLMs, beyond LLMs' memorization of proverbs.

📃 Data request: please write an e-mail to us to request the DiPRomIt dataset and/or the DiPRomIt-scenarios benchmark. The data can be used for non-commercial research purposes only and the user must declare to avoid the redistribution of the dataset to third parties or in online repositories without the consent of the authors, to acknowledge attested sources of proverbs, and to exclude data misuse.

Data format

DiPRomIt data (dipromit.tsv) is provided in a tab-separated format and contains a header line. Each line consists of information about each dialectal proverb (i.e., id, proverb, translation, ...; see below). Specifically, each instance is represented as shown below:

id   proverb                             translation                             meaning                                                     region   province/area  municipality  alternative_ids  variant_ids  reference     URL
829  Tumazzu duru nun pigghia salamora.  Il formaggio duro non prende più sale.  Una persona invecchiata non è in più in grado di cambiare.  Sicilia  Trapani        Erice         828; 830         513          Pitrè (1880)  https://it.wikiquote.org/wiki/Proverbi_siciliani
...

where:

• id: the identifier of the dialectal proverb (integer);
• proverb: the text of the dialectal proverb;
• translation: the literal translation of the dialectal proverb (in Italian);
• meaning: the meaning of the dialectal proverb (in Italian);
• region: the administrative region of origin of the dialectal proverb;
• province/area: the province or historical area of origin of the dialectal proverb, if available;
• municipality: the municipality of origin of the dialectal proverb, if available;
• alternative_ids: a list of IDs for proverb alternatives in the dataset ( ;-separated), if any;
• variant_ids: a list of IDs for proverb variants in the dataset ( ;-separated), if any;
• reference: the source that attests to the existence of the proverb (e.g., book, scientific publication);
• URL: the WikiQuote URL from which the dialectal proverb has been found.

If you are interested in the DiPRomIt-scenarios benchmark, a file containing the hand-crafted contextual scenarios in Italian for the selected proverbs is needed (scenarios.tsv). This file is provided in a tab-separated format and contains a header line. Each line consists of information about each proverb-associated scenario (i.e., id, context/actors, situation/action, consequence/result, hint). Specifically, each instance is represented as shown below:

id   context/actors                                                     situation/action                                                                                                                                 consequence/result                                                                 hint
679  [Una bambina] è con il papà e [la nonna] al mercato {in Sicilia}.  [La bambina] nota della cioccolata in un banco del mercato ed insiste per ottenerla, dopodiché dice al papà: "Sei il papà più bravo del mondo!"  Il papà si fa lusingare [dalla bambina] e le compra il cioccolato tanto agognato.  Che proverbio userebbe [la nonna] in questa situazione?

where:

• id: the identifier of the dialectal proverb (integer) as in the dipromit.tsv file;
• context/actors: an introduction to the context and actors involved in the scene;
• situation/action: a description of an action involving one or more actors;
• consequence/result: a consequence or result of the action involving one or more actors of the scene;
• hint: a standardized sentence focusing on one actor of the scene to trigger a proverb.

Notes: ID 679 refers to the Sicilian proverb "La Carni di l'omu si mancia cu li meli." from the dipromit.tsv data file. Actors are within [] and geographic information within {} to facilitate preprocessing.

Code for the experiments

Setup

Clone this repository on your own path:

git clone https://github.com/dhfbk/dialectal-proverbs.git

Create an environment with your own preferred package manager. We used python 3.9 and dependencies listed in requirements.txt. If you use venv, you can just run the following commands from the root of the project:

python -m venv venv                        # create the environment
source venv/bin/activate                   # activate the environment
pip install -r requirements.txt            # install the required packages

Get the data files for building the DiPRomIt-scenarios benchmark (see DiPRomIt dataset and benchmark), then put the dipromit.tsv and scenarios.tsv files into the data/raw/ folder. Then run the following commands to create the DiPRomIt-scenarios benchmark in its with-geo and without-geo variants:

python scripts/1.create-benchmark.py \
    --proverbs_filepath data/raw/dipromit.tsv \
    --scenarios_filepath data/raw/scenarios.tsv \
    --benchmark_variant with-geo

python scripts/1.create-benchmark.py \
    --proverbs_filepath data/raw/dipromit.tsv \
    --scenarios_filepath data/raw/scenarios.tsv \
    --benchmark_variant without-geo

DiPRomIt-scenarios benchmark data files will be crated in the data/ folder: benchmark-with-geo.json and benchmark-without-geo.json, respectively.

In these json files, each scenario together with its options is provided in the following format:

{
    "147_679": {
        "id": "147_679",
        "scenario_id": "147",
        "proverb_id": "679",
        "scenario_question": "Una bambina è con il papà e la nonna al mercato in Sicilia. La bambina nota della cioccolata in un banco del mercato ed insiste per ottenerla, dopodiché dice al papà: \"Sei il papà più bravo del mondo!\" Il papà si fa lusingare dalla bambina e le compra il cioccolato tanto agognato. Che proverbio userebbe la nonna in questa situazione?",
        "proverb_choices": [
            {
                "id": "A",
                "text": "Bona parola è menzu pagamentu.",
                "translation": "Buona parola è mezzo pagamento.",
                "label": "Wrong (same geographical area)"
            },
            ...
        ],
        "proverb_region": "Sicilia",
        "proverb_province": "",
        "proverb_municipality": "",
        "answer_gold": "B"
    },
    ...
}

where:

• 147_679: the identifier of the contextual scenario, serving as a key;
  • id: the identifier of the contextual scenario as above: the first part is the ID of the scenario itself, whereas the second part is the ID of the proverb it refers to. These two integers are separated by _);
  • scenario_id: the identifier of the contextual scenario (integer);
  • proverb_id: the identifier of the dialectal proverb (integer);
  • scenario_question: the text of the contextual scenario ending as a question (in Italian);
  • proverb_choices: a list of four choices; one Correct, one Wrong (same geographical area), one Wrong (close geographical area), and one Wrong (distant geographical area), as indicated below;
    • id: the identifier of the choice (i.e., A, B, C, or D);
    • text: the text of the choice - i.e., the dialectal proverb;
    • translation: the literal translation of the choice - i.e., translation of the dialectal proverb (in Italian);
    • label: the answer/label for the choice (i.e., Correct, Wrong (same geographical area), Wrong (close geographical area), or Wrong (distant geographical area));
  • proverb_region: the administrative region of origin of the dialectal proverb;
  • proverb_province: the province or historical area of origin of the dialectal proverb, if available;
  • proverb_municipality: the municipality of origin of the dialectal proverb, if available;
  • answer_gold: the correct answer among the four options (i.e., A, B, C, or D).

Note that this refers to the with-geo version of the benchmark. The without-geo version has no "in Sicilia" in the scenario_question text field.

Finally, set the HF_TOKEN variable to your own Hugging Face token in the scripts/2.predict.py file.

Prediction and evaluation

For predicting and evaluating the performance of different LLMs under different conditions on the DiPRomIt-scenarios benchmark, run the following:

sh pred-and-eval.sh

Note that by default the script runs all the 10 LLMs tested in our experiments, all with 5 different seeds and with 3 prompt types, in both the with-geo and without-geo versions of the DiPRomIt-scenarios benchmark. To restrict the run to only some models, prompts, seeds, or a benchmark version, open pred-and-eval.sh and set the variables at lines 2-5 before running it. Specifically:

• BENCHMARKS: possible values are with-geo and without-geo;
• PROMPTS: possible values are type-1, type-2, and type-3;
• MODELS: possible values are llama-3.1-8B, qwen-2.5-7B, mistral-0.3-7B, gemma-2-9B, llamantino-3-anita-8B, minerva-1.0-7B, llama-3.1-70B, qwen-2.5-72B, mixtral-0.1-8x7B, and gemma-2-27B;
• SEEDS: possible values are 0, 42, 101, 399, and 565.

The script automatically runs the prediction script (scripts/2.predict.py) for the selected benchmark variants, prompt types, models, and seeds, and compute scores for each combination of them (scripts/3.evaluate.py).

Results are created in the results/ folder under files named $BENCHMARK_$MODEL_$SEED_$PROMPT, where the variables are the specific choices mentioned above:

• $BENCHMARK_$MODEL_$SEED_$PROMPT.json: a file similar to the input file benchmark-with-geo.json or benchmark-without-geo.json, with an additional answer_pred field storing the output of the configuration tested. The answer_pred can take an A, B, C, or D value;
• $BENCHMARK_$MODEL_$SEED_$PROMPT.scores.json: a file with the scores obtained by the model in a json format (fields for metrics are: accuracy, correct, wrong_same_area, wrong_close_area, wrong_distant_area, and wrong_other);
• $BENCHMARK_$MODEL_$SEED_$PROMPT.log: the log storing the outputs for the configuration tested.

Further information

If you need further information, do not hesitate to get in touch with us by writing an email.

Citation

If you use or build on top of this work, please cite our paper as follows:

@inproceedings{errico-etal-2026-can,
  title = "Can LLMs Match Folk Wisdom? A Contextual Evaluation of Proverb Selection in Romance Dialects of Italy",
  author = "Errico, Gabriele  and
    Ramponi, Alan  and
    Tonelli, Sara  and
    Jezek, Elisabetta",
  booktitle = "Proceedings of the 12th Italian Conference on Computational Linguistics (CLiC-it 2026)",
  month = sep,
  year = "2026",
  address = "Palermo, Italy",
  publisher = "CEUR Workshop Proceedings"
}