MedF2Route (EMNLP 2026 Main Conference)

Official code for MedF2 Route: Evidence-Routed Graph Verification for Hallucination Detection in Clinical LVLMs.

MedF2Route detects faithfulness and factuality hallucinations in clinical vision-language model responses. It builds clinical graphs from the query and response, routes each response triple to the relevant evidence source, and verifies it against patient text, medical images, or a medical knowledge graph.

Setup

Python 3.10 or later is recommended.

Windows PowerShell:

python -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Linux or macOS:

python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

API configuration

Use an OpenAI-compatible chat-completions endpoint and a model that accepts both text and image inputs.

Windows PowerShell:

$env:OPENAI_API_KEY="your_key"
$env:OPENAI_BASE_URL="https://api.openai.com/v1"
$env:OPENAI_MODEL="your_model"

Linux or macOS:

export OPENAI_API_KEY="your_key"
export OPENAI_BASE_URL="https://api.openai.com/v1"
export OPENAI_MODEL="your_model"

Data

MedF2 Bench is a multimodal benchmark for sentence-level hallucination detection in clinical LVLM responses. It contains 500 diagnosis-oriented clinical cases, 585 medical images, and 2,383 annotated response sentences across CT, EKG, MRI, ultrasound, and X-ray. Each case provides a clinical query, an LVLM response, image references, and sentence-level hallucination annotations.

Download MedF2 Bench from Google Drive, then place the code and data directories under the same parent directory:

workspace/
  MedF2Route_Code/
  MedF2Route_500Case_Data/
    cases/
    images/
    metadata/

Knowledge graph

MedF2Route uses LMKG for knowledge-grounded verification. The original LMKG data are available from the official LMKG Google Drive.

The inference code uses an English subset of LMKG together with a FAISS index:

1. Download LMKG_released.json from the official LMKG source.
2. Retain the English entities, aliases, and their relations.
3. Encode the aliases with BioLORD-2023 and build a FAISS inner-product index.

Place the processed files in a directory such as MedF2Route_KG:

MedF2Route_KG/
  aliases_en_highconf.jsonl
  entities_meta_highconf.jsonl
  relations.jsonl
  output_biolord/
    alias_texts.jsonl
  indexes_biolord/
    faiss_index_ip.bin

Pass the prepared directory to run.py with --kg-dir. On first use, sentence-transformers downloads BioLORD-2023 automatically.

Inference

Place MedF2Route_Code, MedF2Route_500Case_Data, and MedF2Route_KG in the same parent directory. Enter the code directory and run:

cd MedF2Route_Code
python run.py --case ../MedF2Route_500Case_Data/cases/ct/ct_128_PMC3015425_01.json --images-root ../MedF2Route_500Case_Data/images --kg-dir ../MedF2Route_KG --output prediction.json

The output includes the query and response graphs, evidence routing results, branch-level verification, the retrieved reference subgraph, and sentence-level predictions.