Modified mt_bench with API and HF scripts for LFMs.
-
Updated
Jul 9, 2025 - Python
#
mt-bench
Here are 8 public repositories matching this topic...
Do LLM judges know when they're wrong? Three open-weight judges (Qwen2.5-7B, kev-8b, auto-j-13b) graded against MT-Bench human votes: calibration, position and padding attacks, and what auto-accepting confident verdicts would cost. Interactive site included.
python machine-learning statistics evaluation data-visualization calibration uncertainty-quantification bayesian-statistics interactive-visualizations numpyro selective-prediction llm vllm llm-eval qwen llm-evaluation llm-as-judge llm-as-a-judge mt-bench position-bias
-
Updated
Sep 25, 2026 - Python
Production-grade LLM-as-Judge evaluation framework with position, verbosity & self-enhancement bias mitigation. FastAPI + Streamlit + Python SDK.
python openai mlops fastapi bias-mitigation streamlit ai-evaluation llm prompt-engineering anthropic llm-evaluation llm-as-judge g-eval mt-bench
-
Updated
Jul 3, 2026 - Python
⚖️ Dual-Judge: 让AI测试结果真正有说服力 | 双LLM交叉验证消除单模型偏见 | 独立于具体Agent的通用评估框架 | Making AI Evaluation Trustworthy
python hotel quality-assurance quality-metrics evaluation-framework hotel-service fastapi ai-agent ai-testing ollama llm-evaluation deepseek react-agent langgraph llm-as-judge mt-bench llm-judge inter-rater-reliability
-
Updated
Jun 18, 2026 - Python
Reference Flutter chatbot + benchmark harness comparing on-device (Gemma) vs cloud (Claude) LLMs — MSc thesis artifact
-
Updated
Jul 16, 2026 - Dart
Korean MT-Bench 문항, 모델 응답, LLM-as-a-Judge 판정 기록 및 논문 결과 재현 코드
benchmark reproducible-research korean-nlp large-language-models llm-evaluation llm-as-a-judge mt-bench korean-mt-bench
-
Updated
Aug 25, 2026 - Python
Reproducible LLM-as-a-judge reliability lab: chance-corrected agreement (Cohen's kappa, Krippendorff's alpha) with bootstrap CIs, computed keyless from a committed MT-Bench snapshot and re-derived in CI as a drift gate.
python benchmark statistics evaluation reproducibility llm llm-evaluation llm-as-a-judge mt-bench inter-rater-reliability
-
Updated
Aug 19, 2026 - Python
Score an LLM judge against human labels: chance-corrected agreement with cluster-bootstrap intervals, the human-human ceiling it has to be read against, and a written drift protocol. Stdlib only, no API key.
bootstrap measurement evaluation-metrics cohens-kappa llm-evaluation llm-as-a-judge mt-bench inter-rater-reliability
-
Updated
Aug 23, 2026 - Python
Add this topic to your repo
To associate your repository with the mt-bench topic, visit your repo's landing page and select "manage topics."