Quarterly snapshot due: 2026-07

[github-actions]

RefusalBench 2026-07 snapshot checklist

A new quarterly snapshot is due. Complete the steps below, then close this issue.

Pre-run

• Review config/sweep_models.json — remove deprecated models, add newly released ones
• Update config/model_lineage.json — add new members to existing lineages; create new lineage entries for new providers
• Check council/v1.0.json — confirm judge model IDs are still valid on Bedrock
• Verify any verify_id: true entries against the AWS Bedrock console and OpenRouter model list
• Confirm AWS_SECRET_ACCESS_KEY (Bedrock) and OPENROUTER_API_KEY are set in .env

Run

python scripts/run_sweep_all.py --snapshot 2026-07 --dry-run   # confirm plan
python scripts/run_sweep_all.py --snapshot 2026-07              # run

• Sweep completes — snapshots/2026-07/eval/ populated, manifest.json has completed_at
• Council categorization run: python scripts/run_council.py --snapshot 2026-07
• snapshots/2026-07/council/adjudicated.csv present

Analysis

• Regenerate figures: python -m refusalbench.analysis.figures --snapshot 2026-07
• Run longitudinal comparison against previous snapshot (if ≥2 snapshots exist): python -c "from refusalbench.analysis.longitudinal import compare_snapshots; ..."
• Write snapshots/2026-07/summary.md — headline stats, notable changes from prior snapshot
• Commit snapshot directory (excluding raw eval CSVs if large — use git-lfs)

Release

• Tag release: git tag snapshot-2026-07 && git push --tags
• Post summary to relevant channels / update project page

---

Opened automatically by the quarterly snapshot workflow.