Production-Grade LLM Prompt & Output Regression Tester CLI
Catch silent regressions, format breakages, latency spikes, and token inflation before pushing prompts to production.
Block regressions before merging code. When prompt engineers or backend developers tweak a system prompt,
promptdiffenforces hard performance & accuracy boundaries in your CI/CD pipeline.
promptdiff test prompts/v1.txt prompts/v2.txt \
--inputs datasets/testcases.jsonl \
--model gpt-4o \
--assert "cost_delta <= 10%, latency_delta <= 15%, json_validity == 1.0"| Exit Code | Condition | CI/CD Action |
|---|---|---|
0 |
All assertions satisfied across test suite | ✅ PR Checks Pass — Safe to merge |
1 |
Cost spike, latency regression, or invalid JSON detected | ❌ PR Checks Blocked — Regression prevented |
When engineering LLM prompts, even small tweaks—such as changing a system rule, adjusting formatting requirements, or rewriting examples—can cause silent production regressions:
- ❌ Format Regressions: The model stops emitting valid JSON or omits mandatory schema fields.
- 💸 Cost Spikes: Output verbosity inflates token counts by 40%, drastically increasing API bills.
- ⏱️ Latency Degradation: Unintended chain-of-thought increases time-to-first-token and overall p95 latency.
- 📉 Output Drift: Key domain information or brand voice is dropped.
Testing prompts manually in web playgrounds is slow, unrepeatable, and disconnected from software engineering workflows.
- Side-by-Side Visual Diffing: Terminal-native 2-column view (like
git diff) highlighting exact word, line, and JSON key modifications. - Multi-Dimensional Metrics: Automated evaluation of
json_validity,latency_delta,token_cost,similarity, andregex_match. - CI/CD Hard Assertions: Enforce
--assertrules with non-zero exit codes for GitHub Actions / GitLab CI. - Deterministic Disk Caching: SQLite SHA-256 caching for $0 re-runs and instant iteration.
- Multi-Provider & Zero-Key Mock Mode: Supports OpenAI, Anthropic Claude, Google Gemini, Ollama, and an offline deterministic
MockProvider. - Multi-Format Export: Generates standalone interactive HTML reports, GitHub PR comment Markdown, and JSON.
promptdiff CLI
│
┌──────────────────────┴──────────────────────┐
▼ ▼
Prompt Version 1 Prompt Version 2
(Baseline Template) (Candidate Template)
│ │
└──────────────┬──────────────────────────────┘
▼
Dataset / Test Cases Loader
(.jsonl, .yaml, .csv, .json)
│
▼
Async Batch Execution Engine
(Semaphore Concurrency + Cache)
│
┌───────────────┼───────────────┐
▼ ▼ ▼
OpenAI/Claude Gemini/Ollama MockProvider
│
▼
Evaluation Registry
┌───────────────────────┼───────────────────────┐
▼ ▼ ▼
JSON Schema & Latency & Cost Text & Semantic
Validity Checker Delta Tracker Similarity Engine
│
▼
CI/CD Assertion Engine
(Threshold Pass/Fail Rules)
│
┌──────────────────────┼──────────────────────┐
▼ ▼ ▼
Terminal UI HTML Report Markdown Summary
(Side-by-Side Diff) (Interactive Dark UI) (GitHub Actions / PR)
Install directly from GitHub or source:
# Direct install from GitHub
pip install git+https://github.com/latryee/promptdiff.git
# Or clone and install editable with dev dependencies
git clone https://github.com/latryee/promptdiff.git
cd promptdiff
pip install -e ".[dev]"You don't need any API keys to try promptdiff. Run our realistic offline mock engine:
promptdiff test examples/prompts/support_bot_v1.txt examples/prompts/support_bot_v2.txt \
--inputs examples/testcases.jsonl \
--eval "json_validity,latency,cost,similarity" \
--mock \
--export-html report.htmlpromptdiff test prompts/v1.txt prompts/v2.txt \
--model gpt-4o \
--eval "json_validity,latency,cost,similarity"promptdiff test prompts/v1.txt prompts/v2.txt \
--inputs datasets/eval_cases.jsonl \
--model claude-3-5-sonnet-latest \
--concurrency 8Check token costs per 1 Million tokens across 40+ models:
promptdiff pricing
# Or filter specific models:
promptdiff pricing geminiDiff two prompt files without invoking models:
promptdiff diff prompts/v1.txt prompts/v2.txtpromptdiff init my-prompt-suitepromptdiff cache stats
promptdiff cache clear| Metric Name | Evaluator Purpose | Output Range / Details |
|---|---|---|
json_validity |
Validates JSON syntax and schema compliance | 1.0 (Valid), 0.0 (Invalid), 0.5 (Schema Mismatch) |
latency |
Measures execution latency delta | Milliseconds delta (-35.4ms (-15.2%)) |
cost |
Computes token dollar cost from pricing tables | USD delta ($0.0012 -> $0.0009 (-25%)) |
similarity |
Measures sequence & token overlap preservation | 0.0 to 1.0 (100% Identical) |
regex_match |
Enforces output regex structure & mandatory keywords | 1.0 (Matched), 0.0 (Failed) |
length_drift |
Tracks output token & character inflation | Delta tokens and percentage drift |
| Provider | Model Identifier Examples | Environment Variable |
|---|---|---|
| OpenAI | gpt-4o, gpt-4o-mini, o1-preview, o3-mini |
OPENAI_API_KEY |
| Anthropic | claude-3-5-sonnet-latest, claude-3-5-haiku-latest, claude-3-opus-latest |
ANTHROPIC_API_KEY |
| Google Gemini | gemini-2.0-flash, gemini-1.5-pro, gemini-1.5-flash |
GEMINI_API_KEY |
| Ollama (Local) | ollama/llama3, ollama/mistral, ollama/deepseek-r1 |
OLLAMA_HOST (Optional) |
| OpenRouter / DeepSeek | deepseek-chat, deepseek-reasoner |
OPENAI_BASE_URL, OPENAI_API_KEY |
| Mock (Offline) | mock, --mock |
None (Zero API keys required) |
Add promptdiff to your .github/workflows/prompt-test.yml to automatically prevent prompt regressions on pull requests:
name: Prompt Regression CI
on:
pull_request:
paths:
- 'prompts/**'
- 'datasets/**'
jobs:
prompt-regression:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install promptdiff
run: pip install git+https://github.com/latryee/promptdiff.git
- name: Run promptdiff Regression Suite
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
promptdiff test prompts/system_v1.txt prompts/system_v2.txt \
--inputs datasets/testcases.jsonl \
--model gpt-4o \
--assert "cost_delta <= 10%, latency_delta <= 20%, json_validity == 1.0" \
--export-markdown comment.md \
--export-html report.html
- name: Comment PR Summary
if: always()
uses: thollander/actions-comment-pull-request@v2
with:
filePath: comment.mdRun unit tests, integration tests, and check test coverage:
# Run pytest with code coverage
pytest --cov=promptdiff --cov-report=term-missing
# Run linter & type checker
ruff check .
mypy promptdiffDistributed under the MIT License. See LICENSE for more information.