[guide] Prompt versioning guide

[lucifertrj]

What & why

This PR adds the guide on: Creating and using prompt versions #43

Version prompts in the Opik Prompt Library, compare versions for hallucination before you
ship one, then run inference against whichever version is currently the latest commit
without hardcoding the prompt text into your application.

Checklist

• Linked to its tracking issue (commented to claim it before starting)
• Example is in the right bucket (integrations / guides / use-cases / scripts)
• Folder name is lowercase_with_underscores
• README.md has all required sections; index tables updated if examples were added/renamed/removed
• Dry-run works with no credentials — bash run.sh exits cleanly (this is what CI's secrets-free job runs)
• uv run ruff check . and uv run ruff format --check . are clean
• No credentials or .env files committed
• Dependencies declared in pyproject.toml (uv project); no requirements.txt, no committed uv.lock
• run.sh exists and starts with set -e
• OPIK_PROJECT_NAME is set — exported in run.sh (scripts) or defined in config.py (use-cases/guides)
• Examples that call LLMs use litellm and read OPIK_EXAMPLES_MODEL

[LeoRoccoBreedt]

Thanks @lucifertrj — this is a solid start and it runs clean in dry-run (I checked out the branch: bash run.sh exits 0 with no creds, and ruff check/format are clean). The Opik API usage is all correct against opik 2.0.74.

Before we merge, we'd like to take this in a slightly different direction that we think teaches prompt versioning better and fits the guides/ bucket more naturally. Would you be up for converting it? Happy to help along the way.

Why a notebook

3 of the 4 existing guides in this repo are notebooks (annotation_queues_with_context, multimodal_online_evaluation, tracing_finetuned_models). More importantly, prompt versioning is a "watch it happen" topic — commit v1 → see a hash, commit v2 → see a new hash, fetch v1 by its old hash (still there), then compare v1 vs v2 as experiments you open side-by-side in the Opik UI. That reveal lands far better cell-by-cell than split across scripts, and it puts the experiment comparison — the real payoff — front and center.

The one conceptual change that matters most

Right now the guide uses two prompts: fintechassistv1 (versioned + run) and summarizerfintechv1 (evaluated). So the "compare versions for hallucination, then ship one" story never connects — you evaluate one prompt but ship a different, un-compared one. We'd like to carry one prompt through the whole loop.

Also: the prompt name shouldn't contain v1. The whole lesson is one name, many commits — a name like fintechassistv1 that then gets a v2 committed into it actively works against that mental model. Use earnings-call-summarizer.

Your existing SUMMARIZER_V1 / SUMMARIZER_V2 and the sample transcript are perfect for this — the summarizer is the one prompt where hallucination scoring is meaningful (summary vs. source transcript) and "run the latest on a new transcript" is a natural capstone. The fintech-advisor chatbot can be dropped from the guide (see the note at the bottom — it has a better home).

Target cell-by-cell flow

1. Markdown intro — the mental model: one name → many immutable commits; compare → promote → run latest.
2. Install — %pip install --quiet --upgrade opik litellm.
3. Setup — env vars, OPIK_PROJECT_NAME = "prompt-versioning", client = opik.Opik(), model read from OPIK_EXAMPLES_MODEL.
4. Commit v1 — client.create_prompt(name="earnings-call-summarizer", prompt=V1, change_description="loose baseline"); print the commit hash.
5. Commit v2 — same name, change_description="strict, facts-only"; print the new hash. Markdown: nothing was overwritten.
6. Prove history is retained — client.get_prompt(name="earnings-call-summarizer", commit=<v1_hash>) still returns v1. (This demonstrates immutability — currently get_version is defined but never called.)
7. Compare as experiments — build a one-item dataset (transcript + context), run evaluate_prompt twice with Hallucination as the scorer, one experiment per version. Markdown points readers to open both experiments in the UI; v2 should score lower.
8. Run the latest, no hardcoding — client.get_prompt(name="earnings-call-summarizer") (no commit) resolves to v2 → one live litellm call on a new transcript, wrapped in @opik.track(project_name=...). Markdown: promoting a v3 later changes what this runs with zero code changes.
9. Wrap-up markdown — recap + links to Prompt Library / Experiments docs.

Concrete fixes to fold in

• Dependencies are backwards. inference.py imports litellm, but pyproject.toml declares openai (never imported) and not litellm — it only works because opik pulls litellm in transitively. In the notebook's pyproject.toml, depend on opik + litellm; drop openai.
• README/PR say "OpenAI SDK" but the code uses litellm. Update the prose to litellm (litellm is the repo convention). Model comes from OPIK_EXAMPLES_MODEL.
• Index tables were not updated. Add a prompt_versioning row to both guides/README.md and the root README.md guides table.
• Prefer change_description=... over metadata={"tag": ...} for labeling each version — it renders as the version's description in the Opik UI, which is more on-message for a versioning guide.

Notebook conventions to follow (lighter contract than scripts)

• Ship the notebook named after the folder — prompt_versioning.ipynb (matches the other three guides) — plus pyproject.toml + README.md. No run.sh, no config.py, no dry-run — notebooks require credentials and teach by logging real traces.
• Commit with outputs cleared (cleaner diffs, no leaked run details).
• Set OPIK_PROJECT_NAME in a cell and pass it via @opik.track(project_name=...).
• README sections: What this does / Prerequisites / Running it / How it works.

Heads-up on CI: notebook execution (test-notebooks.yml) only runs on same-repo PRs, so a fork PR won't get an automated end-to-end run — a maintainer will run it live after review.

Future direction — a fintech-assistant use-case (separate follow-up)

One more thought: the fintech-advisor prompt you wrote (the compliance-reviewed FINTECH_ASSISTANT_V2) is too good to drop. It doesn't fit this guide — an open-ended advisor has no source context for the Hallucination metric, which is exactly why it got split off — but it would make an excellent standalone use-case: a compliance-governed fintech assistant where the Prompt Library is the governance backbone, and your compliance rules ("never name specific securities", "always include the disclaimer", "never guarantee returns") become plain-English test-suite assertions that run_tests measures the assistant against. That's a governance angle none of the current use-cases show, and it builds directly on your work.

We've opened #62 to track it — we'd love for you to take it if you're interested (you'd get first claim as the originator). Keeping it separate from this PR lets the guide land cleanly first.

If you'd rather we take it from here and land it with a Co-authored-by credit to you, just say the word. Either way, thanks for kicking this off.

[lucifertrj]

Understood. I can pick up and resolve the Concrete fixes to fold in section within 1 to 2 weeks.

I'm also fine with the notebook being Co-authored-by. If no one picks it up by the end of the month, I'll do it myself.

[lucifertrj]

@LeoRoccoBreedt (Why a notebook feedback): I have added the notebook i.e., the guide now carries a single prompt (earnings-call-summarizer) through the whole loop: commit v1/v2 → fetch by hash → side-by-side hallucination eval → traced inference on a new transcript.

I have also resolved Concrete fixes to fold in feedback: i.e., fixed pyproject.toml deps (opik + litellm, no unused openai), updated README prose from "OpenAI SDK" to litellm, switched labeling to change_description=..., and added the guide to both README index tables. Notebook outputs are cleared for a clean diff.