Ground the default system prompt: abstain instead of guess on retrieval miss#3
Open
idanshimon wants to merge 1 commit into
Open
Ground the default system prompt: abstain instead of guess on retrieval miss#3idanshimon wants to merge 1 commit into
idanshimon wants to merge 1 commit into
Conversation
…al miss)
The default system_prompt ("You are a helpful assistant with access to a
knowledge base.") does not tell the model to stay grounded in the retrieved
knowledge. When retrieval misses (paraphrase gap, or an absent field), smaller
models substitute a well-known training-data fact instead of abstaining -- e.g.
answering "90 minutes" to a door-to-balloon question whose answer was never in
the corpus and was never retrieved.
Evidence (8 models x trap questions, GitHub Copilot CLI, mechanical grading):
- bare default prompt: 5/72 hallucinations (~7%), ALL from mini models,
ALL training-data substitution on the one
paraphrase-miss question.
- grounded prompt: 0/144 hallucinations.
- re-running the 18 failing mini-model trials with this patched default: 0/18.
Fix: append a grounding + abstention clause to the two default prompts
(layer.py constructor default, infinite_context fallback). Fully backward
compatible -- passing a custom system_prompt is unchanged. Adds 2 tests
(default-is-grounded, custom-still-overrides); suite 129 -> 131, all green.
There was a problem hiding this comment.
Pull request overview
This PR updates ContextForge’s default system prompt(s) to explicitly ground answers in retrieved knowledge and instruct the model to abstain (instead of guessing) when the knowledge provided does not contain the answer, addressing hallucinations on retrieval misses.
Changes:
- Strengthen the default
system_promptinContextForgeto require knowledge-only answers and abstention on missing info. - Strengthen the
InfiniteContextfallback system prompt when no system parts are present. - Add tests ensuring the default prompt is grounded and that a custom
system_promptstill overrides the default.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/test_layer.py | Adds tests validating grounded default prompting and confirming custom prompt override behavior. |
| contextforge/layer.py | Updates ContextForge constructor’s default system_prompt to include grounding + abstention instructions. |
| contextforge/infinite_context.py | Updates the empty-system_parts fallback system prompt to include grounding + abstention instructions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+195
to
+198
| lowered = system.lower() | ||
| assert "only" in lowered | ||
| # must steer toward abstaining rather than guessing / using outside knowledge | ||
| assert "guess" in lowered or "do not have" in lowered |
Comment on lines
+52
to
+57
| system_prompt: str = ( | ||
| "You are a helpful assistant with access to a knowledge base. " | ||
| "Answer using ONLY the information in the provided knowledge. " | ||
| "If the answer is not contained in the knowledge, say you do not have " | ||
| "that information rather than guessing or using outside knowledge." | ||
| ), |
Comment on lines
298
to
305
| messages.append({ | ||
| "role": "system", | ||
| "content": "\n\n".join(system_parts) if system_parts else "You are a helpful assistant.", | ||
| "content": "\n\n".join(system_parts) if system_parts else ( | ||
| "You are a helpful assistant. Answer using ONLY the information " | ||
| "provided to you. If the answer is not available, say you do not " | ||
| "have that information rather than guessing." | ||
| ), | ||
| }) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The default
system_prompt—— doesn't instruct the model to stay grounded in the retrieved knowledge. When retrieval misses (a paraphrase gap, or a genuinely absent field), smaller models substitute a well-known training-data fact instead of abstaining.
Concrete example from testing: ask "longest acceptable time from patient arrival to opening the blocked artery?" against a clinical-registry corpus. The corpus never states a maximum, and BM25 never retrieves the door-to-balloon doc (different words than the query). With the bare default,
gpt-5-miniconfidently answers "Goal: door-to-balloon ≤ 90 minutes... ≤ 120 minutes" — pulled from training data, not the corpus. That's a hallucination by training-data substitution.Evidence
8 models × trap questions via GitHub Copilot CLI, mechanical grading (absence + paraphrase-miss cells):
Same model / same corpus / same retrieval miss,
gpt-5-mini:Re-running the 18 mini-model trials that failed under the old default, with this patched default: 0 / 18.
The absence questions (e.g. "max admission potassium?") stayed at 0 either way — there's no registry-potassium-max in any model's training data to substitute. The failure specifically needs a retrieval miss + a famous answer the model already knows, which is exactly what the grounding clause defends against.
Change
Append a grounding + abstention clause to the two default prompts:
contextforge/layer.py— constructor defaultsystem_promptcontextforge/infinite_context.py— the empty-system_partsfallbackFully backward compatible — passing your own
system_promptis unchanged. Adds 2 tests (TestGroundedDefaultPrompt: default-is-grounded + custom-still-overrides). Suite 129 → 131, all green.Why default-on
Anyone who never thinks about the system prompt — the quickstart path, the copy-paste-the-README user — currently gets the permissive behavior and the silent substitution. Making grounding the default means the safe behavior is free, and opting out is explicit.