Create About Golden QnA.md#623
Conversation
Added a documentation about what is Golden QnA, its purpose how to develop a Golden QnA and points to note while creating a Golden QnA.
|
Warning Review limit reached
More reviews will be available in 49 minutes and 10 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughA new FAQ document is added to the documentation repository, defining Golden QnA Sets as curated datasets used for evaluation consistency. The document covers concept definitions, dataset development methodology with categorized examples, guidelines for creating accurate answers, and a validation checklist. ChangesGolden QnA Set Documentation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
docs/8. FAQ/About Golden QnA.md (1)
73-73: 💤 Low valueConsider more descriptive link text.
The link uses "here" as anchor text. More descriptive text like "in the development guidelines" would improve accessibility and clarity.
♻️ Proposed refactor
-- Are all categories mentioned [here](`#how-to-develop-golden-qnas`) covered adequately? +- Are all categories mentioned [in the development guidelines](`#how-to-develop-golden-qnas`) covered adequately?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/8`. FAQ/About Golden QnA.md at line 73, Replace the generic anchor text "here" in the sentence "Are all categories mentioned [here](`#how-to-develop-golden-qnas`) covered adequately?" with a more descriptive phrase such as "in the development guidelines" (or similar) so the link reads "Are all categories mentioned in the development guidelines covered adequately?" to improve accessibility and clarity; update the markdown link target [`#how-to-develop-golden-qnas`] while keeping the same fragment identifier.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/8`. FAQ/About Golden QnA.md:
- Around line 61-74: Add a new section titled "Formatting Your Golden QnA
Dataset" immediately after the "Final Review Checklist" section that states the
dataset must be a CSV with each row as a question,answer pair (columns:
question, answer), includes the Golden QA CSV template link
(https://docs.google.com/spreadsheets/d/198UpOMeU53s9O-fwbIl0DIJLuD3l24jgkq74CoDfSQM/copy),
and explains the duplication factor (integer 1–5 indicating how many times
questions are repeated during evaluation); also add a short cross-reference
sentence pointing to the "AI Evaluations in Glific.md" for detailed usage
instructions so readers can both format and implement Golden QnAs in Glific.
- Line 53: Fix the typo in the table header: replace the string "<b>What to
avaoid</b>" with "<b>What to avoid</b>" in the header row that currently reads
"<b>To include</b> | <b>What to avaoid</b> | <b>Why it matters?</b>" so the
header correctly shows "What to avoid".
- Line 45: Fix the spelling in the table row that begins with "Practical
Situations" by changing the example question text from "3 monhs ki pregnancy" to
"3 months ki pregnancy"; locate the cell containing "Practical Situations |
Tests whether the chatbot..." and update the second example sentence to the
corrected spelling.
- Line 55: The table row in "About Golden QnA.md" contains an unclosed quotation
after the phrase "It depends"; update that cell by adding the missing closing
quotation mark immediately after It depends (i.e., change `“It depends` to `“It
depends”`) so the grammar and punctuation are correct for the Answers column.
---
Nitpick comments:
In `@docs/8`. FAQ/About Golden QnA.md:
- Line 73: Replace the generic anchor text "here" in the sentence "Are all
categories mentioned [here](`#how-to-develop-golden-qnas`) covered adequately?"
with a more descriptive phrase such as "in the development guidelines" (or
similar) so the link reads "Are all categories mentioned in the development
guidelines covered adequately?" to improve accessibility and clarity; update the
markdown link target [`#how-to-develop-golden-qnas`] while keeping the same
fragment identifier.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 4b4be332-3f05-4683-9147-0d27720a0c8c
📒 Files selected for processing (1)
docs/8. FAQ/About Golden QnA.md
| | <b>Category</b> | <b>Purpose</b> | <b>Example Questions</b> | | ||
| |----------|----------|----------| | ||
| | Important Information (covering the most frequently asked themes) | Tests important factual information the chatbot should know. These should form the majority of the dataset. | 1. हजार दिवस क्या है ? <br> 2. Pregnancy mein aneamia ke kya lakshan hote hai?| | ||
| | Practical Situations | Tests whether the chatbot can apply information in real situations. | 1. C-section ke baad taake lage hai toh uska dekhbhaal kaise karien?<br> 2. 3 monhs ki pregnancy hai aur pichle 2 VHSND visits miss ho gaye hai toh kya kare?| |
There was a problem hiding this comment.
Fix spelling error in example question.
Line 45 contains "3 monhs ki pregnancy" which should be "3 months ki pregnancy".
✏️ Proposed fix
-| Practical Situations | Tests whether the chatbot can apply information in real situations. | 1. C-section ke baad taake lage hai toh uska dekhbhaal kaise karien?<br> 2. 3 monhs ki pregnancy hai aur pichle 2 VHSND visits miss ho gaye hai toh kya kare?|
+| Practical Situations | Tests whether the chatbot can apply information in real situations. | 1. C-section ke baad taake lage hai toh uska dekhbhaal kaise karien?<br> 2. 3 months ki pregnancy hai aur pichle 2 VHSND visits miss ho gaye hai toh kya kare?|📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | Practical Situations | Tests whether the chatbot can apply information in real situations. | 1. C-section ke baad taake lage hai toh uska dekhbhaal kaise karien?<br> 2. 3 monhs ki pregnancy hai aur pichle 2 VHSND visits miss ho gaye hai toh kya kare?| | |
| | Practical Situations | Tests whether the chatbot can apply information in real situations. | 1. C-section ke baad taake lage hai toh uska dekhbhaal kaise karien?<br> 2. 3 months ki pregnancy hai aur pichle 2 VHSND visits miss ho gaye hai toh kya kare?| |
🧰 Tools
🪛 LanguageTool
[grammar] ~45-~45: Ensure spelling is correct
Context: ...h uska dekhbhaal kaise karien?
2. 3 monhs ki pregnancy hai aur pichle 2 VHSND vis...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/8`. FAQ/About Golden QnA.md at line 45, Fix the spelling in the table
row that begins with "Practical Situations" by changing the example question
text from "3 monhs ki pregnancy" to "3 months ki pregnancy"; locate the cell
containing "Practical Situations | Tests whether the chatbot..." and update the
second example sentence to the corrected spelling.
|
|
||
| # Points to Remember While Creating Golden QnAs | ||
|
|
||
| | <b>To include</b> | <b>What to avaoid</b> | <b>Why it matters?</b> | |
There was a problem hiding this comment.
Fix typo in table header.
The table header contains "What to avaoid" which should be "What to avoid".
✏️ Proposed fix
-| <b>To include</b> | <b>What to avaoid</b> | <b>Why it matters?</b> |
+| <b>To include</b> | <b>What to avoid</b> | <b>Why it matters?</b> |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | <b>To include</b> | <b>What to avaoid</b> | <b>Why it matters?</b> | | |
| | <b>To include</b> | <b>What to avoid</b> | <b>Why it matters?</b> | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/8`. FAQ/About Golden QnA.md at line 53, Fix the typo in the table
header: replace the string "<b>What to avaoid</b>" with "<b>What to avoid</b>"
in the header row that currently reads "<b>To include</b> | <b>What to
avaoid</b> | <b>Why it matters?</b>" so the header correctly shows "What to
avoid".
|
|
||
| | <b>To include</b> | <b>What to avaoid</b> | <b>Why it matters?</b> | | ||
| |----------|----------|----------| | ||
| | Write clear, grammatically correct, specific, and confident answers | Answers with typos, broken grammar, or vague phrases like “Maybe”, “Could be”, “It depends | Golden Answers are expected to represent the ideal response. Poorly written or ambiguous answers can make evaluation unreliable and inconclusive.| |
There was a problem hiding this comment.
Add missing closing quotation mark.
The text ends with an unclosed quote after "It depends" which should be closed for proper grammar.
✏️ Proposed fix
-| Write clear, grammatically correct, specific, and confident answers | Answers with typos, broken grammar, or vague phrases like "Maybe", "Could be", "It depends | Golden Answers are expected to represent the ideal response. Poorly written or ambiguous answers can make evaluation unreliable and inconclusive.|
+| Write clear, grammatically correct, specific, and confident answers | Answers with typos, broken grammar, or vague phrases like "Maybe", "Could be", "It depends" | Golden Answers are expected to represent the ideal response. Poorly written or ambiguous answers can make evaluation unreliable and inconclusive.|📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | Write clear, grammatically correct, specific, and confident answers | Answers with typos, broken grammar, or vague phrases like “Maybe”, “Could be”, “It depends | Golden Answers are expected to represent the ideal response. Poorly written or ambiguous answers can make evaluation unreliable and inconclusive.| | |
| | Write clear, grammatically correct, specific, and confident answers | Answers with typos, broken grammar, or vague phrases like "Maybe", "Could be", "It depends" | Golden Answers are expected to represent the ideal response. Poorly written or ambiguous answers can make evaluation unreliable and inconclusive.| |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/8`. FAQ/About Golden QnA.md at line 55, The table row in "About Golden
QnA.md" contains an unclosed quotation after the phrase "It depends"; update
that cell by adding the missing closing quotation mark immediately after It
depends (i.e., change `“It depends` to `“It depends”`) so the grammar and
punctuation are correct for the Answers column.
| # Final Review Checklist | ||
|
|
||
| Refer to the checklist below before finalizing the Golden QnA set. | ||
|
|
||
| - Is the answer factually correct? | ||
| - Is the answer grammatically correct? | ||
| - Does the question sound natural? | ||
| - Does the answer follow the prompt instructions? | ||
| - Is the answer clear and unambiguous? | ||
| - Is only one intent/category being tested? | ||
| - Is the fallback response consistent? | ||
| - Is the category correctly assigned? | ||
| - Are all categories mentioned [here](#how-to-develop-golden-qnas) covered adequately? | ||
|
|
There was a problem hiding this comment.
Add section explaining CSV format requirements and duplication factor.
This document provides excellent conceptual guidance for creating Golden QnAs, but it's missing critical technical information about how to format and use the dataset in Glific. Based on the upstream contract in AI Evaluations in Glific.md, users need to know:
- The Golden QA dataset must be a CSV file
- The CSV format:
question, answerwith one pair per row - What the duplication factor is (number of times questions are repeated during evaluation, allowed values 1-5)
- Link to the Golden QA CSV template
Without this information, users who read this FAQ won't know how to actually implement their Golden QnAs in Glific.
Suggested addition:
Consider adding a new section after the "Final Review Checklist" titled "Formatting Your Golden QnA Dataset" that covers:
- CSV file format requirements
- Column structure (question, answer)
- Link to the Golden QA CSV template
- Brief explanation of duplication factor and its allowed values (1-5)
- Cross-reference to the AI Evaluations documentation for detailed usage instructions
This bridges the gap between conceptual guidance (what makes good Golden QnAs) and practical implementation (how to format them for use in Glific).
🧰 Tools
🪛 LanguageTool
[style] ~71-~71: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...nly one intent/category being tested? - Is the fallback response consistent? - Is ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[style] ~72-~72: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... Is the fallback response consistent? - Is the category correctly assigned? - Are ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
🪛 markdownlint-cli2 (0.22.1)
[warning] 73-73: Link text should be descriptive
(MD059, descriptive-link-text)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/8`. FAQ/About Golden QnA.md around lines 61 - 74, Add a new section
titled "Formatting Your Golden QnA Dataset" immediately after the "Final Review
Checklist" section that states the dataset must be a CSV with each row as a
question,answer pair (columns: question, answer), includes the Golden QA CSV
template link
(https://docs.google.com/spreadsheets/d/198UpOMeU53s9O-fwbIl0DIJLuD3l24jgkq74CoDfSQM/copy),
and explains the duplication factor (integer 1–5 indicating how many times
questions are repeated during evaluation); also add a short cross-reference
sentence pointing to the "AI Evaluations in Glific.md" for detailed usage
instructions so readers can both format and implement Golden QnAs in Glific.
added closing `<br/>` tags
|
🚀 Deployed on https://deploy-preview-623--glific-docs.netlify.app |
2 participants
Added a documentation about what is Golden QnA, its purpose how to develop a Golden QnA and points to note while creating a Golden QnA.
Summary by CodeRabbit
Release Notes