test: expand golden-utterance e2e coverage, fix template gaps - #76
test: expand golden-utterance e2e coverage, fix template gaps#76JarbasAl wants to merge 1 commit into
Conversation
Added natural en-US phrasings for RunScriptCommandIntent and list_scripts
that real users would speak but the fixed single-line .intent templates
did not match (verified as genuine routing gaps via MiniCroft/ovoscope
before the fix, not lookup issues): "run the backup script", "run backup
script" (reversed {script}-then-noun slot), "start script backup" (missing
"start" verb synonym), "execute the backup command", "launch the weather
script", "run my backup script", "show me my scripts", "tell me the
scripts you know", "what scripts do you have".
RunScriptCommandIntent.intent now covers the "start" verb synonym,
optional "the"/"my" determiners, and both noun-then-slot and
slot-then-noun orderings. list_scripts.intent now covers "show me my
...", "what ... do you have", and "tell me the ... you know". Sibling
confusables ("start my day", "start the timer", "show me my calendar",
"what are my favorite songs", "list my reminders", "launch backup
camera") were verified to still not be claimed by the widened templates,
and per-intent RunScriptCommandIntent.blacklist/list_scripts.blacklist
locale files were added as proactive suppression for these and other
adjacent-domain phrases against future template/alias drift.
25 e2e tests (+3 xfailed) grew to 56 (+3 xfailed), all green.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Hello! I've finished running some automated checks on this PR. 👋I've aggregated the results of the automated checks for this PR below. 🌍 Locale BuildHere's the latest update on this check. 🗞️ ✅ Locale properly configured (47 files, 11 languages) Locale directories found:
Localization coverage:
pyproject.toml: ✅
Build manifest: ✅ 47 locale files included in package 🔒 Security (pip-audit)Scanning for any potential privilege escalations. 🪜 ✅ No known vulnerabilities found (73 packages scanned). 📋 Repo HealthScanning for any signs of 'deprecated' acne. 🧴 ✅ All required files present. Latest Version: ✅ 🔍 LintChecking the boxes and crossing the T's. 🖋️ ❌ ruff: issues found — see job log 🏷️ Release PreviewI've performed a final polish on the release notes. 🧼 Current:
✅ PR title follows conventional commit format. 🚀 Release Channel Compatibility Predicted next version:
⚖️ License CheckEnsuring our CLA requirements are met. 🖋️ ✅ No license violations found. Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed. 🔨 Build TestsThe assembly line is hummin' along nicely! 🎶 ✅ All versions pass
🔌 Skill Tests (ovoscope)I've checked the skill's 'privacy' (aka data handling). 🔒 ❌ 0/54 passed, 51 errors ❌ **TestListScriptsIntent** — 0/6
❌ **TestListScriptsIntentNoAlias** — 0/1
❌ **TestRunScriptCommandIntent** — 0/11
❌ **TestRunScriptCommandIntentFile** — 0/1
❌ **test_golden_utterance[can you execute script weather now]** — 0/1
❌ **test_golden_utterance[execute command backup]** — 0/1
❌ **test_golden_utterance[execute script weather]** — 0/1
❌ **test_golden_utterance[execute the backup command]** — 0/1
❌ **test_golden_utterance[launch command backup]** — 0/1
❌ **test_golden_utterance[launch script weather]** — 0/1
❌ **test_golden_utterance[launch the weather script]** — 0/1
❌ **test_golden_utterance[list my aliases]** — 0/1
❌ **test_golden_utterance[please run command backup for me]** — 0/1
❌ **test_golden_utterance[run backup script]** — 0/1
❌ **test_golden_utterance[run command backup]** — 0/1
❌ **test_golden_utterance[run command]** — 0/1
❌ **test_golden_utterance[run my backup script]** — 0/1
❌ **test_golden_utterance[run script weather]** — 0/1
❌ **test_golden_utterance[run the backup script]** — 0/1
❌ **test_golden_utterance[show me my scripts]** — 0/1
❌ **test_golden_utterance[start script backup]** — 0/1
❌ **test_golden_utterance[start the backup script]** — 0/1
❌ **test_golden_utterance[tell me the scripts you know]** — 0/1
❌ **test_golden_utterance[what commands are configured]** — 0/1
❌ **test_golden_utterance[what scripts do you have]** — 0/1
❌ **test_golden_utterance[which scripts can you run]** — 0/1
❌ **test_negative_confusable_not_claimed[execute a wolfram alpha search]** — 0/1
❌ **test_negative_confusable_not_claimed[launch a timer for 5 minutes]** — 0/1
❌ **test_negative_confusable_not_claimed[launch backup camera]** — 0/1
❌ **test_negative_confusable_not_claimed[list my reminders]** — 0/1
❌ **test_negative_confusable_not_claimed[play some music]** — 0/1
❌ **test_negative_confusable_not_claimed[run the news for me]** — 0/1
❌ **test_negative_confusable_not_claimed[search the web for cats]** — 0/1
❌ **test_negative_confusable_not_claimed[show me my calendar]** — 0/1
❌ **test_negative_confusable_not_claimed[start my day]** — 0/1
❌ **test_negative_confusable_not_claimed[start the timer]** — 0/1
❌ **test_negative_confusable_not_claimed[tell me a joke]** — 0/1
❌ **test_negative_confusable_not_claimed[what are my favorite songs]** — 0/1
❌ **test_negative_confusable_not_claimed[what's the weather]** — 0/1
🚌 Bus CoverageIs every event handler pullin' its weight? Let's check! 🏋️♂️ System.exit(0); // With love from OVOS 🖥️ |
Expands the golden-utterance end-to-end test suite for this skill and fixes gaps found in the
RunScriptCommandIntentandlist_scriptsintent templates while writing those tests, adding matching blacklists for phrases that should not trigger the intents.