Skip to content

test: expand golden-utterance e2e coverage, fix template gaps - #76

Draft
JarbasAl wants to merge 1 commit into
devfrom
qa/cmd-golden-utterances
Draft

test: expand golden-utterance e2e coverage, fix template gaps#76
JarbasAl wants to merge 1 commit into
devfrom
qa/cmd-golden-utterances

Conversation

@JarbasAl

@JarbasAl JarbasAl commented Sep 4, 2026

Copy link
Copy Markdown
Member

🤖 Auto-generated by Claude Opus 4.8 (claude-opus-4-8) via Claude Code — NOT human-reviewed. Verify before acting.

Expands the golden-utterance end-to-end test suite for this skill and fixes gaps found in the RunScriptCommandIntent and list_scripts intent templates while writing those tests, adding matching blacklists for phrases that should not trigger the intents.

Added natural en-US phrasings for RunScriptCommandIntent and list_scripts
that real users would speak but the fixed single-line .intent templates
did not match (verified as genuine routing gaps via MiniCroft/ovoscope
before the fix, not lookup issues): "run the backup script", "run backup
script" (reversed {script}-then-noun slot), "start script backup" (missing
"start" verb synonym), "execute the backup command", "launch the weather
script", "run my backup script", "show me my scripts", "tell me the
scripts you know", "what scripts do you have".

RunScriptCommandIntent.intent now covers the "start" verb synonym,
optional "the"/"my" determiners, and both noun-then-slot and
slot-then-noun orderings. list_scripts.intent now covers "show me my
...", "what ... do you have", and "tell me the ... you know". Sibling
confusables ("start my day", "start the timer", "show me my calendar",
"what are my favorite songs", "list my reminders", "launch backup
camera") were verified to still not be claimed by the widened templates,
and per-intent RunScriptCommandIntent.blacklist/list_scripts.blacklist
locale files were added as proactive suppression for these and other
adjacent-domain phrases against future template/alias drift.

25 e2e tests (+3 xfailed) grew to 56 (+3 xfailed), all green.
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Hello! I've finished running some automated checks on this PR. 👋

I've aggregated the results of the automated checks for this PR below.

🌍 Locale Build

Here's the latest update on this check. 🗞️

✅ Locale properly configured (47 files, 11 languages)

Locale directories found:

  • ovos_skill_cmd/locale

Localization coverage:

  • ovos_skill_cmd/locale: 47 files in 11 languages (ca-ES, pt-PT, de-DE, fr-FR, da-DK...)

pyproject.toml:[tool.setuptools.package-data.ovos_skill_cmd] includes locale

  • locale/**/*

Build manifest: ✅ 47 locale files included in package

🔒 Security (pip-audit)

Scanning for any potential privilege escalations. 🪜

✅ No known vulnerabilities found (73 packages scanned).

📋 Repo Health

Scanning for any signs of 'deprecated' acne. 🧴

✅ All required files present.

Latest Version: 0.4.0a1

ovos_skill_cmd/version.py — Version file
README.md — README
LICENSE.txt — License file (consider renaming to LICENSE)
pyproject.toml — pyproject.toml
⚠️ setup.py — setup.py
🗑️ MANIFEST.in — legacy: pyproject.toml is present; this file is no longer needed
CHANGELOG.md — Changelog
ovos_skill_cmd/version.py has valid version block markers

🔍 Lint

Checking the boxes and crossing the T's. 🖋️

ruff: issues found — see job log

🏷️ Release Preview

I've performed a final polish on the release notes. 🧼

Current: 0.4.0a1Next: 0.4.0a2

Signal Value
Label (none)
PR title test: expand golden-utterance e2e coverage, fix template gaps
Bump alpha

✅ PR title follows conventional commit format.


🚀 Release Channel Compatibility

Predicted next version: 0.4.0a2

Channel Status Note Current Constraint
Stable Too new (must be <0.3.0) ovos-skill-cmd>=0.2.11,<0.3.0
Testing Compatible ovos-skill-cmd>=0.2.11,<1.0.0
Alpha Compatible ovos-skill-cmd>=0.4.0a1

⚖️ License Check

Ensuring our CLA requirements are met. 🖋️

✅ No license violations found.

Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed.

🔨 Build Tests

The assembly line is hummin' along nicely! 🎶

✅ All versions pass

Python Build Install Tests
3.10
3.11
3.12
3.13
3.14

🔌 Skill Tests (ovoscope)

I've checked the skill's 'privacy' (aka data handling). 🔒

0/54 passed, 51 errors

❌ **TestListScriptsIntent** — 0/6
Test Result
test_list_my_aliases ❌ error
test_show_me_my_scripts ❌ error
test_tell_me_the_scripts_you_know ❌ error
test_what_scripts_do_you_know ❌ error
test_which_commands_can_you_run ❌ error
test_what_scripts_do_you_have ❌ error
❌ **TestListScriptsIntentNoAlias** — 0/1
Test Result
test_no_scripts_configured ❌ error
❌ **TestRunScriptCommandIntent** — 0/11
Test Result
test_execute_the_backup_command ❌ error
test_run_command_alias ❌ error
test_run_the_backup_script ❌ error
test_start_script_backup ❌ error
test_start_the_backup_script ❌ error
test_unknown_alias_does_not_run ❌ error
test_launch_command_alias ❌ error
test_run_my_backup_script ❌ error
test_execute_script_alias ❌ error
test_run_backup_script_reversed_slot ❌ error
test_launch_the_weather_script ❌ error
❌ **TestRunScriptCommandIntentFile** — 0/1
Test Result
test_run_script_backup_padatious_only ❌ error
❌ **test_golden_utterance[can you execute script weather now]** — 0/1
Test Result
test_golden_utterance[can you execute script weather now] ❌ xfailed
❌ **test_golden_utterance[execute command backup]** — 0/1
Test Result
test_golden_utterance[execute command backup] ❌ error
❌ **test_golden_utterance[execute script weather]** — 0/1
Test Result
test_golden_utterance[execute script weather] ❌ error
❌ **test_golden_utterance[execute the backup command]** — 0/1
Test Result
test_golden_utterance[execute the backup command] ❌ error
❌ **test_golden_utterance[launch command backup]** — 0/1
Test Result
test_golden_utterance[launch command backup] ❌ error
❌ **test_golden_utterance[launch script weather]** — 0/1
Test Result
test_golden_utterance[launch script weather] ❌ error
❌ **test_golden_utterance[launch the weather script]** — 0/1
Test Result
test_golden_utterance[launch the weather script] ❌ error
❌ **test_golden_utterance[list my aliases]** — 0/1
Test Result
test_golden_utterance[list my aliases] ❌ error
❌ **test_golden_utterance[please run command backup for me]** — 0/1
Test Result
test_golden_utterance[please run command backup for me] ❌ xfailed
❌ **test_golden_utterance[run backup script]** — 0/1
Test Result
test_golden_utterance[run backup script] ❌ error
❌ **test_golden_utterance[run command backup]** — 0/1
Test Result
test_golden_utterance[run command backup] ❌ error
❌ **test_golden_utterance[run command]** — 0/1
Test Result
test_golden_utterance[run command] ❌ xfailed
❌ **test_golden_utterance[run my backup script]** — 0/1
Test Result
test_golden_utterance[run my backup script] ❌ error
❌ **test_golden_utterance[run script weather]** — 0/1
Test Result
test_golden_utterance[run script weather] ❌ error
❌ **test_golden_utterance[run the backup script]** — 0/1
Test Result
test_golden_utterance[run the backup script] ❌ error
❌ **test_golden_utterance[show me my scripts]** — 0/1
Test Result
test_golden_utterance[show me my scripts] ❌ error
❌ **test_golden_utterance[start script backup]** — 0/1
Test Result
test_golden_utterance[start script backup] ❌ error
❌ **test_golden_utterance[start the backup script]** — 0/1
Test Result
test_golden_utterance[start the backup script] ❌ error
❌ **test_golden_utterance[tell me the scripts you know]** — 0/1
Test Result
test_golden_utterance[tell me the scripts you know] ❌ error
❌ **test_golden_utterance[what commands are configured]** — 0/1
Test Result
test_golden_utterance[what commands are configured] ❌ error
❌ **test_golden_utterance[what scripts do you have]** — 0/1
Test Result
test_golden_utterance[what scripts do you have] ❌ error
❌ **test_golden_utterance[which scripts can you run]** — 0/1
Test Result
test_golden_utterance[which scripts can you run] ❌ error
❌ **test_negative_confusable_not_claimed[execute a wolfram alpha search]** — 0/1
Test Result
test_negative_confusable_not_claimed[execute a wolfram alpha search] ❌ error
❌ **test_negative_confusable_not_claimed[launch a timer for 5 minutes]** — 0/1
Test Result
test_negative_confusable_not_claimed[launch a timer for 5 minutes] ❌ error
❌ **test_negative_confusable_not_claimed[launch backup camera]** — 0/1
Test Result
test_negative_confusable_not_claimed[launch backup camera] ❌ error
❌ **test_negative_confusable_not_claimed[list my reminders]** — 0/1
Test Result
test_negative_confusable_not_claimed[list my reminders] ❌ error
❌ **test_negative_confusable_not_claimed[play some music]** — 0/1
Test Result
test_negative_confusable_not_claimed[play some music] ❌ error
❌ **test_negative_confusable_not_claimed[run the news for me]** — 0/1
Test Result
test_negative_confusable_not_claimed[run the news for me] ❌ error
❌ **test_negative_confusable_not_claimed[search the web for cats]** — 0/1
Test Result
test_negative_confusable_not_claimed[search the web for cats] ❌ error
❌ **test_negative_confusable_not_claimed[show me my calendar]** — 0/1
Test Result
test_negative_confusable_not_claimed[show me my calendar] ❌ error
❌ **test_negative_confusable_not_claimed[start my day]** — 0/1
Test Result
test_negative_confusable_not_claimed[start my day] ❌ error
❌ **test_negative_confusable_not_claimed[start the timer]** — 0/1
Test Result
test_negative_confusable_not_claimed[start the timer] ❌ error
❌ **test_negative_confusable_not_claimed[tell me a joke]** — 0/1
Test Result
test_negative_confusable_not_claimed[tell me a joke] ❌ error
❌ **test_negative_confusable_not_claimed[what are my favorite songs]** — 0/1
Test Result
test_negative_confusable_not_claimed[what are my favorite songs] ❌ error
❌ **test_negative_confusable_not_claimed[what's the weather]** — 0/1
Test Result
test_negative_confusable_not_claimed[what's the weather] ❌ error

🚌 Bus Coverage

Is every event handler pullin' its weight? Let's check! 🏋️‍♂️

⚠️ Bus coverage report unavailable — check the job log.


System.exit(0); // With love from OVOS 🖥️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant