test(skills): guard the frontmatter fix with a real malformed SKILL.md on disk - #148
Merged
Conversation
…d on disk The tests added with the parser fix build their own SKILL.md files in tmp directories. They keep passing even if every real skill bundle in the repo is normalised, so nothing exercises discovery against a genuinely malformed file sitting on disk — which is the only form the original bug ever took. Adds one to the example app. `examples/.../skills/smoke_frontmatter/SKILL.md` is written the way a Windows or Office editor writes one: first three bytes are a UTF-8 BOM (EF BB BF, invisible in every editor), and the frontmatter keys use underscores rather than hyphens. Either one, before the fix, made the whole block parse as body: the skill owned no tools *and* `allowed-tools:` leaked into the rendered PROCEDURE. Both halves were silent. Three tests in tests/scaffold/test_skill_resolve.py assert on the shipped bytes rather than a copy: - `test_fixture_is_still_malformed` guards the guard. An editor that strips BOMs on save would otherwise quietly turn this file into an ordinary skill and take the regression with it, with every test still green. - `test_shipped_fixture_parses_through_discovery` runs `parse_skill_bundle` over the real directory: tools parsed, requires-actions parsed, no frontmatter in the body. It asserts on tokens that only appear inside the frontmatter block — the body legitimately mentions `allowed-tools:` while explaining itself. - `test_fixture_is_exposed_by_the_example_agent` reads agent.yaml, because a fixture no agent loads is not exercising discovery at all. Mutation-checked four ways, each caught by the intended test: strip the BOM, hyphenate the keys, unregister the skill from agent.yaml, and revert the parser's BOM strip in skill_resolve.py. The SKILL.md body states that it is deliberately malformed and names the test that depends on it, so the next person to open it does not helpfully repair it. agent.yaml gets the registration line only. The gating and the deliberate whatsapp/whatsapp_call sibling gap used while verifying this by hand are NOT included: the example documents itself as gating nothing, and the sibling gap would make `jvagent validate` emit an advisory on the shipped example app for every user. `validate` and `validate --strict` both exit 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #147. That PR fixed the parser and added tests — but every one of those tests builds its own
SKILL.mdin a tmp directory. They would all keep passing if every real skill bundle in the repo were normalised, so nothing exercises discovery against a genuinely malformed file on disk, which is the only form the bug ever took in the field.This adds one, and asserts on it.
The fixture
examples/jvagent_app/agents/jvagent/orchestrator_agent/skills/smoke_frontmatter/SKILL.mdis written the way a Windows or Office editor writes one:EF BB BF) — invisible in every editorallowed_tools,requires_actions) rather than hyphensBefore #147, either one made the whole block parse as body: the skill owned no tools and
allowed-tools:leaked into the renderedPROCEDURE. Both halves silent.The file's own body says it is deliberately malformed and names the test that depends on it, so the next person to open it doesn't helpfully repair it.
The tests
Three in
tests/scaffold/test_skill_resolve.py, asserting on the shipped bytes rather than a copy:test_fixture_is_still_malformedtest_shipped_fixture_parses_through_discoveryparse_skill_bundleover the real directory: tools parsed,requires-actionsparsed, no frontmatter in the bodytest_fixture_is_exposed_by_the_example_agentThe second asserts on tokens that only ever appear inside the frontmatter block (
name: smoke_frontmatter,spec: jv), because the body legitimately mentionsallowed-tools:while explaining itself.Mutation-checked four ways, each caught by the intended test and nothing else:
test_fixture_is_still_malformedredtest_fixture_is_still_malformedredagent.yaml→test_fixture_is_exposed_by_the_example_agentredskill_resolve.py→test_shipped_fixture_parses_through_discoveryredWhat is deliberately NOT here
While verifying this by hand through jvmessenger I also set
skill_only_toolsand introduced awhatsapp/whatsapp_callsibling gap to exercise the gate and the #142 advisory. Neither is included:skill_only_tools: []with a comment saying so)jvagent validateprint an advisory on the shipped example app for every useragent.yamlgets the registration line and a comment.jvagent validateandvalidate --strictboth exit 0.Verification
pre-commit run --all-files— cleanpytest tests/— 3359 passed, 4 skipped (3 more thanmain)allowed_toolshad parsed and conferred ownership.