Skip to content

test(skills): guard the frontmatter fix with a real malformed SKILL.md on disk - #148

Merged
eldonm merged 2 commits into
mainfrom
test/skill-frontmatter-fixture
Aug 8, 2026
Merged

test(skills): guard the frontmatter fix with a real malformed SKILL.md on disk#148
eldonm merged 2 commits into
mainfrom
test/skill-frontmatter-fixture

Conversation

@eldonm

@eldonm eldonm commented Aug 8, 2026

Copy link
Copy Markdown
Member

Follow-up to #147. That PR fixed the parser and added tests — but every one of those tests builds its own SKILL.md in a tmp directory. They would all keep passing if every real skill bundle in the repo were normalised, so nothing exercises discovery against a genuinely malformed file on disk, which is the only form the bug ever took in the field.

This adds one, and asserts on it.

The fixture

examples/jvagent_app/agents/jvagent/orchestrator_agent/skills/smoke_frontmatter/SKILL.md is written the way a Windows or Office editor writes one:

  • first three bytes are a UTF-8 BOM (EF BB BF) — invisible in every editor
  • frontmatter keys use underscores (allowed_tools, requires_actions) rather than hyphens

Before #147, either one made the whole block parse as body: the skill owned no tools and allowed-tools: leaked into the rendered PROCEDURE. Both halves silent.

The file's own body says it is deliberately malformed and names the test that depends on it, so the next person to open it doesn't helpfully repair it.

The tests

Three in tests/scaffold/test_skill_resolve.py, asserting on the shipped bytes rather than a copy:

Test What it catches
test_fixture_is_still_malformed Guards the guard — an editor that strips BOMs on save would turn this into an ordinary skill and take the regression with it, every test still green
test_shipped_fixture_parses_through_discovery parse_skill_bundle over the real directory: tools parsed, requires-actions parsed, no frontmatter in the body
test_fixture_is_exposed_by_the_example_agent A fixture no agent loads is not exercising discovery at all

The second asserts on tokens that only ever appear inside the frontmatter block (name: smoke_frontmatter, spec: jv), because the body legitimately mentions allowed-tools: while explaining itself.

Mutation-checked four ways, each caught by the intended test and nothing else:

  1. strip the BOM → test_fixture_is_still_malformed red
  2. hyphenate the keys → test_fixture_is_still_malformed red
  3. unregister the skill from agent.yamltest_fixture_is_exposed_by_the_example_agent red
  4. revert the BOM strip in skill_resolve.pytest_shipped_fixture_parses_through_discovery red

What is deliberately NOT here

While verifying this by hand through jvmessenger I also set skill_only_tools and introduced a whatsapp / whatsapp_call sibling gap to exercise the gate and the #142 advisory. Neither is included:

  • the example documents itself as gating nothing (skill_only_tools: [] with a comment saying so)
  • the sibling gap would make jvagent validate print an advisory on the shipped example app for every user

agent.yaml gets the registration line and a comment. jvagent validate and validate --strict both exit 0.

Verification

  • pre-commit run --all-files — clean
  • pytest tests/ — 3359 passed, 4 skipped (3 more than main)
  • Verified end-to-end through jvmessenger against a live agent before reducing to this fixture: the gated tool ran only once the skill activated, proving the BOM'd underscore allowed_tools had parsed and conferred ownership.

Eldon Marks and others added 2 commits August 8, 2026 11:51
…d on disk

The tests added with the parser fix build their own SKILL.md files in tmp
directories. They keep passing even if every real skill bundle in the repo is
normalised, so nothing exercises discovery against a genuinely malformed file
sitting on disk — which is the only form the original bug ever took.

Adds one to the example app. `examples/.../skills/smoke_frontmatter/SKILL.md`
is written the way a Windows or Office editor writes one: first three bytes are
a UTF-8 BOM (EF BB BF, invisible in every editor), and the frontmatter keys use
underscores rather than hyphens. Either one, before the fix, made the whole
block parse as body: the skill owned no tools *and* `allowed-tools:` leaked into
the rendered PROCEDURE. Both halves were silent.

Three tests in tests/scaffold/test_skill_resolve.py assert on the shipped bytes
rather than a copy:

- `test_fixture_is_still_malformed` guards the guard. An editor that strips BOMs
  on save would otherwise quietly turn this file into an ordinary skill and take
  the regression with it, with every test still green.
- `test_shipped_fixture_parses_through_discovery` runs `parse_skill_bundle` over
  the real directory: tools parsed, requires-actions parsed, no frontmatter in
  the body. It asserts on tokens that only appear inside the frontmatter block —
  the body legitimately mentions `allowed-tools:` while explaining itself.
- `test_fixture_is_exposed_by_the_example_agent` reads agent.yaml, because a
  fixture no agent loads is not exercising discovery at all.

Mutation-checked four ways, each caught by the intended test: strip the BOM,
hyphenate the keys, unregister the skill from agent.yaml, and revert the
parser's BOM strip in skill_resolve.py.

The SKILL.md body states that it is deliberately malformed and names the test
that depends on it, so the next person to open it does not helpfully repair it.

agent.yaml gets the registration line only. The gating and the deliberate
whatsapp/whatsapp_call sibling gap used while verifying this by hand are NOT
included: the example documents itself as gating nothing, and the sibling gap
would make `jvagent validate` emit an advisory on the shipped example app for
every user. `validate` and `validate --strict` both exit 0.
@eldonm
eldonm merged commit 3153217 into main Aug 8, 2026
3 checks passed
@eldonm
eldonm deleted the test/skill-frontmatter-fixture branch August 8, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant