Skip to content

Skill preflight rejects nested reference folders that the SDK accepts #111

Closed

Description

@sbroenne

Problem

Released pytest-skill-engineering==1.0.1 rejects the Excel CLI skill before sending a model message because its reference documentation is organized in a subdirectory.

Sanitized error:

Skill setup failed before model execution: Invalid references entry (must be a file): ...\excel-cli\references\commands

The skill contains SKILL.md with links to references\commands\*.md. Real, unbilled public-SDK ensure_loaded() / list() checks discover and enable this exact skill directory successfully. The package's additional preflight, not SDK discovery, blocks execution.

Reproduction

Supply a valid skill directory with this shape using client_mode="empty":

excel-cli\SKILL.md
excel-cli\references\commands\range.md
excel-cli\references\commands\table.md

Run the normal public evaluation with that directory in skill_directories. Observe failure before the model send. The sibling Excel MCP skill has flat reference files and passed this check.

Expected behavior

Accept SDK-supported reference subdirectories while retaining safe containment, readability, and applicable symlink/cycle validation. Missing, unsafe, or invalid skills must still fail before spending on a model.

Please verify the reference-directory rules against the Agent Skills specification, add a focused failing regression, and provide a public unbilled preflight path that exercises the same validation as actual evaluation. Consumers should not need a paid evaluation to discover this disagreement.

Impact

A release-only skill-value comparison on 1.0.1 verified three control executions, then stopped on its first CLI treatment without sending a model message. No skill-value conclusion was drawn from the incomplete run. This is a follow-up to #109 / #110, not a request to weaken discovery or isolation checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions