Skip to content

fix(skill): guide graders away from checks the starting state passes - #21

Merged
jhermesn merged 1 commit into
mainfrom
docs/grader-and-cleanup-rules
Nov 5, 2026
Merged

jhermesn merged 1 commit into
mainfrom
docs/grader-and-cleanup-rules

Conversation

@jhermesn

@jhermesn jhermesn commented Nov 5, 2026

Copy link
Copy Markdown
Owner

Authoring-guide rules for defect classes that only the real-account self-test caught:

  • checks that the freshly deployed template already satisfies, and guard checks ("keeps its 60-second timeout") that pass before the learner acts;
  • JMESPath filters placed right after a projection, which return an empty list even on the correct state;
  • jq fromjson on an attribute that --output text already prints as a JSON document;
  • length() on binary attributes inside --query;
  • cleanup that misses children the reference solution attaches to stack resources (API Gateway routes), which leaves the stack in DELETE_FAILED.

Guide-only change; the self-test already rejects these labs, and this keeps the agent from writing them.

Real-account self-tests rejected labs whose grader passed before the learner
acted, misread JSON or binary attributes, or filtered after a projection, and
labs whose cleanup missed what the solution created.
@jhermesn jhermesn self-assigned this Nov 5, 2026
@jhermesn
jhermesn merged commit 6209227 into main Nov 5, 2026
1 check passed
@jhermesn
jhermesn deleted the docs/grader-and-cleanup-rules branch November 5, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant