File: .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py (affects all 88/93 scored workshop files)
Overall Score: N/A — tooling defect, not a content defect (corpus mean: 6.15)
Flagged Dimensions:
| Dimension |
Score |
Benchmark |
Delta |
| checkpoint_quality (weight 2.0) |
0.0 on 88/93 files |
checklist items × 2.5, capped at 10 |
-10.0 (near-universal false negative) |
Root Cause (≤ 2 sentences):
CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE) only matches a literal ✅ Unicode character, but the workshop's own style convention (enforced in AGENTS.md/.github/workflows/guidelines.md) requires every step to end with ## :white_check_mark: Checkpoint using the GitHub emoji shortcode, not the raw glyph. As a result 88 of 93 files (95%) are scored has_checkpoint: false and checkpoint_quality: 0.0 even though they contain a fully compliant checkpoint section with 4+ checklist items.
Evidence (quoted from the file):
CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE)
Contrast with actual workshop content (e.g. workshop/04-github-actions-intro.md):
## :white_check_mark: Checkpoint
- [ ] You can identify \on`, `jobs`, and `steps` in a workflow file`
Learning Science Rationale:
This is a measurement-validity failure rather than a curriculum-design failure: the rubric's checkpoint_quality dimension (weight 2.0, the highest tier alongside cognitive load and active learning) is meant to operationalize the well-established value of retrieval-practice checkpoints (Roediger & Karpicke's testing effect) for consolidating learning. A broken detector silently zeroes out this signal corpus-wide, which both understates every file's true overall score by roughly 1.7–2.0 points (2.0 weight ÷ total weight ≈ 20% of the composite) and — more importantly — hides real gaps: the 5 files that genuinely have thin or missing checkpoints (per the findings list) cannot be distinguished from the 83 files that already comply, because the detector currently returns 0.0 for all of them alike.
Improvement Prompt (for an agent):
Open .github/skills/curriculum-quantitative-assessment/curriculum_assessment.py and fix CHECKPOINT_RE so it matches the GitHub emoji shortcode actually used across workshop/*.md files, not just the literal Unicode ✅ character. The workshop style convention (see AGENTS.md) requires the exact heading "## :white_check_mark: Checkpoint" (case-insensitive). Update the regex to something like:
CHECKPOINT_RE = re.compile(r"##\s+(?:✅|:white_check_mark:)\s*Checkpoint", re.IGNORECASE)
Then re-run the rubric scorer across workshop/*.md and confirm has_checkpoint flips to true for files that contain "## :white_check_mark: Checkpoint" with checklist items, while files that are genuinely missing a checkpoint section (or use a non-compliant heading) still score has_checkpoint: false. Do not touch any workshop content files as part of this fix — this is a scoring-script-only change. Add or update any existing unit test for CHECKPOINT_RE if one exists in the same directory.
Expected Score After Fix: Corpus mean should rise from 6.15 / 10.0 to roughly 7.8–8.0 / 10.0 once checkpoint_quality correctly reflects the 83+ files that already comply; the 5 currently-flagged findings should remain the lowest scorers and can then be triaged on real signal instead of a false universal zero.
Generated by 🔬 Curriculum Quality Evaluator · copilot · auto · 62.1 AIC · ⌖ 23.2 AIC · ⊞ 9.5K · ◷
File:
.github/skills/curriculum-quantitative-assessment/curriculum_assessment.py(affects all 88/93 scored workshop files)Overall Score: N/A — tooling defect, not a content defect (corpus mean:
6.15)Flagged Dimensions:
0.0on 88/93 files-10.0(near-universal false negative)Root Cause (≤ 2 sentences):
CHECKPOINT_RE = re.compile(r"##\s+✅\s*Checkpoint", re.IGNORECASE)only matches a literal✅Unicode character, but the workshop's own style convention (enforced inAGENTS.md/.github/workflows/guidelines.md) requires every step to end with## :white_check_mark: Checkpointusing the GitHub emoji shortcode, not the raw glyph. As a result 88 of 93 files (95%) are scoredhas_checkpoint: falseandcheckpoint_quality: 0.0even though they contain a fully compliant checkpoint section with 4+ checklist items.Evidence (quoted from the file):
Contrast with actual workshop content (e.g.
workshop/04-github-actions-intro.md):Learning Science Rationale:
This is a measurement-validity failure rather than a curriculum-design failure: the rubric's
checkpoint_qualitydimension (weight 2.0, the highest tier alongside cognitive load and active learning) is meant to operationalize the well-established value of retrieval-practice checkpoints (Roediger & Karpicke's testing effect) for consolidating learning. A broken detector silently zeroes out this signal corpus-wide, which both understates every file's true overall score by roughly 1.7–2.0 points (2.0 weight ÷ total weight ≈ 20% of the composite) and — more importantly — hides real gaps: the 5 files that genuinely have thin or missing checkpoints (per thefindingslist) cannot be distinguished from the 83 files that already comply, because the detector currently returns0.0for all of them alike.Improvement Prompt (for an agent):
Expected Score After Fix: Corpus mean should rise from
6.15 / 10.0to roughly7.8–8.0 / 10.0oncecheckpoint_qualitycorrectly reflects the 83+ files that already comply; the 5 currently-flagged findings should remain the lowest scorers and can then be triaged on real signal instead of a false universal zero.