[workshop-sim] Workshop Simulation Report — 2026-09-09 (Run #79, 1000×Monte Carlo)

[github-actions]

Overview

• Date: 2026-09-09
• Students simulated: 46 × 1,000 Monte Carlo runs
• Workshop steps available: 30/30
• Overall success rate: 23.6% (95% Monte Carlo interval: 23.2%–24.0%)
• Highest-dropout step: 07-first-workflow (22.1% conditional dropout among 22,633 at-risk runs; 95% Monte Carlo interval: 21.6%–22.7%)
• Lowest curriculum quality step: 04-github-actions-intro.md (overall score 5.39/10)
• Learning KPI index: 2.88/10 (active_learning 4.18 · checkpoint_quality 0.0 · scaffolding 5.0) — checkpoint_quality is capped at 0 for every step because the rubric's ## Checkpoint detector expects a literal unicode ✅ but all 30 files use the :white_check_mark: shortcode; treat the checkpoint_quality component as an instrumentation gap, not a real absence of checkpoints (every mapped page we inspected has a full checklist).
• Model: 2026-07-survival-model-v2 / 2026-07-assumption-model-v2 (parameter hash 2024391902)
• Limitation: synthetic results reflect explicit model assumptions; intervals exclude model and population-assumption uncertainty

Part Summary

Part	Files	Mean Score	Std Dev
Part 1 — core path (lessons 00–14)	15	6.69 / 10.0	±1.74
Part 2 — advanced (lessons 15+)	15	6.07 / 10.0	±0.36
Overall corpus	30	6.38 / 10.0	±1.19 (approx.)

No pages are classified as other.

Critical Findings

1. A simulator scoring bug, not a real workshop defect, was driving a spurious 100% dropout at 08-run-your-workflow. Step 7's lexical workflowLockPublishCueCount cue-detector scored 0 because the commit/push instructions (lines 105–110 of 07-your-first-workflow.md) sit outside a 500-character window from the literal string .lock.yml, even though the checkpoint explicitly requires both files pushed to main. We supplied a semantic evaluation confirming the source, compile, and publish steps are all present, which corrected the model — post-fix, Step 8 shows only normal billing/trigger friction, not universal failure.
2. 04-github-actions-intro and 05-agentic-workflows-intro/05c/05b (the concept quartet before the first hands-on workflow) account for the largest real dropout mass — concept-overload (6,414 failures) and agentic-concept-gap/agentic-classification-gap/agentic-security-gap (combined ~10,000+ failures). This is a genuine learning barrier: 28 new concepts land in one page with zero callouts and only reveal-answer exercises, no worked troubleshooting.
3. Beginner and ui_preferred learners are structurally at risk on the core path: beginners average 0.5% success vs. 49.5% for advanced learners, and ui_preferred: true students average 12.0% vs. 34.2% for terminal-preferring students — largely because Steps 6–8 require the Codespace terminal regardless of UI preference.
4. Learning-quality health is weak for everyone who stays in: the Learning KPI index of 2.88/10 is dragged down almost entirely by the checkpoint_quality instrumentation bug (see Overview) and low active_learning on the concept-heavy pages (2.4–3.9/10 on 05-agentic-workflows-intro.md, 04-github-actions-intro.md, 08-run-your-workflow.md). Once checkpoint_quality is corrected, we expect the true KPI to sit meaningfully higher, but active_learning still needs work on the concept pages.
5. The most important repair belongs to Part 1 (00–14) — every one of the top-5 dropout steps and the lowest-scoring curriculum file are in the core path; Part 2 dropout (e.g., 17-add-mcp-tools at 4.7%) is an order of magnitude smaller because far fewer learners reach it.

Top Repairs to Prioritize

Some student dropout is expected and acceptable. Repairs below aim to maintain or improve the learning KPI index — none lower the cognitive bar or remove practice to chase headline completion numbers.

1. Fix the shared curriculum-assessment CHECKPOINT_RE regex to also match the :white_check_mark: shortcode used across all 30 workshop files (completion impact: ↔ · learning KPI impact: ↑ — this alone should lift checkpoint_quality from 0 to its true score across the corpus, materially raising the Learning KPI index without touching any workshop content).
2. Add one worked troubleshooting example and a callout to 04-github-actions-intro.md so learners get feedback on the labeling exercise instead of only "Reveal the labels" (completion impact: ↑ · learning KPI impact: ↑ — targets active_learning and troubleshooting scaffolding, the two weakest dimensions on the lowest-scoring page).
3. Add a single explicit "you pushed both files" confirmation cue directly adjacent to the .lock.yml commit/push commands in 07-your-first-workflow.md (e.g., an inline note right after the git push block) so both human learners and the automated content model can unambiguously verify the publish step succeeded before Step 8 (completion impact: ↑ · learning KPI impact: ↔ — reduces confusion risk without changing scope or difficulty).

Dropout by step

Step	At-risk runs	Dropouts	Conditional dropout rate	95% MC interval	Failure mode	Top reason
07-first-workflow	22,633	~5,003	22.1%	21.6%–22.7%	Access barrier	Missing Copilot model access (4,051) plus workflow-authoring friction (952)
05-agentic-intro	37,752	~6,991	18.5%	18.1%–18.9%	Learning barrier	Learner does not internalize the shift from deterministic jobs to goal-oriented agentic workflows (4,776)
04-actions-intro	44,166	~6,414	14.5%	14.2%–14.9%	Learning barrier	Concept overload — 28 new terms in one page (6,414)
05c-agentic-practice	30,761	~3,859	12.6%	12.2%–12.9%	Learning barrier	Cannot reliably classify agentic vs. deterministic tasks (3,859)
05b-agentic-security	26,902	~2,629	9.8%	9.4%–10.1%	Learning barrier	Does not grasp the two-layer sandbox + safe-outputs security model (2,629)
06-install-gh-aw	24,273	~1,640	6.8%	6.4%–7.1%	Access barrier	Extension install friction (auth/token scope/path switching) (1,640)
17-add-mcp-tools	15,086	~715	4.7%	4.4%–5.1%	Access barrier	MCP tooling setup friction (715)
02-setup	46,000	~1,834	4.0%	3.8%–4.2%	Access barrier	Codespace setup friction (1,834)
15-conditional-logic	15,999	~635	4.0%	3.7%–4.3%	Learning barrier	Conditional-logic concept friction (635)
18-share-and-reuse	14,371	~559	3.9%	3.6%–4.2%	Access barrier	Workflow reuse/sharing friction (559)

Curriculum quality and learning KPIs

Step file	Overall score	Active learning	Checkpoint quality*	Scaffolding	Learning KPI index*	Lowest rubric dimension	Repair priority
04-github-actions-intro.md	5.39	3.9	0.0	5.0	2.78	checkpoint_quality (instrumentation bug)	High
05-agentic-workflows-intro.md	5.43	2.4	0.0	5.0	2.24	active_learning	High
05b-agentic-workflows-security.md	5.75	2.5	0.0	5.0	2.27	active_learning	Medium
08-run-your-workflow.md	5.67	3.0	0.0	5.0	2.45	checkpoint_quality (instrumentation bug)	Medium
07-your-first-workflow.md	6.25	6.3	0.0	5.0	3.65	checkpoint_quality (instrumentation bug)	Medium
05c-agentic-workflows-practice.md	6.22	6.2	0.0	5.0	3.62	checkpoint_quality (instrumentation bug)	Low
02a-setup-codespace.md	6.09	5.0	0.0	5.0	3.18	checkpoint_quality (instrumentation bug)	Low
Cohort mean (30 steps)	6.38	4.18	0.0	5.0	2.88	—	—

*checkpoint_quality is 0.0 for all 30 files due to the CHECKPOINT_RE unicode-vs-shortcode detection bug described in Overview — every mapped page we inspected has a full ## :white_check_mark: Checkpoint checklist. Learning KPI values above are understated until that rubric bug is fixed.

Segment breakdowns

By technical level

Level	Students	Mean success rate
beginner	11	0.5%
github-basic	19	15.8%
actions-user	11	48.4%
advanced	5	49.5%

By personality

Personality	Students	Mean success rate
confused	6	22.1%
curious	15	20.7%
methodical	12	26.3%
impatient	6	26.9%
skeptical	7	23.5%

By UI preference

UI preference	Students	Mean success rate
Prefers browser (ui_preferred: true)	22	12.0%
Prefers CLI (ui_preferred: false)	24	34.2%

Notable student journeys (3)

• Surprising success — Learner 037 (beginner, methodical, enterprise-dev, CLI, ui_preferred: false): despite a 38% level baseline, this learner's methodical personality and terminal-first preference let them clear the Codespace-required Steps 6–8 far more often than typical beginners (4.2% success vs. the 0.5% beginner average), most commonly stalling later at 05-agentic-intro rather than the install/first-workflow gauntlet.
• Unexpected dropout — Learner 028 (actions-user, confused, enterprise-dev, CLI): actions-user learners average 48.4% success, but this student's confused personality dragged them to 20.0%, most often failing at 05-agentic-intro — evidence that prior Actions YAML familiarity does not transfer to agentic-concept comprehension without more scaffolding.
• Content-gap case — Learner 001 (github-basic, confused, data-science, VS Code, ui_preferred: true): failed overwhelmingly at 04-actions-intro (251 of 1,000 runs), the lowest-scoring curriculum page (5.39/10, active_learning 3.9/10, 28 new concepts, zero callouts) — a clear case where the content itself, not just learner profile, is the primary driver.

Generated by 🔬 Workshop Student Simulator · copilot · auto · 219.4 AIC · ⌖ 10.3 AIC · ⊞ 15.2K · ◷

• expires on Sep 10, 2026, 5:51 PM UTC

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3139

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #3140

View newer issue

---

This action was performed automatically by the Workshop Student Simulator workflow.