feat(.github): add recurring module scorecard workflow#1012
Merged
Conversation
bpmct
marked this pull request as ready for review
July 14, 2026 16:44
DevelopmentCats
approved these changes
Jul 14, 2026
DevelopmentCats
left a comment
DevelopmentCats
left a comment
Collaborator
There was a problem hiding this comment.
I cant say that I have any complaints since we went through most of this together @bpmct but Jakub might :P
david-fraley
left a comment
david-fraley
left a comment
Contributor
There was a problem hiding this comment.
Two primary thoughts for ya @bpmct
- Adding in a complementary skill for how to interact with all this, that points at
scorecard.md. Could help with the inevitable AI use in module creation/maintenance that might occur. I'm thinking of this like school -- you, the prof, just handed out the rubric; can you give students a study sheet (the skill) so that they build what you want - Is it possible to extend this to templates?
bpmct
added a commit
that referenced
this pull request
Jul 15, 2026
…overall math (#1013) Follow-up to #1012, which merged before the last two commits on its branch landed. This carries the scoring calibration fix and the index intro restructure. ## Why An audit of the initial batch found the model systematically over-crediting Restricted-Network Readiness. 10 modules scored 20/20; most were wrong. The model's own notes rationalized credit with reasoning like "the URL **could be proxied/mirrored at network level**" (any URL can) and endpoints being "**implicit** in API key usage" (implicit is not documented). claude-code scored 20/20 on restricted-network with no mirror variable and no air-gap docs. ## Changes - **`SCORECARD.md`**: tightened criteria definitions. Mirrorable artifact source requires an actual module variable overriding the tool's own download URL (auxiliary config URLs, proxies, and DNS rewrites do not count). Egress transparency requires a dedicated section enumerating endpoints. New Strictness rule: never credit out-of-module workarounds or infer undocumented behavior. - **`score-modules.ts`**: calibration anchors in the scoring prompt. Highlights: name the exact URL-override variable or score 0; offline/skip-install modes count only toward bring-your-own binary; theme-level N/A is for modules that download nothing, 0 is for missing capability; inline placeholder keys in README examples cap the secrets criterion at half; perfect theme scores trigger a self-recheck. - **`score-modules.ts`**: `fixOverall` now catches a utility overall emitted as `X / 75` and normalizes it to `/ 100` (previously passed through unrecomputed). - **`scorecard-index.ts`**: index intro restructured as what / why / how it's generated. ## Validation All 48 discussions were re-scored with this code. Only 3 modules retain 20/20 restricted-network, each verified legitimate: Amazon Q (`q_install_url` + air-gapped section), Portable Desktop (`url` + checksum), Mux (`registry_url`). claude-code dropped to 67 (matches a manual hand-score of ~65), code-server 97 → 87. Live results: [📊 Module Scorecards](#1011). Known limitation, unchanged by this PR: ±3-5 run-to-run noise on subjective criteria even at temperature 0. If the PR regression comments get noisy, a follow-up can add a tolerance band. 🤖 Generated with [Coder Agents](https://coder.com/docs/ai-coder/agents)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a recurring workflow that scores every module in the
codernamespace against a documented rubric and posts the results as GitHub Discussions: one dedicated discussion per module, plus a pinned 📊 Module Scorecards index grouped by module type (Agent / IDE / Utility). A separate PR workflow compares changed modules against their current score and comments with the diff.Why a scorecard? It gives us a shared definition of quality (and a goalpost) for modules: what we care about for each one, in one place, for any contributor. It is not a strict gate today, though it may become one, and the criteria will evolve over time. It also helps catch regressions, for example a feature or documented capability being removed, since scores are re-evaluated whenever a module changes.
The rubric lives in
.github/scorecard/SCORECARD.mdand is read directly by the scoring script at runtime, so editing that one file changes the criteria for every future run. No script changes needed.Suggested review order
.github/scorecard/SCORECARD.md- the contract (80 lines).github/scorecard/lib.ts- shared parsing + GraphQL helpers.github/scorecard/score-modules.ts- scoring + discussion upserts + PR reports.github/scorecard/scorecard-index.ts- index aggregationRather than simulating the prompt, validate against live output: all 48 modules were scored and posted from this code. Example module discussion: Claude Code module; index: #1011.
Changes
.github/scorecard/SCORECARD.md- the rubric: universal themes (Presentation & Onboarding, Credential Hygiene, Restricted-Network Readiness, Engineering Quality) plus a track section (Agent / IDE / Utility) out of 100.github/scorecard/lib.ts- shared constants, summary-table parsing, GraphQL client, discussion lookup.github/scorecard/score-modules.ts- scores modules with Claude (claude-sonnet-4-5, temperature 0), recomputes the overall deterministically from theme scores, and upserts each module's discussion.--pr-reportmode compares against the current discussion instead of posting.github/scorecard/scorecard-index.ts- rebuilds the pinned index from the module discussions, grouped by track.github/workflows/module-scorecards.yaml- weekly cron,workflow_dispatch, push tomain: rubric change triggers a full run, module changes trigger a scoped run of just those modules; index rebuilds either way.github/workflows/module-scorecard-check.yaml- on PRs touching modules (internal only), scores the changed modules and posts a sticky comment flagging regressions, improvements, or unchanged scores. Read-only against discussionsSecrets
Both are already added to this repo's Actions secrets, so the workflows are ready to run on merge:
SCORECARD_ANTHROPIC_API_KEY- Anthropic API key for scoringSCORECARD_DISCUSSIONS_TOKEN- GitHub PAT with Discussions read/writeDesign notes
git-clonedownloads nothing), and the score normalizes to 100.coder-utils,agentapi,vscode-desktop-core) are excluded from scoring entirely via a deterministic README check.temperature: 0keeps re-scores stable (verified: identical re-score on an unchanged module).module-scorecardHTML comment marker; the index and PR comments usemodule-scorecard-indexandmodule-scorecard-prmarkers respectively.🤖 Generated with Coder Agents