From 0ccb4d40393be4f6065d026fc5fc9ecd911126ae Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:33:11 +0000 Subject: [PATCH 1/8] feat(.github): add recurring module scorecard workflow --- .github/scorecard/.gitignore | 1 + .github/scorecard/SCORECARD.md | 278 ++++++++++++++ .github/scorecard/score-modules.ts | 447 +++++++++++++++++++++++ .github/scorecard/scorecard-index.ts | 354 ++++++++++++++++++ .github/workflows/module-scorecards.yaml | 56 +++ 5 files changed, 1136 insertions(+) create mode 100644 .github/scorecard/.gitignore create mode 100644 .github/scorecard/SCORECARD.md create mode 100644 .github/scorecard/score-modules.ts create mode 100644 .github/scorecard/scorecard-index.ts create mode 100644 .github/workflows/module-scorecards.yaml diff --git a/.github/scorecard/.gitignore b/.github/scorecard/.gitignore new file mode 100644 index 000000000..28b422503 --- /dev/null +++ b/.github/scorecard/.gitignore @@ -0,0 +1 @@ +.scorecard-results.json diff --git a/.github/scorecard/SCORECARD.md b/.github/scorecard/SCORECARD.md new file mode 100644 index 000000000..16112dfb7 --- /dev/null +++ b/.github/scorecard/SCORECARD.md @@ -0,0 +1,278 @@ +# Coder Registry Module Scorecard + +**100 pts** = **Universal criteria (75)** + **one track (25)** + +Score each criterion as **0 / half / full**. + +## Scoring rubric + +### Universal criteria — 75 pts + +| Criterion | Pts | Pass = | +| -------------------------------------------------- | -----: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Presentation & Onboarding** | **25** | | +| Configuration-mode examples | 12 | If the module has many options, each major mode has a documented example with sensible defaults, for example provider choice, app vs headless | +| Coder-context framing | 8 | Explains what the module adds on top of Coder, names both Coder and the target tool, and shows where Coder fits in the flow | +| Visual preview | 5 | README includes an image, GIF, or video of the module in action | +| **Credential Hygiene** | **20** | | +| Secrets marked sensitive | 16 | Sensitive inputs are marked `sensitive = true`, and README examples avoid inline secrets | +| Non-hardcoded auth path | 4 | If possible, README shows a path that avoids pasting raw keys into templates, for example ServiceAccount, IAM/OAuth/external auth, API key helper, or AI Gateway | +| **Restricted-Network Readiness** _(if applicable)_ | **20** | | +| Mirrorable artifact source | 10 | Download or install URL is configurable so it can point to an internal mirror or artifact store instead of the public internet | +| Bring-your-own binary | 6 | Download or install can be disabled entirely when the tool is already baked into the image | +| Egress transparency | 4 | External endpoints are documented, plus notes for restricted or air-gapped environments | +| **Engineering Quality** | **10** | | +| Input quality | 6 | Inputs have clear descriptions, sensible defaults, and `validation` where appropriate | +| Test coverage | 4 | Clear testing story, `.tftest.hcl` primarily covers business logic, TypeScript tests cover end-to-end behavior | + +### Agent track — 25 pts + +| Criterion | Pts | Pass = | +| --------------------- | --: | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| AI governance | 10 | Documented support for Coder AI Gateway and/or Agent Firewall, including how Coder governs auth, routing, or policy enforcement for the agent | +| Dashboard entry point | 5 | Documented `coder_app` support, built-in or example | +| Session continuity | 5 | Documented support for continuing an existing agent session across reconnects or relaunches, either through native resume/session ID support or a persistent session manager such as boo, tmux, or screen | +| Managed configuration | 5 | Documented support for managed MCP, settings, policies, or workdir config | + +### IDE track — 25 pts + +| Criterion | Pts | Pass = | +| ------------------------------------------ | --: | ---------------------------------------------------------------------------- | +| Dashboard entry point | 7 | `coder_app` support with proper launch behavior | +| Managed configuration | 6 | Documented support for managed IDE settings or config | +| Configurable folder or workdir | 6 | Documented support for opening or starting in a configured folder or workdir | +| Pre-installed extensions _(web IDEs only)_ | 6 | For web IDEs, documented support for pre-installing extensions | + +### Utility track + +Utility modules are scored on Universal criteria only, then normalized: + +`round(raw / 75 * 100)` + +## Notes + +| Topic | Rule | +| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Track assignment | Every score should record which track was used | +| If applicable | Criteria or themes marked _if applicable_ are excluded when the concern does not exist by construction, for example a module that downloads nothing. Excluded points are removed from the denominator and the final score is normalized to 100 | +| N/A handling | Outside _if applicable_ criteria, missing support scores zero. Only Utility modules skip the track section | +| Scoring | Full = implemented and documented; half = partial, awkward, or under-documented; zero = absent | + +## Grading + +| Score | Badge | +| -----: | ---------- | +| 90-100 | Exemplary | +| 75-89 | Strong | +| 50-74 | Adequate | +| <50 | Needs work | + +## Output format + +Every scorecard leads with a theme-level summary table, then a collapsed +drilldown with per-criterion tables grouped by theme. + +## Sample scorecards + +### `coder/claude-code` + +| Presentation & Onboarding | Agent Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +| ------------------------: | ----------------: | -----------------: | ---------------------------: | ------------------: | -----------: | +| **20 / 25** | **15 / 25** | **12 / 20** | **8 / 20** | **10 / 10** | **65 / 100** | + +
+Drilldown + +#### Presentation & Onboarding — 20 / 25 + +| Criterion | Max | Score | Notes | +| --------------------------- | --: | ----: | ------------------------------------------------------------------------------------- | +| Configuration-mode examples | 12 | 12 | Strong examples for API key, OAuth, AI Gateway, Bedrock, Vertex, and managed settings | +| Coder-context framing | 8 | 8 | Clearly explains what the module adds on top of Coder | +| Visual preview | 5 | 0 | No image, GIF, or video in README | + +#### Agent Integration — 15 / 25 + +| Criterion | Max | Score | Notes | +| --------------------- | --: | ----: | ------------------------------------------------------------ | +| AI governance | 10 | 5 | Strong AI Gateway support, no current Agent Firewall support | +| Dashboard entry point | 5 | 5 | README includes `coder_app` example | +| Session continuity | 5 | 0 | No documented resume or persistence pattern | +| Managed configuration | 5 | 5 | Strong managed settings and MCP support | + +#### Credential Hygiene — 12 / 20 + +| Criterion | Max | Score | Notes | +| ------------------------ | --: | ----: | ----------------------------------------------------------------------- | +| Secrets marked sensitive | 16 | 8 | Sensitive vars exist, but README still shows inline secrets in examples | +| Non-hardcoded auth path | 4 | 4 | Documents OAuth and AI Gateway paths | + +#### Restricted-Network Readiness — 8 / 20 + +| Criterion | Max | Score | Notes | +| -------------------------- | --: | ----: | ------------------------------------------------------------------------ | +| Mirrorable artifact source | 10 | 0 | Installer source is not configurable to use an internal mirror | +| Bring-your-own binary | 6 | 6 | Supports pre-baked binary path and disabling install | +| Egress transparency | 4 | 2 | External calls are somewhat implied, but not clearly listed in one place | + +#### Engineering Quality — 10 / 10 + +| Criterion | Max | Score | Notes | +| ------------- | --: | ----: | --------------------------------------------------------------------------------------- | +| Input quality | 6 | 6 | Inputs are well-described, with defaults and validation | +| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover end-to-end script behavior | + +#### Overall — 65 / 100 + +
+ +### `coder/code-server` + +| Presentation & Onboarding | IDE Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +| ------------------------: | --------------: | -----------------: | ---------------------------: | ------------------: | -----------: | +| **25 / 25** | **19 / 25** | **16 / 20** | **3 / 20** | **10 / 10** | **73 / 100** | + +
+Drilldown + +#### Presentation & Onboarding — 25 / 25 + +| Criterion | Max | Score | Notes | +| --------------------------- | --: | ----: | -------------------------------------------------------------------------------------------------- | +| Configuration-mode examples | 12 | 12 | Good examples for version pinning, extensions, settings, workspace vs folder, offline/cached modes | +| Coder-context framing | 8 | 8 | Clearly explains code-server in a Coder workspace | +| Visual preview | 5 | 5 | README includes a screenshot | + +#### IDE Integration — 19 / 25 + +| Criterion | Max | Score | Notes | +| ------------------------------------------ | --: | ----: | -------------------------------------------------------------------------------------- | +| Dashboard entry point | 7 | 7 | Built-in `coder_app` with launch behavior controls | +| Managed configuration | 6 | 6 | Supports user and machine settings | +| Configurable folder or workdir | 6 | 6 | Supports both `folder` and `workspace` | +| Pre-installed extensions _(web IDEs only)_ | 6 | 0 | Supports extensions, but no documented private marketplace or enterprise-managed story | + +#### Credential Hygiene — 16 / 20 + +| Criterion | Max | Score | Notes | +| ------------------------ | --: | ----: | ------------------------------------------- | +| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | +| Non-hardcoded auth path | 4 | 0 | No relevant auth-path guidance | + +#### Restricted-Network Readiness — 3 / 20 + +| Criterion | Max | Score | Notes | +| -------------------------- | --: | ----: | ---------------------------------------------------------------------------------------- | +| Mirrorable artifact source | 10 | 0 | Installer source is not configurable for an internal mirror | +| Bring-your-own binary | 6 | 3 | Offline and cached modes help, but there is no first-class custom binary/source override | +| Egress transparency | 4 | 0 | External endpoints are not clearly documented | + +#### Engineering Quality — 10 / 10 + +| Criterion | Max | Score | Notes | +| ------------- | --: | ----: | ----------------------------------------------------------------------------------------------------- | +| Input quality | 6 | 6 | Inputs are well-described, with defaults and validation | +| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover script behavior and settings merge flows | + +#### Overall — 73 / 100 + +
+ +### `coder/vscode-web` + +| Presentation & Onboarding | IDE Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +| ------------------------: | --------------: | -----------------: | ---------------------------: | ------------------: | -----------: | +| **25 / 25** | **19 / 25** | **16 / 20** | **0 / 20** | **7 / 10** | **67 / 100** | + +
+Drilldown + +#### Presentation & Onboarding — 25 / 25 + +| Criterion | Max | Score | Notes | +| --------------------------- | --: | ----: | ----------------------------------------------------------------------------------------------- | +| Configuration-mode examples | 12 | 12 | Good examples for folder, workspace, extensions, settings, pinning, cached and offline behavior | +| Coder-context framing | 8 | 8 | Clearly explains VS Code Web in a Coder workspace | +| Visual preview | 5 | 5 | README includes a GIF | + +#### IDE Integration — 19 / 25 + +| Criterion | Max | Score | Notes | +| ------------------------------------------ | --: | ----: | -------------------------------------------------------------------------------------- | +| Dashboard entry point | 7 | 7 | Built-in `coder_app` with launch behavior controls | +| Managed configuration | 6 | 6 | Supports machine settings and trust-related config | +| Configurable folder or workdir | 6 | 6 | Supports both `folder` and `workspace` | +| Pre-installed extensions _(web IDEs only)_ | 6 | 0 | Supports extensions, but no documented private marketplace or enterprise-managed story | + +#### Credential Hygiene — 16 / 20 + +| Criterion | Max | Score | Notes | +| ------------------------ | --: | ----: | ------------------------------------------- | +| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | +| Non-hardcoded auth path | 4 | 0 | No relevant auth-path guidance | + +#### Restricted-Network Readiness — 0 / 20 + +| Criterion | Max | Score | Notes | +| -------------------------- | --: | ----: | -------------------------------------------------------------------------------------------------- | +| Mirrorable artifact source | 10 | 0 | Download source is not configurable for an internal mirror | +| Bring-your-own binary | 6 | 0 | Offline and cached behavior exist, but no documented bring-your-own binary or custom artifact path | +| Egress transparency | 4 | 0 | External endpoints are not clearly documented | + +#### Engineering Quality — 7 / 10 + +| Criterion | Max | Score | Notes | +| ------------- | --: | ----: | ------------------------------------------------------------------------------------------------------------- | +| Input quality | 6 | 3 | Inputs are mostly well-described, but some validation and polish are less complete than the strongest modules | +| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover script behavior and settings merge flows | + +#### Overall — 67 / 100 + +
+ +### `coder/git-clone` (Utility) + +Utility module: no track section. Restricted-Network Readiness is not applicable, the module downloads nothing and clones a caller-controlled URL. Denominator is 55, normalized to 100. + +| Presentation & Onboarding | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +| ------------------------: | -----------------: | ---------------------------: | ------------------: | -----------: | +| **20 / 25** | **20 / 20** | N/A | **8 / 10** | **87 / 100** | + +
+Drilldown + +#### Presentation & Onboarding — 20 / 25 + +| Criterion | Max | Score | Notes | +| --------------------------- | --: | ----: | -------------------------------------------------------------------------------------------- | +| Configuration-mode examples | 12 | 12 | Examples for custom path, branch clones, GitHub/GitLab, self-hosted providers, external auth | +| Coder-context framing | 8 | 8 | Clearly explains cloning into a Coder workspace and composing with other modules | +| Visual preview | 5 | 0 | No image, GIF, or video in README | + +#### Credential Hygiene — 20 / 20 + +| Criterion | Max | Score | Notes | +| ------------------------ | --: | ----: | -------------------------------------------------------------------------------- | +| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | +| Non-hardcoded auth path | 4 | 4 | Documents Coder external auth (`coder_external_auth`) instead of embedded tokens | + +#### Restricted-Network Readiness — N/A + +| Criterion | Max | Score | Notes | +| -------------------------- | --: | ----: | ---------------------------------------------------------------------- | +| Mirrorable artifact source | — | N/A | Clone URL is fully caller-controlled, nothing to mirror | +| Bring-your-own binary | — | N/A | Module downloads nothing; relies on `git` already present in the image | +| Egress transparency | — | N/A | Only contacts the URL the caller provides | + +#### Engineering Quality — 8 / 10 + +| Criterion | Max | Score | Notes | +| ------------- | --: | ----: | -------------------------------------------------------------------------- | +| Input quality | 6 | 6 | Inputs are well-described with sensible defaults | +| Test coverage | 4 | 2 | Has TypeScript end-to-end tests, but no `.tftest.hcl` business-logic tests | + +#### Overall — 87 / 100 + +Raw 48 / 55 → `round(48 / 55 × 100)` = **87** + +
diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts new file mode 100644 index 000000000..3afc99212 --- /dev/null +++ b/.github/scorecard/score-modules.ts @@ -0,0 +1,447 @@ +#!/usr/bin/env bun +/** + * Scores Coder Registry modules against SCORECARD.md using Claude, then + * posts (or updates) one GitHub Discussion per module in coder/registry. + * + * Env (required): + * ANTHROPIC_API_KEY Anthropic API key + * GITHUB_DISCUSSIONS_TOKEN GitHub PAT with Discussions read/write + * + * Usage: + * bun run score-modules.ts [--modules a,b,c] [--dry-run] [--limit N] + * + * --dry-run prints scorecards to stdout and skips GitHub entirely. + */ + +import { readdir, readFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import path from "node:path"; + +const REGISTRY_ROOT = path.resolve(import.meta.dir, "..", ".."); +const MODULES_DIR = path.join(REGISTRY_ROOT, "registry", "coder", "modules"); +const SCORECARD_PATH = path.join(import.meta.dir, "SCORECARD.md"); + +const REPO_ID = "R_kgDOOVbRAA"; // coder/registry +const CATEGORY_ID = "DIC_kwDOOVbRAM4DBMLT"; // "Modules" category +const REPO_OWNER = "coder"; +const REPO_NAME = "registry"; + +const ANTHROPIC_MODEL = process.env.ANTHROPIC_MODEL ?? "claude-sonnet-4-5"; +const MAX_FILE_BYTES = 30_000; + +const MARKER = ""; +const RESULTS_PATH = path.join(import.meta.dir, ".scorecard-results.json"); + +interface SummaryScores { + track: "Agent" | "IDE" | "Utility"; + presentation: string; + integration: string; + credential: string; + network: string; + engineering: string; + overall: string; + overallNum: number; +} + +interface Result extends SummaryScores { + module: string; + name: string; + url: string; + scoredAt: string; +} + +function parseSummary(scorecard: string): SummaryScores | null { + const lines = scorecard.split("\n"); + const headerIdx = lines.findIndex( + (l) => + l.startsWith("|") && l.includes("Overall") && l.includes("Presentation"), + ); + if (headerIdx === -1) return null; + const headers = lines[headerIdx] + .split("|") + .map((c) => c.trim()) + .filter(Boolean); + const cells = lines[headerIdx + 2] + ?.split("|") + .map((c) => c.trim().replace(/\*\*/g, "")) + .filter(Boolean); + if (!cells || cells.length !== headers.length) return null; + const get = (name: string) => { + const i = headers.findIndex((h) => h.toLowerCase().includes(name)); + return i === -1 ? "\u2014" : cells[i]; + }; + const integrationIdx = headers.findIndex((h) => /integration/i.test(h)); + const track: SummaryScores["track"] = + integrationIdx === -1 + ? "Utility" + : /agent/i.test(headers[integrationIdx]) + ? "Agent" + : "IDE"; + const overall = get("overall"); + return { + track, + presentation: get("presentation"), + integration: integrationIdx === -1 ? "\u2014" : cells[integrationIdx], + credential: get("credential"), + network: get("restricted"), + engineering: get("engineering"), + overall, + overallNum: Number(overall.match(/(\d+)\s*\/\s*100/)?.[1] ?? 0), + }; +} + +async function saveResult(result: Result): Promise { + let results: Record = {}; + if (existsSync(RESULTS_PATH)) { + results = JSON.parse(await readFile(RESULTS_PATH, "utf8")); + } + results[result.module] = result; + await Bun.write(RESULTS_PATH, JSON.stringify(results, null, 2) + "\n"); +} + +async function displayName(moduleName: string): Promise { + const readme = await readFile( + path.join(MODULES_DIR, moduleName, "README.md"), + "utf8", + ); + const match = readme.match(/^display_name:\s*["']?([^"'\n]+)["']?\s*$/m); + return match ? match[1].trim() : moduleName; +} + +interface Args { + modules?: string[]; + dryRun: boolean; + limit?: number; +} + +function parseArgs(): Args { + const args: Args = { dryRun: false }; + const argv = process.argv.slice(2); + for (let i = 0; i < argv.length; i++) { + switch (argv[i]) { + case "--modules": + args.modules = argv[++i].split(",").map((s) => s.trim()); + break; + case "--dry-run": + args.dryRun = true; + break; + case "--limit": + args.limit = Number(argv[++i]); + break; + default: + console.error(`Unknown argument: ${argv[i]}`); + process.exit(1); + } + } + return args; +} + +async function readTruncated(filePath: string): Promise { + const content = await readFile(filePath, "utf8"); + if (content.length <= MAX_FILE_BYTES) return content; + return content.slice(0, MAX_FILE_BYTES) + "\n... [truncated]"; +} + +async function gatherModuleContext(moduleName: string): Promise { + const dir = path.join(MODULES_DIR, moduleName); + const parts: string[] = []; + const files = await readdir(dir, { recursive: true }); + const wanted = files.filter( + (f) => + typeof f === "string" && + (f.endsWith(".md") || + f.endsWith(".tf") || + f.endsWith(".tftest.hcl") || + f.endsWith(".test.ts") || + f.endsWith(".sh") || + f.endsWith(".tftpl")), + ) as string[]; + for (const f of wanted.sort()) { + const full = path.join(dir, f); + parts.push(`=== FILE: ${f} ===\n${await readTruncated(full)}`); + } + return parts.join("\n\n"); +} + +function fixOverall(scorecard: string): string { + // Find the summary table's data row: first row whose cells are bold + // "**a / b**" fractions (or N/A), ending with the overall cell. + const lines = scorecard.split("\n"); + const rowIdx = lines.findIndex( + (l) => /^\|\s*\*\*\d/.test(l.trim()) && l.includes("/ 100"), + ); + if (rowIdx === -1) return scorecard; + const cells = lines[rowIdx] + .split("|") + .map((c) => c.trim()) + .filter((c) => c !== ""); + const themeCells = cells.slice(0, -1); + let raw = 0; + let denom = 0; + for (const cell of themeCells) { + if (/n\/a/i.test(cell)) continue; + const m = cell.match(/(\d+(?:\.\d+)?)\s*\/\s*(\d+)/); + if (!m) return scorecard; // unexpected cell, leave untouched + raw += Number(m[1]); + denom += Number(m[2]); + } + if (denom === 0) return scorecard; + const overall = Math.round((raw / denom) * 100); + cells[cells.length - 1] = `**${overall} / 100**`; + lines[rowIdx] = `| ${cells.join(" | ")} |`; + let out = lines.join("\n"); + out = out.replace( + /#### Overall — \d+(?:\.\d+)? \/ 100/g, + `#### Overall — ${overall} / 100`, + ); + // Rewrite or append the normalization line under the Overall heading. + const normLine = + denom === 100 + ? "" + : `\n\nRaw ${raw} / ${denom} → round(${raw} / ${denom} × 100) = **${overall}**`; + out = out.replace(/\n+Raw \d+(?:\.\d+)?\s*\/\s*\d+[^\n]*/g, ""); + out = out.replace( + new RegExp(`(#### Overall — ${overall} / 100)`), + `$1${normLine}`, + ); + return out; +} + +async function scoreModule( + moduleName: string, + rubric: string, +): Promise { + const context = await gatherModuleContext(moduleName); + const prompt = `You are scoring a Coder Registry module against a scorecard rubric. + + +${rubric} + + + +${context} + + +Score the module strictly against the rubric. Rules: +- First decide the track: Agent, IDE, or Utility. Agent = AI coding agents (CLI agents like Claude Code, Goose, Aider). IDE = editors/IDEs users open (code-server, JetBrains, VS Code). Utility = everything else (auth, regions, git helpers, file browsers, etc). +- Score each criterion 0, half, or full. Full = implemented AND documented. Half = partial, awkward, or under-documented. Zero = absent. +- Apply "if applicable" exclusions per the rubric: when a concern does not exist by construction, mark the criterion N/A, exclude its points from the denominator, and normalize the final score to 100. +- Utility modules skip the track section (denominator 75 before any N/A exclusions), then normalize: round(raw / denominator * 100). +- Notes must be specific and evidence-based (reference actual variables, README sections, or files you saw). Never invent features. + +Output ONLY the scorecard markdown in EXACTLY this structure (this example shows an Agent module; use IDE Integration or omit the track section for Utility): + +| Presentation & Onboarding | Agent Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +|---:|---:|---:|---:|---:|---:| +| **X / 25** | **X / 25** | **X / 20** | **X / 20 or N/A** | **X / 10** | **X / 100** | + +
+Drilldown + +#### Presentation & Onboarding — X / 25 + +| Criterion | Max | Score | Notes | +|---|---:|---:|---| +| Configuration-mode examples | 12 | X | ... | +| Coder-context framing | 8 | X | ... | +| Visual preview | 5 | X | ... | + +(...remaining theme tables in rubric order, highest weight first, each with per-criterion rows...) + +#### Overall — X / 100 + +(If normalized, show: Raw X / Y → round(X / Y × 100) = **Z**) + +
+ +Do not add any prose before or after the scorecard.`; + + const res = await fetch("https://api.anthropic.com/v1/messages", { + method: "POST", + headers: { + "x-api-key": process.env.ANTHROPIC_API_KEY!, + "anthropic-version": "2023-06-01", + "content-type": "application/json", + }, + body: JSON.stringify({ + model: ANTHROPIC_MODEL, + max_tokens: 4096, + messages: [{ role: "user", content: prompt }], + }), + }); + if (!res.ok) { + throw new Error(`Anthropic API error ${res.status}: ${await res.text()}`); + } + const data = (await res.json()) as { + content: { type: string; text?: string }[]; + }; + const text = data.content + .filter((c) => c.type === "text") + .map((c) => c.text) + .join("\n"); + return fixOverall(text.trim()); +} + +async function graphql( + query: string, + variables: Record, +): Promise { + const res = await fetch("https://api.github.com/graphql", { + method: "POST", + headers: { + Authorization: `Bearer ${process.env.GITHUB_DISCUSSIONS_TOKEN}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ query, variables }), + }); + const body = (await res.json()) as { data?: T; errors?: unknown[] }; + if (!res.ok || body.errors?.length) { + throw new Error( + `GitHub GraphQL error: ${JSON.stringify(body.errors ?? body)}`, + ); + } + return body.data!; +} + +async function findDiscussion( + title: string, +): Promise<{ id: string; url: string } | null> { + const q = `"${title}" in:title repo:${REPO_OWNER}/${REPO_NAME}`; + const data = await graphql<{ + search: { nodes: { id: string; title: string; url: string }[] }; + }>( + ` + query ($q: String!) { + search(query: $q, type: DISCUSSION, first: 10) { + nodes { + ... on Discussion { + id + title + url + } + } + } + } + `, + { q }, + ); + return data.search.nodes.find((n) => n.title === title) ?? null; +} + +async function upsertDiscussion( + title: string, + body: string, +): Promise<{ id: string; url: string; created: boolean }> { + const existing = await findDiscussion(title); + if (existing) { + await graphql( + ` + mutation ($id: ID!, $body: String!) { + updateDiscussion(input: { discussionId: $id, body: $body }) { + discussion { + id + } + } + } + `, + { id: existing.id, body }, + ); + return { id: existing.id, url: existing.url, created: false }; + } + const data = await graphql<{ + createDiscussion: { discussion: { id: string; url: string } }; + }>( + ` + mutation ($repoId: ID!, $catId: ID!, $title: String!, $body: String!) { + createDiscussion( + input: { + repositoryId: $repoId + categoryId: $catId + title: $title + body: $body + } + ) { + discussion { + id + url + } + } + } + `, + { repoId: REPO_ID, catId: CATEGORY_ID, title, body }, + ); + return { ...data.createDiscussion.discussion, created: true }; +} + +function discussionBody( + moduleName: string, + name: string, + scorecard: string, +): string { + const now = new Date().toISOString().slice(0, 10); + return `A discussion dedicated to the [${name}](https://registry.coder.com/modules/coder/${moduleName}) module. Share your thoughts, questions, and feedback here. + +## Module Scorecard + +${scorecard} + +--- +Scored against [SCORECARD.md](https://github.com/${REPO_OWNER}/${REPO_NAME}/blob/main/.github/scorecard/SCORECARD.md) on ${now} with \`${ANTHROPIC_MODEL}\`. +${MARKER}`; +} + +async function main() { + const args = parseArgs(); + if (!process.env.ANTHROPIC_API_KEY) + throw new Error("ANTHROPIC_API_KEY not set"); + if (!args.dryRun && !process.env.GITHUB_DISCUSSIONS_TOKEN) { + throw new Error("GITHUB_DISCUSSIONS_TOKEN not set"); + } + + const rubric = await readFile(SCORECARD_PATH, "utf8"); + let modules = + args.modules ?? + (await readdir(MODULES_DIR, { withFileTypes: true })) + .filter((d) => d.isDirectory()) + .map((d) => d.name) + .sort(); + if (args.limit) modules = modules.slice(0, args.limit); + + for (const mod of modules) { + if (!existsSync(path.join(MODULES_DIR, mod, "README.md"))) { + console.error(`skip ${mod}: no README.md`); + continue; + } + process.stderr.write(`scoring ${mod}... `); + try { + const scorecard = await scoreModule(mod, rubric); + if (args.dryRun) { + console.log(`\n===== coder/${mod} =====\n${scorecard}\n`); + process.stderr.write("done (dry-run)\n"); + continue; + } + const name = await displayName(mod); + const { url, created } = await upsertDiscussion( + `${name} module`, + discussionBody(mod, name, scorecard), + ); + const summary = parseSummary(scorecard); + if (summary) { + await saveResult({ + module: mod, + name, + url, + scoredAt: new Date().toISOString(), + ...summary, + }); + } else { + process.stderr.write(`warn: could not parse summary for ${mod}\n`); + } + process.stderr.write(`${created ? "created" : "updated"} ${url}\n`); + } catch (err) { + process.stderr.write(`FAILED: ${err}\n`); + } + } +} + +main(); diff --git a/.github/scorecard/scorecard-index.ts b/.github/scorecard/scorecard-index.ts new file mode 100644 index 000000000..395c32b75 --- /dev/null +++ b/.github/scorecard/scorecard-index.ts @@ -0,0 +1,354 @@ +#!/usr/bin/env bun +/** + * Builds (or updates) the pinned index discussion listing every module + * scorecard with per-theme scores, from local results written by + * score-modules.ts (.scorecard-results.json). + * + * Env (required): + * GITHUB_DISCUSSIONS_TOKEN GitHub PAT with Discussions read/write + * + * Usage: + * bun run scorecard-index.ts [--dry-run] [--downloads downloads.json] [--from-github] + * + * --from-github: one-time backfill of .scorecard-results.json from existing + * discussions (for runs made before local result tracking existed). + * downloads.json (optional): {"claude-code": 12345, ...} module dir -> count. + * When provided, the table is ranked by downloads; otherwise by overall score. + */ + +import { readFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import path from "node:path"; + +const RESULTS_PATH = path.join(import.meta.dir, ".scorecard-results.json"); + +const REPO_ID = "R_kgDOOVbRAA"; // coder/registry +const CATEGORY_ID = "DIC_kwDOOVbRAM4DBMLT"; // "Modules" +const MARKER = ""; +const INDEX_MARKER = ""; +const INDEX_TITLE = "📊 Module Scorecards"; + +interface Result { + module: string; + name: string; + url: string; + scoredAt: string; + track: "Agent" | "IDE" | "Utility"; + presentation: string; + integration: string; + credential: string; + network: string; + engineering: string; + overall: string; + overallNum: number; +} + +interface Args { + dryRun: boolean; + downloadsPath?: string; + fromGithub: boolean; +} + +function parseArgs(): Args { + const args: Args = { dryRun: false, fromGithub: false }; + const argv = process.argv.slice(2); + for (let i = 0; i < argv.length; i++) { + switch (argv[i]) { + case "--dry-run": + args.dryRun = true; + break; + case "--downloads": + args.downloadsPath = argv[++i]; + break; + case "--from-github": + args.fromGithub = true; + break; + default: + console.error(`Unknown argument: ${argv[i]}`); + process.exit(1); + } + } + return args; +} + +async function graphql( + query: string, + variables: Record = {}, +): Promise { + const res = await fetch("https://api.github.com/graphql", { + method: "POST", + headers: { + Authorization: `Bearer ${process.env.GITHUB_DISCUSSIONS_TOKEN}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ query, variables }), + }); + const body = (await res.json()) as { data?: T; errors?: unknown[] }; + if (!res.ok || body.errors?.length) { + throw new Error( + `GitHub GraphQL error: ${JSON.stringify(body.errors ?? body)}`, + ); + } + return body.data!; +} + +function parseSummaryFromBody( + title: string, + url: string, + body: string, +): Result | null { + const dirMatch = body.match( + /registry\.coder\.com\/modules\/coder\/([a-z0-9-]+)/, + ); + const lines = body.split("\n"); + const headerIdx = lines.findIndex( + (l) => + l.startsWith("|") && l.includes("Overall") && l.includes("Presentation"), + ); + if (headerIdx === -1) return null; + const headers = lines[headerIdx] + .split("|") + .map((c) => c.trim()) + .filter(Boolean); + const cells = lines[headerIdx + 2] + ?.split("|") + .map((c) => c.trim().replace(/\*\*/g, "")) + .filter(Boolean); + if (!cells || cells.length !== headers.length) return null; + const get = (name: string) => { + const i = headers.findIndex((h) => h.toLowerCase().includes(name)); + return i === -1 ? "—" : cells[i]; + }; + const integrationIdx = headers.findIndex((h) => /integration/i.test(h)); + const track: Result["track"] = + integrationIdx === -1 + ? "Utility" + : /agent/i.test(headers[integrationIdx]) + ? "Agent" + : "IDE"; + const overall = get("overall"); + return { + module: dirMatch?.[1] ?? "", + name: title.replace(/ module$/, ""), + url, + scoredAt: new Date().toISOString(), + track, + presentation: get("presentation"), + integration: integrationIdx === -1 ? "—" : cells[integrationIdx], + credential: get("credential"), + network: get("restricted"), + engineering: get("engineering"), + overall, + overallNum: Number(overall.match(/(\d+)\s*\/\s*100/)?.[1] ?? 0), + }; +} + +async function backfillFromGithub(): Promise> { + const results: Record = {}; + let cursor: string | null = null; + for (;;) { + const data: { + repository: { + discussions: { + pageInfo: { hasNextPage: boolean; endCursor: string }; + nodes: { title: string; url: string; body: string }[]; + }; + }; + } = await graphql( + `query($cursor: String) { + repository(owner: "coder", name: "registry") { + discussions(first: 100, after: $cursor, categoryId: "${CATEGORY_ID}") { + pageInfo { hasNextPage endCursor } + nodes { title url body } + } + } + }`, + { cursor }, + ); + for (const d of data.repository.discussions.nodes) { + if (!d.body.includes(MARKER) || d.body.includes(INDEX_MARKER)) continue; + const r = parseSummaryFromBody(d.title, d.url, d.body); + if (r?.module) results[r.module] = r; + else console.error(`warn: could not parse "${d.title}"`); + } + if (!data.repository.discussions.pageInfo.hasNextPage) break; + cursor = data.repository.discussions.pageInfo.endCursor; + } + await Bun.write(RESULTS_PATH, JSON.stringify(results, null, 2) + "\n"); + console.error( + `backfilled ${Object.keys(results).length} results to ${RESULTS_PATH}`, + ); + return results; +} + +function buildBody( + results: Record, + downloads?: Record, +): string { + const rank = (a: Result, b: Result) => { + if (downloads) { + const da = downloads[a.module] ?? 0; + const db = downloads[b.module] ?? 0; + if (db !== da) return db - da; + } + return b.overallNum - a.overallNum; + }; + + const dlHeader = downloads ? " Downloads |" : ""; + const dlSep = downloads ? "---:|" : ""; + + const section = ( + title: string, + blurb: string, + rows: Result[], + withIntegration: boolean, + ): string => { + if (rows.length === 0) return ""; + const integrationHeader = withIntegration ? ` ${title} Integration |` : ""; + const integrationSep = withIntegration ? "---:|" : ""; + const body = rows + .sort(rank) + .map((r) => { + const dl = downloads + ? ` ${(downloads[r.module] ?? 0).toLocaleString()} |` + : ""; + const integration = withIntegration ? ` ${r.integration} |` : ""; + return `| [${r.name}](${r.url}) |${dl} ${r.presentation} |${integration} ${r.credential} | ${r.network} | ${r.engineering} | **${r.overall}** |`; + }) + .join("\n"); + return `### ${title} modules + +${blurb} + +| Module |${dlHeader} Presentation & Onboarding |${integrationHeader} Credential Hygiene | Restricted-Network | Engineering Quality | Overall | +|---|${dlSep}---:|${integrationSep}---:|---:|---:|---:| +${body} +`; + }; + + const all = Object.values(results); + const sections = [ + section( + "Agent", + "AI coding agents you can run in a workspace.", + all.filter((r) => r.track === "Agent"), + true, + ), + section( + "IDE", + "Editors and IDEs users open from the dashboard.", + all.filter((r) => r.track === "IDE"), + true, + ), + section( + "Utility", + "Auth, regions, git helpers, and other workspace plumbing. Scored on the universal themes only, normalized to 100.", + all.filter((r) => r.track === "Utility"), + false, + ), + ] + .filter(Boolean) + .join("\n"); + + const now = new Date().toISOString().slice(0, 10); + return `Every module in the \`coder\` namespace, scored against [SCORECARD.md](https://github.com/coder/registry/blob/main/.github/scorecard/SCORECARD.md). Each module links to its dedicated discussion, where you can share thoughts, questions, and feedback. + +**Looking for a way to contribute?** Low scores are contribution opportunities. Pick a module, open its discussion to see exactly which criteria it misses (visual previews, air-gapped install docs, tests, session persistence, and so on), and open a PR against [\`registry/coder/modules/\`](https://github.com/coder/registry/tree/main/registry/coder/modules). + +${sections} +**Notes**: N/A means the theme does not apply by construction and is excluded from the denominator. + +--- +Updated ${now}. Scores refresh when modules change. +${INDEX_MARKER}`; +} + +async function upsertIndex(body: string): Promise { + const data = await graphql<{ + search: { nodes: { id: string; title: string; url: string }[] }; + }>( + ` + query ($q: String!) { + search(query: $q, type: DISCUSSION, first: 10) { + nodes { + ... on Discussion { + id + title + url + } + } + } + } + `, + { q: `"${INDEX_TITLE}" in:title repo:coder/registry` }, + ); + const existing = data.search.nodes.find((n) => n.title === INDEX_TITLE); + if (existing) { + await graphql( + ` + mutation ($id: ID!, $body: String!) { + updateDiscussion(input: { discussionId: $id, body: $body }) { + discussion { + id + } + } + } + `, + { id: existing.id, body }, + ); + return existing.url; + } + const created = await graphql<{ + createDiscussion: { discussion: { url: string } }; + }>( + ` + mutation ($repoId: ID!, $catId: ID!, $title: String!, $body: String!) { + createDiscussion( + input: { + repositoryId: $repoId + categoryId: $catId + title: $title + body: $body + } + ) { + discussion { + url + } + } + } + `, + { repoId: REPO_ID, catId: CATEGORY_ID, title: INDEX_TITLE, body }, + ); + return created.createDiscussion.discussion.url; +} + +async function main() { + const args = parseArgs(); + if (!process.env.GITHUB_DISCUSSIONS_TOKEN) { + throw new Error("GITHUB_DISCUSSIONS_TOKEN not set"); + } + let results: Record; + if (args.fromGithub) { + results = await backfillFromGithub(); + } else { + if (!existsSync(RESULTS_PATH)) { + throw new Error( + `${RESULTS_PATH} not found. Run score-modules.ts first, or use --from-github to backfill.`, + ); + } + results = JSON.parse(await readFile(RESULTS_PATH, "utf8")); + } + let downloads: Record | undefined; + if (args.downloadsPath) { + downloads = JSON.parse(await readFile(args.downloadsPath, "utf8")); + } + const body = buildBody(results, downloads); + if (args.dryRun) { + console.log(body); + return; + } + const url = await upsertIndex(body); + console.error(`index: ${url}`); +} + +main(); diff --git a/.github/workflows/module-scorecards.yaml b/.github/workflows/module-scorecards.yaml new file mode 100644 index 000000000..91b89cc9f --- /dev/null +++ b/.github/workflows/module-scorecards.yaml @@ -0,0 +1,56 @@ +# Scores every module in the coder namespace against +# .github/scorecard/SCORECARD.md using Claude, then posts (or updates) one +# GitHub Discussion per module plus the pinned "📊 Module Scorecards" index. +# +# The rubric lives in .github/scorecard/SCORECARD.md. The scoring script +# reads it directly at runtime, so editing that file changes the criteria +# for the next run, and pushing a change to it re-runs scoring immediately. +# +# Required repository secrets: +# SCORECARD_ANTHROPIC_API_KEY Anthropic API key used for scoring +# SCORECARD_DISCUSSIONS_TOKEN GitHub PAT with Discussions read/write + +name: Module Scorecards + +on: + schedule: + # Weekly, Mondays 06:00 UTC + - cron: "0 6 * * 1" + push: + branches: + - main + paths: + - ".github/scorecard/SCORECARD.md" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: module-scorecards + cancel-in-progress: false + +jobs: + score: + name: Score modules and update discussions + runs-on: ubuntu-latest + timeout-minutes: 45 + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Bun + uses: oven-sh/setup-bun@v2 + with: + bun-version: latest + + - name: Score modules and update module discussions + env: + ANTHROPIC_API_KEY: ${{ secrets.SCORECARD_ANTHROPIC_API_KEY }} + GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} + run: bun run .github/scorecard/score-modules.ts + + - name: Update index discussion + env: + GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} + run: bun run .github/scorecard/scorecard-index.ts From 748607ae78af1ec82a39313dd71aee8c8d7dfddf Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:35:42 +0000 Subject: [PATCH 2/8] feat(.github/scorecard): exclude internal building-block modules from scoring --- .github/scorecard/SCORECARD.md | 13 +++++++------ .github/scorecard/score-modules.ts | 17 +++++++++++++++++ 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/.github/scorecard/SCORECARD.md b/.github/scorecard/SCORECARD.md index 16112dfb7..842f6491d 100644 --- a/.github/scorecard/SCORECARD.md +++ b/.github/scorecard/SCORECARD.md @@ -51,12 +51,13 @@ Utility modules are scored on Universal criteria only, then normalized: ## Notes -| Topic | Rule | -| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Track assignment | Every score should record which track was used | -| If applicable | Criteria or themes marked _if applicable_ are excluded when the concern does not exist by construction, for example a module that downloads nothing. Excluded points are removed from the denominator and the final score is normalized to 100 | -| N/A handling | Outside _if applicable_ criteria, missing support scores zero. Only Utility modules skip the track section | -| Scoring | Full = implemented and documented; half = partial, awkward, or under-documented; zero = absent | +| Topic | Rule | +| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Track assignment | Every score should record which track was used | +| Internal building blocks | Modules whose README cautions against direct use (for example "we do not recommend using this module directly") are excluded from scoring entirely | +| If applicable | Criteria or themes marked _if applicable_ are excluded when the concern does not exist by construction, for example a module that downloads nothing. Excluded points are removed from the denominator and the final score is normalized to 100 | +| N/A handling | Outside _if applicable_ criteria, missing support scores zero. Only Utility modules skip the track section | +| Scoring | Full = implemented and documented; half = partial, awkward, or under-documented; zero = absent | ## Grading diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts index 3afc99212..20d692a7b 100644 --- a/.github/scorecard/score-modules.ts +++ b/.github/scorecard/score-modules.ts @@ -108,6 +108,19 @@ async function displayName(moduleName: string): Promise { return match ? match[1].trim() : moduleName; } +// Internal building-block modules caution against direct use and are not +// meant to be consumed by template authors, so they are excluded from +// scoring entirely. +async function isInternalBuildingBlock(moduleName: string): Promise { + const readme = await readFile( + path.join(MODULES_DIR, moduleName, "README.md"), + "utf8", + ); + return /do not recommend using this module directly|not intended to be used directly|internal building block/i.test( + readme, + ); +} + interface Args { modules?: string[]; dryRun: boolean; @@ -412,6 +425,10 @@ async function main() { console.error(`skip ${mod}: no README.md`); continue; } + if (await isInternalBuildingBlock(mod)) { + process.stderr.write(`skip ${mod}: internal building block\n`); + continue; + } process.stderr.write(`scoring ${mod}... `); try { const scorecard = await scoreModule(mod, rubric); From ad0d056b5bc570f3521da9e4ee06c39a4178d5c5 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:37:32 +0000 Subject: [PATCH 3/8] fix(.github/workflows): pin actions to commit SHAs in module-scorecards --- .github/workflows/module-scorecards.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/module-scorecards.yaml b/.github/workflows/module-scorecards.yaml index 91b89cc9f..043352fb7 100644 --- a/.github/workflows/module-scorecards.yaml +++ b/.github/workflows/module-scorecards.yaml @@ -37,10 +37,10 @@ jobs: timeout-minutes: 45 steps: - name: Checkout - uses: actions/checkout@v4 + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - name: Setup Bun - uses: oven-sh/setup-bun@v2 + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: bun-version: latest From 0abad3d9ca17e5780bbfaf07cc76c1f8b6d120a2 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:43:26 +0000 Subject: [PATCH 4/8] feat(.github): scope scorecard runs to changed modules and add PR check --- .github/scorecard/score-modules.ts | 78 ++++++++++++++++- .github/workflows/module-scorecard-check.yaml | 83 +++++++++++++++++++ .github/workflows/module-scorecards.yaml | 49 ++++++++++- 3 files changed, 204 insertions(+), 6 deletions(-) create mode 100644 .github/workflows/module-scorecard-check.yaml diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts index 20d692a7b..e6fc1f4c3 100644 --- a/.github/scorecard/score-modules.ts +++ b/.github/scorecard/score-modules.ts @@ -125,6 +125,7 @@ interface Args { modules?: string[]; dryRun: boolean; limit?: number; + prReport?: string; } function parseArgs(): Args { @@ -141,6 +142,9 @@ function parseArgs(): Args { case "--limit": args.limit = Number(argv[++i]); break; + case "--pr-report": + args.prReport = argv[++i]; + break; default: console.error(`Unknown argument: ${argv[i]}`); process.exit(1); @@ -279,6 +283,7 @@ Do not add any prose before or after the scorecard.`; body: JSON.stringify({ model: ANTHROPIC_MODEL, max_tokens: 4096, + temperature: 0, messages: [{ role: "user", content: prompt }], }), }); @@ -318,10 +323,12 @@ async function graphql( async function findDiscussion( title: string, -): Promise<{ id: string; url: string } | null> { +): Promise<{ id: string; url: string; body: string } | null> { const q = `"${title}" in:title repo:${REPO_OWNER}/${REPO_NAME}`; const data = await graphql<{ - search: { nodes: { id: string; title: string; url: string }[] }; + search: { + nodes: { id: string; title: string; url: string; body: string }[]; + }; }>( ` query ($q: String!) { @@ -331,6 +338,7 @@ async function findDiscussion( id title url + body } } } @@ -386,6 +394,48 @@ async function upsertDiscussion( return { ...data.createDiscussion.discussion, created: true }; } +// Builds one PR-report section comparing a fresh scorecard against the +// module's current discussion (the last scoring from main), so PRs can see +// whether they improve, regress, or hold the score. +function prReportSection( + mod: string, + name: string, + scorecard: string, + summary: SummaryScores | null, + baseline: (SummaryScores & { url: string }) | null, +): string { + const details = `
\nFull scorecard for this PR\n\n${scorecard}\n\n
`; + if (!summary) { + return `### \`coder/${mod}\`\n\nCould not parse the generated scorecard; see full output below.\n\n${details}`; + } + if (!baseline) { + return `### \`coder/${mod}\`: first scorecard, **${summary.overall}**\n\nNo existing scorecard discussion found for ${name}; this is the initial score. A dedicated discussion is created after merge.\n\n${details}`; + } + const delta = summary.overallNum - baseline.overallNum; + const themes: [string, string, string][] = [ + ["Presentation & Onboarding", baseline.presentation, summary.presentation], + ["Integration", baseline.integration, summary.integration], + ["Credential Hygiene", baseline.credential, summary.credential], + ["Restricted-Network", baseline.network, summary.network], + ["Engineering Quality", baseline.engineering, summary.engineering], + ["**Overall**", `**${baseline.overall}**`, `**${summary.overall}**`], + ]; + const table = [ + "| Theme | Before | After |", + "|---|---:|---:|", + ...themes.map(([t, b, a]) => `| ${t} | ${b} | ${a} |`), + ].join("\n"); + let verdict: string; + if (delta < 0) { + verdict = `\u26a0\ufe0f **Score regression**: ${baseline.overallNum} \u2192 ${summary.overallNum} (${delta}). Check the drilldown for which criteria dropped.`; + } else if (delta > 0) { + verdict = `\u2705 **Score improvement**: ${baseline.overallNum} \u2192 ${summary.overallNum} (+${delta}).`; + } else { + verdict = `\u2705 **Score unchanged** at ${summary.overall}. This PR does not affect the module's scorecard; the results are still good.`; + } + return `### [\`coder/${mod}\`](${baseline.url}): ${baseline.overallNum} \u2192 ${summary.overallNum}\n\n${verdict}\n\n${table}\n\n${details}`; +} + function discussionBody( moduleName: string, name: string, @@ -420,6 +470,7 @@ async function main() { .sort(); if (args.limit) modules = modules.slice(0, args.limit); + const prSections: string[] = []; for (const mod of modules) { if (!existsSync(path.join(MODULES_DIR, mod, "README.md"))) { console.error(`skip ${mod}: no README.md`); @@ -438,6 +489,20 @@ async function main() { continue; } const name = await displayName(mod); + if (args.prReport) { + // PR mode: compare against the module's current discussion and + // build a report instead of touching any discussion. + const summary = parseSummary(scorecard); + const existing = await findDiscussion(`${name} module`); + const parsed = existing ? parseSummary(existing.body) : null; + const baseline = + existing && parsed ? { ...parsed, url: existing.url } : null; + prSections.push( + prReportSection(mod, name, scorecard, summary, baseline), + ); + process.stderr.write("compared\n"); + continue; + } const { url, created } = await upsertDiscussion( `${name} module`, discussionBody(mod, name, scorecard), @@ -459,6 +524,15 @@ async function main() { process.stderr.write(`FAILED: ${err}\n`); } } + + if (args.prReport) { + const report = + prSections.length > 0 + ? `## Module Scorecard Check\n\n${prSections.join("\n\n")}\n\n---\nScored against [SCORECARD.md](https://github.com/${REPO_OWNER}/${REPO_NAME}/blob/main/.github/scorecard/SCORECARD.md) with \`${ANTHROPIC_MODEL}\`. Language-model scores are advisory.\n` + : ""; + await Bun.write(args.prReport, report); + process.stderr.write(`wrote PR report to ${args.prReport}\n`); + } } main(); diff --git a/.github/workflows/module-scorecard-check.yaml b/.github/workflows/module-scorecard-check.yaml new file mode 100644 index 000000000..e36f431a6 --- /dev/null +++ b/.github/workflows/module-scorecard-check.yaml @@ -0,0 +1,83 @@ +# Scores modules changed in a pull request against +# .github/scorecard/SCORECARD.md and posts a sticky comment comparing the +# PR's score to the module's current scorecard discussion. Flags +# regressions, celebrates improvements, and confirms unchanged scores. +# +# Never creates or updates discussions; the comparison is read-only against +# GitHub and only writes the PR comment. Skipped for fork PRs because it +# requires secrets. +# +# Required repository secrets: +# SCORECARD_ANTHROPIC_API_KEY Anthropic API key used for scoring +# SCORECARD_DISCUSSIONS_TOKEN GitHub PAT with Discussions read (baseline lookup) + +name: Module Scorecard Check + +on: + pull_request: + paths: + - "registry/coder/modules/**" + +permissions: + contents: read + pull-requests: write + +concurrency: + group: module-scorecard-check-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + check: + name: Compare module scores against main + if: github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - name: Checkout + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 0 + + - name: Setup Bun + uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 + with: + bun-version: latest + + - name: Determine changed modules + id: changed + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + run: | + MODULES=$(git diff --name-only "${BASE_SHA}"...HEAD | grep -oP '^registry/coder/modules/\K[^/]+' | sort -u | paste -sd, -) + echo "modules=${MODULES}" >> "${GITHUB_OUTPUT}" + echo "Changed modules: ${MODULES:-none}" + + - name: Score changed modules + if: steps.changed.outputs.modules != '' + env: + ANTHROPIC_API_KEY: ${{ secrets.SCORECARD_ANTHROPIC_API_KEY }} + GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} + MODULES: ${{ steps.changed.outputs.modules }} + run: bun run .github/scorecard/score-modules.ts --modules "${MODULES}" --pr-report /tmp/scorecard-report.md + + - name: Upsert PR comment + if: steps.changed.outputs.modules != '' + env: + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + # No report means every changed module was skipped (for example, + # internal building blocks); leave no comment in that case. + if [[ ! -s /tmp/scorecard-report.md ]]; then + echo "No scorecard report generated; skipping comment." + exit 0 + fi + MARKER="" + COMMENT_ID=$(gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" --paginate --jq ".[] | select(.body | contains(\"${MARKER}\")) | .id" | head -1) + if [[ -n "${COMMENT_ID}" ]]; then + gh api --method PATCH "repos/${GITHUB_REPOSITORY}/issues/comments/${COMMENT_ID}" -F body=@/tmp/scorecard-report.md > /dev/null + echo "Updated comment ${COMMENT_ID}" + else + gh api "repos/${GITHUB_REPOSITORY}/issues/${PR_NUMBER}/comments" -F body=@/tmp/scorecard-report.md > /dev/null + echo "Created comment" + fi diff --git a/.github/workflows/module-scorecards.yaml b/.github/workflows/module-scorecards.yaml index 043352fb7..9e773c522 100644 --- a/.github/workflows/module-scorecards.yaml +++ b/.github/workflows/module-scorecards.yaml @@ -1,10 +1,12 @@ -# Scores every module in the coder namespace against +# Scores modules in the coder namespace against # .github/scorecard/SCORECARD.md using Claude, then posts (or updates) one # GitHub Discussion per module plus the pinned "📊 Module Scorecards" index. # # The rubric lives in .github/scorecard/SCORECARD.md. The scoring script # reads it directly at runtime, so editing that file changes the criteria -# for the next run, and pushing a change to it re-runs scoring immediately. +# for the next run, and pushing a change to it re-runs full scoring. +# Pushes that touch individual modules re-score only those modules; the +# index is rebuilt either way. # # Required repository secrets: # SCORECARD_ANTHROPIC_API_KEY Anthropic API key used for scoring @@ -21,6 +23,7 @@ on: - main paths: - ".github/scorecard/SCORECARD.md" + - "registry/coder/modules/**" workflow_dispatch: permissions: @@ -38,19 +41,57 @@ jobs: steps: - name: Checkout uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + fetch-depth: 0 - name: Setup Bun uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 with: bun-version: latest + - name: Determine scope + id: scope + env: + EVENT_NAME: ${{ github.event_name }} + BEFORE: ${{ github.event.before }} + AFTER: ${{ github.sha }} + run: | + # Full run for schedule, manual dispatch, rubric changes, or when + # the previous commit is unavailable (force push, new branch). + if [[ "${EVENT_NAME}" != "push" ]]; then + echo "args=" >> "${GITHUB_OUTPUT}" + echo "Full run (${EVENT_NAME})" + exit 0 + fi + if [[ "${BEFORE}" == "0000000000000000000000000000000000000000" ]] || ! git cat-file -e "${BEFORE}" 2>/dev/null; then + echo "args=" >> "${GITHUB_OUTPUT}" + echo "Full run (no previous commit)" + exit 0 + fi + CHANGED=$(git diff --name-only "${BEFORE}" "${AFTER}") + if grep -q "^\.github/scorecard/SCORECARD\.md$" <<< "${CHANGED}"; then + echo "args=" >> "${GITHUB_OUTPUT}" + echo "Full run (rubric changed)" + exit 0 + fi + MODULES=$(grep -oP '^registry/coder/modules/\K[^/]+' <<< "${CHANGED}" | sort -u | paste -sd, -) + if [[ -z "${MODULES}" ]]; then + echo "args=" >> "${GITHUB_OUTPUT}" + echo "Full run (no module paths matched)" + else + echo "args=--modules ${MODULES}" >> "${GITHUB_OUTPUT}" + echo "Scoped run: ${MODULES}" + fi + - name: Score modules and update module discussions env: ANTHROPIC_API_KEY: ${{ secrets.SCORECARD_ANTHROPIC_API_KEY }} GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} - run: bun run .github/scorecard/score-modules.ts + run: bun run .github/scorecard/score-modules.ts ${{ steps.scope.outputs.args }} - name: Update index discussion env: GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} - run: bun run .github/scorecard/scorecard-index.ts + # --from-github rebuilds the full result set from the module + # discussions, so a single-module run still produces a complete index. + run: bun run .github/scorecard/scorecard-index.ts --from-github From e29b1ad5ddc51e3623c4d9d2376a5d15da1e04c5 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:53:39 +0000 Subject: [PATCH 5/8] refactor(.github/scorecard): trim rubric samples and extract shared lib --- .github/scorecard/.gitignore | 1 - .github/scorecard/SCORECARD.md | 207 +-------------------- .github/scorecard/lib.ts | 124 +++++++++++++ .github/scorecard/score-modules.ts | 151 ++-------------- .github/scorecard/scorecard-index.ts | 220 +++++------------------ .github/workflows/module-scorecards.yaml | 6 +- 6 files changed, 185 insertions(+), 524 deletions(-) delete mode 100644 .github/scorecard/.gitignore create mode 100644 .github/scorecard/lib.ts diff --git a/.github/scorecard/.gitignore b/.github/scorecard/.gitignore deleted file mode 100644 index 28b422503..000000000 --- a/.github/scorecard/.gitignore +++ /dev/null @@ -1 +0,0 @@ -.scorecard-results.json diff --git a/.github/scorecard/SCORECARD.md b/.github/scorecard/SCORECARD.md index 842f6491d..584a759a2 100644 --- a/.github/scorecard/SCORECARD.md +++ b/.github/scorecard/SCORECARD.md @@ -73,207 +73,8 @@ Utility modules are scored on Universal criteria only, then normalized: Every scorecard leads with a theme-level summary table, then a collapsed drilldown with per-criterion tables grouped by theme. -## Sample scorecards +## Live scorecards -### `coder/claude-code` - -| Presentation & Onboarding | Agent Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | -| ------------------------: | ----------------: | -----------------: | ---------------------------: | ------------------: | -----------: | -| **20 / 25** | **15 / 25** | **12 / 20** | **8 / 20** | **10 / 10** | **65 / 100** | - -
-Drilldown - -#### Presentation & Onboarding — 20 / 25 - -| Criterion | Max | Score | Notes | -| --------------------------- | --: | ----: | ------------------------------------------------------------------------------------- | -| Configuration-mode examples | 12 | 12 | Strong examples for API key, OAuth, AI Gateway, Bedrock, Vertex, and managed settings | -| Coder-context framing | 8 | 8 | Clearly explains what the module adds on top of Coder | -| Visual preview | 5 | 0 | No image, GIF, or video in README | - -#### Agent Integration — 15 / 25 - -| Criterion | Max | Score | Notes | -| --------------------- | --: | ----: | ------------------------------------------------------------ | -| AI governance | 10 | 5 | Strong AI Gateway support, no current Agent Firewall support | -| Dashboard entry point | 5 | 5 | README includes `coder_app` example | -| Session continuity | 5 | 0 | No documented resume or persistence pattern | -| Managed configuration | 5 | 5 | Strong managed settings and MCP support | - -#### Credential Hygiene — 12 / 20 - -| Criterion | Max | Score | Notes | -| ------------------------ | --: | ----: | ----------------------------------------------------------------------- | -| Secrets marked sensitive | 16 | 8 | Sensitive vars exist, but README still shows inline secrets in examples | -| Non-hardcoded auth path | 4 | 4 | Documents OAuth and AI Gateway paths | - -#### Restricted-Network Readiness — 8 / 20 - -| Criterion | Max | Score | Notes | -| -------------------------- | --: | ----: | ------------------------------------------------------------------------ | -| Mirrorable artifact source | 10 | 0 | Installer source is not configurable to use an internal mirror | -| Bring-your-own binary | 6 | 6 | Supports pre-baked binary path and disabling install | -| Egress transparency | 4 | 2 | External calls are somewhat implied, but not clearly listed in one place | - -#### Engineering Quality — 10 / 10 - -| Criterion | Max | Score | Notes | -| ------------- | --: | ----: | --------------------------------------------------------------------------------------- | -| Input quality | 6 | 6 | Inputs are well-described, with defaults and validation | -| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover end-to-end script behavior | - -#### Overall — 65 / 100 - -
- -### `coder/code-server` - -| Presentation & Onboarding | IDE Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | -| ------------------------: | --------------: | -----------------: | ---------------------------: | ------------------: | -----------: | -| **25 / 25** | **19 / 25** | **16 / 20** | **3 / 20** | **10 / 10** | **73 / 100** | - -
-Drilldown - -#### Presentation & Onboarding — 25 / 25 - -| Criterion | Max | Score | Notes | -| --------------------------- | --: | ----: | -------------------------------------------------------------------------------------------------- | -| Configuration-mode examples | 12 | 12 | Good examples for version pinning, extensions, settings, workspace vs folder, offline/cached modes | -| Coder-context framing | 8 | 8 | Clearly explains code-server in a Coder workspace | -| Visual preview | 5 | 5 | README includes a screenshot | - -#### IDE Integration — 19 / 25 - -| Criterion | Max | Score | Notes | -| ------------------------------------------ | --: | ----: | -------------------------------------------------------------------------------------- | -| Dashboard entry point | 7 | 7 | Built-in `coder_app` with launch behavior controls | -| Managed configuration | 6 | 6 | Supports user and machine settings | -| Configurable folder or workdir | 6 | 6 | Supports both `folder` and `workspace` | -| Pre-installed extensions _(web IDEs only)_ | 6 | 0 | Supports extensions, but no documented private marketplace or enterprise-managed story | - -#### Credential Hygiene — 16 / 20 - -| Criterion | Max | Score | Notes | -| ------------------------ | --: | ----: | ------------------------------------------- | -| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | -| Non-hardcoded auth path | 4 | 0 | No relevant auth-path guidance | - -#### Restricted-Network Readiness — 3 / 20 - -| Criterion | Max | Score | Notes | -| -------------------------- | --: | ----: | ---------------------------------------------------------------------------------------- | -| Mirrorable artifact source | 10 | 0 | Installer source is not configurable for an internal mirror | -| Bring-your-own binary | 6 | 3 | Offline and cached modes help, but there is no first-class custom binary/source override | -| Egress transparency | 4 | 0 | External endpoints are not clearly documented | - -#### Engineering Quality — 10 / 10 - -| Criterion | Max | Score | Notes | -| ------------- | --: | ----: | ----------------------------------------------------------------------------------------------------- | -| Input quality | 6 | 6 | Inputs are well-described, with defaults and validation | -| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover script behavior and settings merge flows | - -#### Overall — 73 / 100 - -
- -### `coder/vscode-web` - -| Presentation & Onboarding | IDE Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | -| ------------------------: | --------------: | -----------------: | ---------------------------: | ------------------: | -----------: | -| **25 / 25** | **19 / 25** | **16 / 20** | **0 / 20** | **7 / 10** | **67 / 100** | - -
-Drilldown - -#### Presentation & Onboarding — 25 / 25 - -| Criterion | Max | Score | Notes | -| --------------------------- | --: | ----: | ----------------------------------------------------------------------------------------------- | -| Configuration-mode examples | 12 | 12 | Good examples for folder, workspace, extensions, settings, pinning, cached and offline behavior | -| Coder-context framing | 8 | 8 | Clearly explains VS Code Web in a Coder workspace | -| Visual preview | 5 | 5 | README includes a GIF | - -#### IDE Integration — 19 / 25 - -| Criterion | Max | Score | Notes | -| ------------------------------------------ | --: | ----: | -------------------------------------------------------------------------------------- | -| Dashboard entry point | 7 | 7 | Built-in `coder_app` with launch behavior controls | -| Managed configuration | 6 | 6 | Supports machine settings and trust-related config | -| Configurable folder or workdir | 6 | 6 | Supports both `folder` and `workspace` | -| Pre-installed extensions _(web IDEs only)_ | 6 | 0 | Supports extensions, but no documented private marketplace or enterprise-managed story | - -#### Credential Hygiene — 16 / 20 - -| Criterion | Max | Score | Notes | -| ------------------------ | --: | ----: | ------------------------------------------- | -| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | -| Non-hardcoded auth path | 4 | 0 | No relevant auth-path guidance | - -#### Restricted-Network Readiness — 0 / 20 - -| Criterion | Max | Score | Notes | -| -------------------------- | --: | ----: | -------------------------------------------------------------------------------------------------- | -| Mirrorable artifact source | 10 | 0 | Download source is not configurable for an internal mirror | -| Bring-your-own binary | 6 | 0 | Offline and cached behavior exist, but no documented bring-your-own binary or custom artifact path | -| Egress transparency | 4 | 0 | External endpoints are not clearly documented | - -#### Engineering Quality — 7 / 10 - -| Criterion | Max | Score | Notes | -| ------------- | --: | ----: | ------------------------------------------------------------------------------------------------------------- | -| Input quality | 6 | 3 | Inputs are mostly well-described, but some validation and polish are less complete than the strongest modules | -| Test coverage | 4 | 4 | `.tftest.hcl` covers plan-time rules, TypeScript tests cover script behavior and settings merge flows | - -#### Overall — 67 / 100 - -
- -### `coder/git-clone` (Utility) - -Utility module: no track section. Restricted-Network Readiness is not applicable, the module downloads nothing and clones a caller-controlled URL. Denominator is 55, normalized to 100. - -| Presentation & Onboarding | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | -| ------------------------: | -----------------: | ---------------------------: | ------------------: | -----------: | -| **20 / 25** | **20 / 20** | N/A | **8 / 10** | **87 / 100** | - -
-Drilldown - -#### Presentation & Onboarding — 20 / 25 - -| Criterion | Max | Score | Notes | -| --------------------------- | --: | ----: | -------------------------------------------------------------------------------------------- | -| Configuration-mode examples | 12 | 12 | Examples for custom path, branch clones, GitHub/GitLab, self-hosted providers, external auth | -| Coder-context framing | 8 | 8 | Clearly explains cloning into a Coder workspace and composing with other modules | -| Visual preview | 5 | 0 | No image, GIF, or video in README | - -#### Credential Hygiene — 20 / 20 - -| Criterion | Max | Score | Notes | -| ------------------------ | --: | ----: | -------------------------------------------------------------------------------- | -| Secrets marked sensitive | 16 | 16 | No secret inputs or secret-bearing examples | -| Non-hardcoded auth path | 4 | 4 | Documents Coder external auth (`coder_external_auth`) instead of embedded tokens | - -#### Restricted-Network Readiness — N/A - -| Criterion | Max | Score | Notes | -| -------------------------- | --: | ----: | ---------------------------------------------------------------------- | -| Mirrorable artifact source | — | N/A | Clone URL is fully caller-controlled, nothing to mirror | -| Bring-your-own binary | — | N/A | Module downloads nothing; relies on `git` already present in the image | -| Egress transparency | — | N/A | Only contacts the URL the caller provides | - -#### Engineering Quality — 8 / 10 - -| Criterion | Max | Score | Notes | -| ------------- | --: | ----: | -------------------------------------------------------------------------- | -| Input quality | 6 | 6 | Inputs are well-described with sensible defaults | -| Test coverage | 4 | 2 | Has TypeScript end-to-end tests, but no `.tftest.hcl` business-logic tests | - -#### Overall — 87 / 100 - -Raw 48 / 55 → `round(48 / 55 × 100)` = **87** - -
+Every scored module is listed in the pinned +[📊 Module Scorecards](https://github.com/coder/registry/discussions/1011) +discussion, which links each module's dedicated scorecard discussion. diff --git a/.github/scorecard/lib.ts b/.github/scorecard/lib.ts new file mode 100644 index 000000000..f39dff8a6 --- /dev/null +++ b/.github/scorecard/lib.ts @@ -0,0 +1,124 @@ +/** + * Shared constants, types, and helpers for the module scorecard scripts. + */ + +export const REPO_OWNER = "coder"; +export const REPO_NAME = "registry"; +export const REPO_ID = "R_kgDOOVbRAA"; // coder/registry +export const CATEGORY_ID = "DIC_kwDOOVbRAM4DBMLT"; // "Modules" category + +export const MARKER = ""; +export const INDEX_MARKER = ""; +export const INDEX_TITLE = "📊 Module Scorecards"; + +export interface SummaryScores { + track: "Agent" | "IDE" | "Utility"; + presentation: string; + integration: string; + credential: string; + network: string; + engineering: string; + overall: string; + overallNum: number; +} + +export interface Result extends SummaryScores { + module: string; + name: string; + url: string; + scoredAt: string; +} + +export async function graphql( + query: string, + variables: Record = {}, +): Promise { + const res = await fetch("https://api.github.com/graphql", { + method: "POST", + headers: { + Authorization: `Bearer ${process.env.GITHUB_DISCUSSIONS_TOKEN}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ query, variables }), + }); + const body = (await res.json()) as { data?: T; errors?: unknown[] }; + if (!res.ok || body.errors?.length) { + throw new Error( + `GitHub GraphQL error: ${JSON.stringify(body.errors ?? body)}`, + ); + } + return body.data!; +} + +// Parses the theme-level summary table out of scorecard markdown (either a +// raw scorecard or a discussion body containing one). +export function parseSummary(markdown: string): SummaryScores | null { + const lines = markdown.split("\n"); + const headerIdx = lines.findIndex( + (l) => + l.startsWith("|") && l.includes("Overall") && l.includes("Presentation"), + ); + if (headerIdx === -1) return null; + const headers = lines[headerIdx] + .split("|") + .map((c) => c.trim()) + .filter(Boolean); + const cells = lines[headerIdx + 2] + ?.split("|") + .map((c) => c.trim().replace(/\*\*/g, "")) + .filter(Boolean); + if (!cells || cells.length !== headers.length) return null; + const get = (name: string) => { + const i = headers.findIndex((h) => h.toLowerCase().includes(name)); + return i === -1 ? "—" : cells[i]; + }; + const integrationIdx = headers.findIndex((h) => /integration/i.test(h)); + const track: SummaryScores["track"] = + integrationIdx === -1 + ? "Utility" + : /agent/i.test(headers[integrationIdx]) + ? "Agent" + : "IDE"; + const overall = get("overall"); + return { + track, + presentation: get("presentation"), + integration: integrationIdx === -1 ? "—" : cells[integrationIdx], + credential: get("credential"), + network: get("restricted"), + engineering: get("engineering"), + overall, + overallNum: Number(overall.match(/(\d+)\s*\/\s*100/)?.[1] ?? 0), + }; +} + +export interface DiscussionRef { + id: string; + title: string; + url: string; + body: string; +} + +export async function findDiscussionByTitle( + title: string, +): Promise { + const q = `"${title}" in:title repo:${REPO_OWNER}/${REPO_NAME}`; + const data = await graphql<{ search: { nodes: DiscussionRef[] } }>( + ` + query ($q: String!) { + search(query: $q, type: DISCUSSION, first: 10) { + nodes { + ... on Discussion { + id + title + url + body + } + } + } + } + `, + { q }, + ); + return data.search.nodes.find((n) => n.title === title) ?? null; +} diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts index e6fc1f4c3..98212ce7d 100644 --- a/.github/scorecard/score-modules.ts +++ b/.github/scorecard/score-modules.ts @@ -16,89 +16,25 @@ import { readdir, readFile } from "node:fs/promises"; import { existsSync } from "node:fs"; import path from "node:path"; +import { + CATEGORY_ID, + MARKER, + REPO_ID, + REPO_NAME, + REPO_OWNER, + type SummaryScores, + findDiscussionByTitle, + graphql, + parseSummary, +} from "./lib"; const REGISTRY_ROOT = path.resolve(import.meta.dir, "..", ".."); const MODULES_DIR = path.join(REGISTRY_ROOT, "registry", "coder", "modules"); const SCORECARD_PATH = path.join(import.meta.dir, "SCORECARD.md"); -const REPO_ID = "R_kgDOOVbRAA"; // coder/registry -const CATEGORY_ID = "DIC_kwDOOVbRAM4DBMLT"; // "Modules" category -const REPO_OWNER = "coder"; -const REPO_NAME = "registry"; - const ANTHROPIC_MODEL = process.env.ANTHROPIC_MODEL ?? "claude-sonnet-4-5"; const MAX_FILE_BYTES = 30_000; -const MARKER = ""; -const RESULTS_PATH = path.join(import.meta.dir, ".scorecard-results.json"); - -interface SummaryScores { - track: "Agent" | "IDE" | "Utility"; - presentation: string; - integration: string; - credential: string; - network: string; - engineering: string; - overall: string; - overallNum: number; -} - -interface Result extends SummaryScores { - module: string; - name: string; - url: string; - scoredAt: string; -} - -function parseSummary(scorecard: string): SummaryScores | null { - const lines = scorecard.split("\n"); - const headerIdx = lines.findIndex( - (l) => - l.startsWith("|") && l.includes("Overall") && l.includes("Presentation"), - ); - if (headerIdx === -1) return null; - const headers = lines[headerIdx] - .split("|") - .map((c) => c.trim()) - .filter(Boolean); - const cells = lines[headerIdx + 2] - ?.split("|") - .map((c) => c.trim().replace(/\*\*/g, "")) - .filter(Boolean); - if (!cells || cells.length !== headers.length) return null; - const get = (name: string) => { - const i = headers.findIndex((h) => h.toLowerCase().includes(name)); - return i === -1 ? "\u2014" : cells[i]; - }; - const integrationIdx = headers.findIndex((h) => /integration/i.test(h)); - const track: SummaryScores["track"] = - integrationIdx === -1 - ? "Utility" - : /agent/i.test(headers[integrationIdx]) - ? "Agent" - : "IDE"; - const overall = get("overall"); - return { - track, - presentation: get("presentation"), - integration: integrationIdx === -1 ? "\u2014" : cells[integrationIdx], - credential: get("credential"), - network: get("restricted"), - engineering: get("engineering"), - overall, - overallNum: Number(overall.match(/(\d+)\s*\/\s*100/)?.[1] ?? 0), - }; -} - -async function saveResult(result: Result): Promise { - let results: Record = {}; - if (existsSync(RESULTS_PATH)) { - results = JSON.parse(await readFile(RESULTS_PATH, "utf8")); - } - results[result.module] = result; - await Bun.write(RESULTS_PATH, JSON.stringify(results, null, 2) + "\n"); -} - async function displayName(moduleName: string): Promise { const readme = await readFile( path.join(MODULES_DIR, moduleName, "README.md"), @@ -300,60 +236,11 @@ Do not add any prose before or after the scorecard.`; return fixOverall(text.trim()); } -async function graphql( - query: string, - variables: Record, -): Promise { - const res = await fetch("https://api.github.com/graphql", { - method: "POST", - headers: { - Authorization: `Bearer ${process.env.GITHUB_DISCUSSIONS_TOKEN}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ query, variables }), - }); - const body = (await res.json()) as { data?: T; errors?: unknown[] }; - if (!res.ok || body.errors?.length) { - throw new Error( - `GitHub GraphQL error: ${JSON.stringify(body.errors ?? body)}`, - ); - } - return body.data!; -} - -async function findDiscussion( - title: string, -): Promise<{ id: string; url: string; body: string } | null> { - const q = `"${title}" in:title repo:${REPO_OWNER}/${REPO_NAME}`; - const data = await graphql<{ - search: { - nodes: { id: string; title: string; url: string; body: string }[]; - }; - }>( - ` - query ($q: String!) { - search(query: $q, type: DISCUSSION, first: 10) { - nodes { - ... on Discussion { - id - title - url - body - } - } - } - } - `, - { q }, - ); - return data.search.nodes.find((n) => n.title === title) ?? null; -} - async function upsertDiscussion( title: string, body: string, ): Promise<{ id: string; url: string; created: boolean }> { - const existing = await findDiscussion(title); + const existing = await findDiscussionByTitle(title); if (existing) { await graphql( ` @@ -493,7 +380,7 @@ async function main() { // PR mode: compare against the module's current discussion and // build a report instead of touching any discussion. const summary = parseSummary(scorecard); - const existing = await findDiscussion(`${name} module`); + const existing = await findDiscussionByTitle(`${name} module`); const parsed = existing ? parseSummary(existing.body) : null; const baseline = existing && parsed ? { ...parsed, url: existing.url } : null; @@ -507,18 +394,6 @@ async function main() { `${name} module`, discussionBody(mod, name, scorecard), ); - const summary = parseSummary(scorecard); - if (summary) { - await saveResult({ - module: mod, - name, - url, - scoredAt: new Date().toISOString(), - ...summary, - }); - } else { - process.stderr.write(`warn: could not parse summary for ${mod}\n`); - } process.stderr.write(`${created ? "created" : "updated"} ${url}\n`); } catch (err) { process.stderr.write(`FAILED: ${err}\n`); diff --git a/.github/scorecard/scorecard-index.ts b/.github/scorecard/scorecard-index.ts index 395c32b75..d4effe2f4 100644 --- a/.github/scorecard/scorecard-index.ts +++ b/.github/scorecard/scorecard-index.ts @@ -1,68 +1,41 @@ #!/usr/bin/env bun /** * Builds (or updates) the pinned index discussion listing every module - * scorecard with per-theme scores, from local results written by - * score-modules.ts (.scorecard-results.json). + * scorecard with per-theme scores, grouped by track. Results are read back + * from the module scorecard discussions, so partial scoring runs still + * produce a complete index. * * Env (required): * GITHUB_DISCUSSIONS_TOKEN GitHub PAT with Discussions read/write * * Usage: - * bun run scorecard-index.ts [--dry-run] [--downloads downloads.json] [--from-github] - * - * --from-github: one-time backfill of .scorecard-results.json from existing - * discussions (for runs made before local result tracking existed). - * downloads.json (optional): {"claude-code": 12345, ...} module dir -> count. - * When provided, the table is ranked by downloads; otherwise by overall score. + * bun run scorecard-index.ts [--dry-run] */ -import { readFile } from "node:fs/promises"; -import { existsSync } from "node:fs"; -import path from "node:path"; - -const RESULTS_PATH = path.join(import.meta.dir, ".scorecard-results.json"); - -const REPO_ID = "R_kgDOOVbRAA"; // coder/registry -const CATEGORY_ID = "DIC_kwDOOVbRAM4DBMLT"; // "Modules" -const MARKER = ""; -const INDEX_MARKER = ""; -const INDEX_TITLE = "📊 Module Scorecards"; - -interface Result { - module: string; - name: string; - url: string; - scoredAt: string; - track: "Agent" | "IDE" | "Utility"; - presentation: string; - integration: string; - credential: string; - network: string; - engineering: string; - overall: string; - overallNum: number; -} +import { + CATEGORY_ID, + INDEX_MARKER, + INDEX_TITLE, + MARKER, + REPO_ID, + type Result, + findDiscussionByTitle, + graphql, + parseSummary, +} from "./lib"; interface Args { dryRun: boolean; - downloadsPath?: string; - fromGithub: boolean; } function parseArgs(): Args { - const args: Args = { dryRun: false, fromGithub: false }; + const args: Args = { dryRun: false }; const argv = process.argv.slice(2); for (let i = 0; i < argv.length; i++) { switch (argv[i]) { case "--dry-run": args.dryRun = true; break; - case "--downloads": - args.downloadsPath = argv[++i]; - break; - case "--from-github": - args.fromGithub = true; - break; default: console.error(`Unknown argument: ${argv[i]}`); process.exit(1); @@ -71,79 +44,7 @@ function parseArgs(): Args { return args; } -async function graphql( - query: string, - variables: Record = {}, -): Promise { - const res = await fetch("https://api.github.com/graphql", { - method: "POST", - headers: { - Authorization: `Bearer ${process.env.GITHUB_DISCUSSIONS_TOKEN}`, - "Content-Type": "application/json", - }, - body: JSON.stringify({ query, variables }), - }); - const body = (await res.json()) as { data?: T; errors?: unknown[] }; - if (!res.ok || body.errors?.length) { - throw new Error( - `GitHub GraphQL error: ${JSON.stringify(body.errors ?? body)}`, - ); - } - return body.data!; -} - -function parseSummaryFromBody( - title: string, - url: string, - body: string, -): Result | null { - const dirMatch = body.match( - /registry\.coder\.com\/modules\/coder\/([a-z0-9-]+)/, - ); - const lines = body.split("\n"); - const headerIdx = lines.findIndex( - (l) => - l.startsWith("|") && l.includes("Overall") && l.includes("Presentation"), - ); - if (headerIdx === -1) return null; - const headers = lines[headerIdx] - .split("|") - .map((c) => c.trim()) - .filter(Boolean); - const cells = lines[headerIdx + 2] - ?.split("|") - .map((c) => c.trim().replace(/\*\*/g, "")) - .filter(Boolean); - if (!cells || cells.length !== headers.length) return null; - const get = (name: string) => { - const i = headers.findIndex((h) => h.toLowerCase().includes(name)); - return i === -1 ? "—" : cells[i]; - }; - const integrationIdx = headers.findIndex((h) => /integration/i.test(h)); - const track: Result["track"] = - integrationIdx === -1 - ? "Utility" - : /agent/i.test(headers[integrationIdx]) - ? "Agent" - : "IDE"; - const overall = get("overall"); - return { - module: dirMatch?.[1] ?? "", - name: title.replace(/ module$/, ""), - url, - scoredAt: new Date().toISOString(), - track, - presentation: get("presentation"), - integration: integrationIdx === -1 ? "—" : cells[integrationIdx], - credential: get("credential"), - network: get("restricted"), - engineering: get("engineering"), - overall, - overallNum: Number(overall.match(/(\d+)\s*\/\s*100/)?.[1] ?? 0), - }; -} - -async function backfillFromGithub(): Promise> { +async function fetchResults(): Promise> { const results: Record = {}; let cursor: string | null = null; for (;;) { @@ -167,35 +68,30 @@ async function backfillFromGithub(): Promise> { ); for (const d of data.repository.discussions.nodes) { if (!d.body.includes(MARKER) || d.body.includes(INDEX_MARKER)) continue; - const r = parseSummaryFromBody(d.title, d.url, d.body); - if (r?.module) results[r.module] = r; - else console.error(`warn: could not parse "${d.title}"`); + const summary = parseSummary(d.body); + const module = d.body.match( + /registry\.coder\.com\/modules\/coder\/([a-z0-9-]+)/, + )?.[1]; + if (!summary || !module) { + console.error(`warn: could not parse "${d.title}"`); + continue; + } + results[module] = { + ...summary, + module, + name: d.title.replace(/ module$/, ""), + url: d.url, + scoredAt: new Date().toISOString(), + }; } if (!data.repository.discussions.pageInfo.hasNextPage) break; cursor = data.repository.discussions.pageInfo.endCursor; } - await Bun.write(RESULTS_PATH, JSON.stringify(results, null, 2) + "\n"); - console.error( - `backfilled ${Object.keys(results).length} results to ${RESULTS_PATH}`, - ); return results; } -function buildBody( - results: Record, - downloads?: Record, -): string { - const rank = (a: Result, b: Result) => { - if (downloads) { - const da = downloads[a.module] ?? 0; - const db = downloads[b.module] ?? 0; - if (db !== da) return db - da; - } - return b.overallNum - a.overallNum; - }; - - const dlHeader = downloads ? " Downloads |" : ""; - const dlSep = downloads ? "---:|" : ""; +function buildBody(results: Record): string { + const rank = (a: Result, b: Result) => b.overallNum - a.overallNum; const section = ( title: string, @@ -209,19 +105,16 @@ function buildBody( const body = rows .sort(rank) .map((r) => { - const dl = downloads - ? ` ${(downloads[r.module] ?? 0).toLocaleString()} |` - : ""; const integration = withIntegration ? ` ${r.integration} |` : ""; - return `| [${r.name}](${r.url}) |${dl} ${r.presentation} |${integration} ${r.credential} | ${r.network} | ${r.engineering} | **${r.overall}** |`; + return `| [${r.name}](${r.url}) | ${r.presentation} |${integration} ${r.credential} | ${r.network} | ${r.engineering} | **${r.overall}** |`; }) .join("\n"); return `### ${title} modules ${blurb} -| Module |${dlHeader} Presentation & Onboarding |${integrationHeader} Credential Hygiene | Restricted-Network | Engineering Quality | Overall | -|---|${dlSep}---:|${integrationSep}---:|---:|---:|---:| +| Module | Presentation & Onboarding |${integrationHeader} Credential Hygiene | Restricted-Network | Engineering Quality | Overall | +|---|---:|${integrationSep}---:|---:|---:|---:| ${body} `; }; @@ -264,25 +157,7 @@ ${INDEX_MARKER}`; } async function upsertIndex(body: string): Promise { - const data = await graphql<{ - search: { nodes: { id: string; title: string; url: string }[] }; - }>( - ` - query ($q: String!) { - search(query: $q, type: DISCUSSION, first: 10) { - nodes { - ... on Discussion { - id - title - url - } - } - } - } - `, - { q: `"${INDEX_TITLE}" in:title repo:coder/registry` }, - ); - const existing = data.search.nodes.find((n) => n.title === INDEX_TITLE); + const existing = await findDiscussionByTitle(INDEX_TITLE); if (existing) { await graphql( ` @@ -327,22 +202,9 @@ async function main() { if (!process.env.GITHUB_DISCUSSIONS_TOKEN) { throw new Error("GITHUB_DISCUSSIONS_TOKEN not set"); } - let results: Record; - if (args.fromGithub) { - results = await backfillFromGithub(); - } else { - if (!existsSync(RESULTS_PATH)) { - throw new Error( - `${RESULTS_PATH} not found. Run score-modules.ts first, or use --from-github to backfill.`, - ); - } - results = JSON.parse(await readFile(RESULTS_PATH, "utf8")); - } - let downloads: Record | undefined; - if (args.downloadsPath) { - downloads = JSON.parse(await readFile(args.downloadsPath, "utf8")); - } - const body = buildBody(results, downloads); + const results = await fetchResults(); + console.error(`found ${Object.keys(results).length} module scorecards`); + const body = buildBody(results); if (args.dryRun) { console.log(body); return; diff --git a/.github/workflows/module-scorecards.yaml b/.github/workflows/module-scorecards.yaml index 9e773c522..fe737e2dc 100644 --- a/.github/workflows/module-scorecards.yaml +++ b/.github/workflows/module-scorecards.yaml @@ -92,6 +92,6 @@ jobs: - name: Update index discussion env: GITHUB_DISCUSSIONS_TOKEN: ${{ secrets.SCORECARD_DISCUSSIONS_TOKEN }} - # --from-github rebuilds the full result set from the module - # discussions, so a single-module run still produces a complete index. - run: bun run .github/scorecard/scorecard-index.ts --from-github + # The index reads scores back from the module discussions, so a + # single-module run still produces a complete index. + run: bun run .github/scorecard/scorecard-index.ts From ba8635390344dd3127c22b51d86513c9c7eace1b Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 16:59:15 +0000 Subject: [PATCH 6/8] feat(.github/scorecard): flag display_name renames in the PR check --- .github/scorecard/lib.ts | 32 ++++++++++++++++++++++++++++++ .github/scorecard/score-modules.ts | 17 ++++++++++++++-- 2 files changed, 47 insertions(+), 2 deletions(-) diff --git a/.github/scorecard/lib.ts b/.github/scorecard/lib.ts index f39dff8a6..774baf549 100644 --- a/.github/scorecard/lib.ts +++ b/.github/scorecard/lib.ts @@ -122,3 +122,35 @@ export async function findDiscussionByTitle( ); return data.search.nodes.find((n) => n.title === title) ?? null; } + +// Finds a module's scorecard discussion by the registry URL in its body, +// regardless of title. Used to detect display_name renames, where the +// title lookup misses but the discussion still exists. +export async function findDiscussionByModuleUrl( + module: string, +): Promise { + const url = `registry.coder.com/modules/coder/${module}`; + const q = `"${url}" in:body repo:${REPO_OWNER}/${REPO_NAME}`; + const data = await graphql<{ search: { nodes: DiscussionRef[] } }>( + ` + query ($q: String!) { + search(query: $q, type: DISCUSSION, first: 10) { + nodes { + ... on Discussion { + id + title + url + body + } + } + } + } + `, + { q }, + ); + return ( + data.search.nodes.find( + (n) => n.body.includes(MARKER) && n.body.includes(`/${module})`), + ) ?? null + ); +} diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts index 98212ce7d..f9b8b21a1 100644 --- a/.github/scorecard/score-modules.ts +++ b/.github/scorecard/score-modules.ts @@ -23,6 +23,7 @@ import { REPO_NAME, REPO_OWNER, type SummaryScores, + findDiscussionByModuleUrl, findDiscussionByTitle, graphql, parseSummary, @@ -380,12 +381,24 @@ async function main() { // PR mode: compare against the module's current discussion and // build a report instead of touching any discussion. const summary = parseSummary(scorecard); - const existing = await findDiscussionByTitle(`${name} module`); + let existing = await findDiscussionByTitle(`${name} module`); + let renameWarning = ""; + if (!existing) { + // A display_name rename breaks the title lookup while the old + // discussion still exists. Fall back to the registry URL and + // flag it so a reviewer retitles or deletes the old one; the + // post-merge run creates a fresh discussion under the new name. + existing = await findDiscussionByModuleUrl(mod); + if (existing) { + renameWarning = `\n\n> [!WARNING]\n> This module's display name appears to have changed. Its existing scorecard discussion is titled "${existing.title}" (${existing.url}). After merge, a new discussion named "${name} module" will be created; a maintainer should retitle or delete the old one to avoid duplicates.`; + } + } const parsed = existing ? parseSummary(existing.body) : null; const baseline = existing && parsed ? { ...parsed, url: existing.url } : null; prSections.push( - prReportSection(mod, name, scorecard, summary, baseline), + prReportSection(mod, name, scorecard, summary, baseline) + + renameWarning, ); process.stderr.write("compared\n"); continue; From 2d555d5f5d6be87fb10eaf5ddb557c463964adc6 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 17:00:42 +0000 Subject: [PATCH 7/8] fix(.github/scorecard): make module URL lookup survive GitHub search tokenization --- .github/scorecard/lib.ts | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/.github/scorecard/lib.ts b/.github/scorecard/lib.ts index 774baf549..5c026284f 100644 --- a/.github/scorecard/lib.ts +++ b/.github/scorecard/lib.ts @@ -125,16 +125,16 @@ export async function findDiscussionByTitle( // Finds a module's scorecard discussion by the registry URL in its body, // regardless of title. Used to detect display_name renames, where the -// title lookup misses but the discussion still exists. +// title lookup misses but the discussion still exists. GitHub search +// tokenizes URLs, so search the module slug and filter by the exact URL. export async function findDiscussionByModuleUrl( module: string, ): Promise { - const url = `registry.coder.com/modules/coder/${module}`; - const q = `"${url}" in:body repo:${REPO_OWNER}/${REPO_NAME}`; + const q = `${module} in:body repo:${REPO_OWNER}/${REPO_NAME}`; const data = await graphql<{ search: { nodes: DiscussionRef[] } }>( ` query ($q: String!) { - search(query: $q, type: DISCUSSION, first: 10) { + search(query: $q, type: DISCUSSION, first: 20) { nodes { ... on Discussion { id @@ -150,7 +150,10 @@ export async function findDiscussionByModuleUrl( ); return ( data.search.nodes.find( - (n) => n.body.includes(MARKER) && n.body.includes(`/${module})`), + (n) => + n.body.includes(MARKER) && + !n.body.includes(INDEX_MARKER) && + n.body.includes(`registry.coder.com/modules/coder/${module})`), ) ?? null ); } From 9e66b76847a2fdbfb850841e3db8a8123be46a70 Mon Sep 17 00:00:00 2001 From: Ben Potter Date: Tue, 14 Jul 2026 17:06:56 +0000 Subject: [PATCH 8/8] feat(.github/scorecard): explain the scorecard's purpose in the index intro --- .github/scorecard/scorecard-index.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/scorecard/scorecard-index.ts b/.github/scorecard/scorecard-index.ts index d4effe2f4..768293c02 100644 --- a/.github/scorecard/scorecard-index.ts +++ b/.github/scorecard/scorecard-index.ts @@ -146,6 +146,8 @@ ${body} const now = new Date().toISOString().slice(0, 10); return `Every module in the \`coder\` namespace, scored against [SCORECARD.md](https://github.com/coder/registry/blob/main/.github/scorecard/SCORECARD.md). Each module links to its dedicated discussion, where you can share thoughts, questions, and feedback. +**Why a scorecard?** It gives us a shared definition of quality (and a goalpost) for modules: what we care about for each one, in one place, for any contributor. It is not a strict gate today, though it may become one, and the criteria will evolve over time. It also helps catch regressions, for example a feature or documented capability being removed, since scores are re-evaluated whenever a module changes. + **Looking for a way to contribute?** Low scores are contribution opportunities. Pick a module, open its discussion to see exactly which criteria it misses (visual previews, air-gapped install docs, tests, session persistence, and so on), and open a PR against [\`registry/coder/modules/\`](https://github.com/coder/registry/tree/main/registry/coder/modules). ${sections}