docs: name the prior art on token measurement, and state the estimator's real error

README.md

@@ -34,7 +34,9 @@ compiled 1 skill for 3 targets
 
 The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target.
 
-Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number.
+Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you.
+
+Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer. Measured against `o200k_base` it is off by anywhere from 7% low to 36% high depending on what the skill contains, and it understates the gap on the 47× fixture, which is really 68×. The per-target *story* survives that error; the exact figures should be read as estimates. `npm run tokencheck` regenerates the comparison, and [the benchmark](docs/benchmarks/README.md) carries it.)
 
 Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures.
 

docs/benchmarks/README.md

@@ -11,7 +11,7 @@ Two costs matter:
 
 One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in.
 
-Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`.
+Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (`prereview`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered `review-checklist` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and `--strict` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: `prereview`'s 14x is **14.7x** under a real tokenizer, and `review-checklist`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. Regenerate that comparison with `npm run tokencheck` inside `packages/cli`: it measures the same artifacts this script emits against `o200k_base` and prints the range and both ratios. The tokenizer is a devDependency and runs in no shipped code path — kitbash's runtime dependency count is still zero. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`.
 
 ## `prereview` — manifested (budget 1500, lazy)
 

packages/cli/package.json

@@ -24,10 +24,12 @@
     "dev": "node --experimental-strip-types src/index.ts",
     "test": "npm run build && node scripts/test.mjs",
     "bench": "npm run build && node scripts/benchmark.mjs",
-    "prepublishOnly": "npm test"
+    "prepublishOnly": "npm test",
+    "tokencheck": "npm run build && node scripts/tokenizer-check.mjs"
   },
   "devDependencies": {
     "@types/node": "^26.1.1",
+    "gpt-tokenizer": "^4.0.0",
     "typescript": "^5.6.0"
   },
   "keywords": [

packages/cli/scripts/bench-fixtures.mjs

@@ -0,0 +1,117 @@
+/**
+ * The benchmark corpus, built once and shared.
+ *
+ * Both `benchmark.mjs` (which publishes the token table) and
+ * `tokenizer-check.mjs` (which measures how wrong the estimator is) have to
+ * look at *the same bytes*, or the error bar the table quotes is an error bar
+ * for some other corpus. Extracted here so there is one definition of the
+ * fixtures and one extractor per target, not two that drift.
+ */
+import { spawnSync } from "node:child_process";
+import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { dirname, join, resolve } from "node:path";
+import { fileURLToPath } from "node:url";
+import { standingStub } from "../dist/ksf.js";
+import { ADAPTERS } from "../dist/adapters.js";
+
+const here = dirname(fileURLToPath(import.meta.url));
+export const repoRoot = resolve(here, "../../..");
+const cli = join(here, "../dist/index.js");
+
+// agent-plugins is an opt-in publishing target (agent-plugins.org): it is not part
+// of a repo's auto-detected fan-out and would not fire in this fixture's compile
+// (no plugin.json). It is lazy, so it carries the same stub cost as every other
+// lazy target and adds no standing-tax story — this benchmark measures the
+// always-on tax across the targets a repo compiles to by default, so it is excluded.
+const BENCH_ADAPTERS = ADAPTERS.filter((a) => a.id !== "agent-plugins");
+
+// How each target loads a skill, read from the adapters themselves rather than
+// restated here — a second copy of this map is exactly how the published numbers
+// drift away from what the compiler actually emits.
+export const LOADING = Object.fromEntries(BENCH_ADAPTERS.map((a) => [a.id, a.loading]));
+
+export const SKILLS = [
+  { name: "prereview", kind: "manifested (budget 1500, lazy)", standing: 60 },
+  { name: "review-checklist", kind: "bare / unmanifested (no budget)", standing: null },
+];
+
+function run(args, cwd) {
+  const r = spawnSync("node", [cli, ...args], { cwd, encoding: "utf8" });
+  if (r.status !== 0) throw new Error(`kitbash ${args.join(" ")} failed:\n${r.stdout}${r.stderr}`);
+  return `${r.stdout}${r.stderr}`;
+}
+
+/** The compiled text of one skill's contribution to a target's output. */
+export function artifactText(tmp, target, skillName) {
+  const read = (rel) => readFileSync(join(tmp, rel), "utf8");
+  switch (target) {
+    case "claude-code":
+      return read(`.claude/skills/${skillName}/SKILL.md`);
+    case "cursor":
+      return read(`.cursor/rules/${skillName}.mdc`);
+    // Every one of these is served by the vendor-neutral path and reads the same
+    // bytes, so they measure identically. Zed and cline compile there directly;
+    // copilot and gemini also read it, so their own skills directory is not
+    // written when it is present (see VENDOR_NEUTRAL_ALIASES). Listed separately
+    // rather than folded together, because a reader looking up "what does Zed
+    // cost" must find a row.
+    case "agents":
+    case "zed":
+    case "cline":
+    case "copilot":
+    case "gemini":
+      return read(`.agents/skills/${skillName}/SKILL.md`);
+    case "windsurf":
+      return read(`.windsurf/rules/${skillName}.md`);
+    case "aider":
+    case "agentsmd": {
+      const file = target === "aider" ? "CONVENTIONS.md" : "AGENTS.md";
+      const m = read(file).match(new RegExp(`<!-- kitbash:begin ${skillName} -->[\\s\\S]*?<!-- kitbash:end ${skillName} -->`));
+      return m ? m[0] : "";
+    }
+    default:
+      return "";
+  }
+}
+
+/** The standing stub an agent keeps in context for a lazy target. */
+export function stubText(tmp, skillName) {
+  const body = readFileSync(join(tmp, ".kitbash/skills", skillName, "SKILL.md"), "utf8").replace(/^---[\s\S]*?---\n/, "");
+  return standingStub(body);
+}
+
+/**
+ * Build the corpus in a temp repo with every target present so every adapter
+ * fires, hand it to `fn`, and clean up. `fn` receives the workspace path.
+ */
+export function withBenchWorkspace(fn) {
+  const tmp = mkdtempSync(join(tmpdir(), "kitbash-bench-"));
+  try {
+    for (const d of [".claude", ".cursor", ".agents", ".zed", ".clinerules", ".windsurf", ".github"]) mkdirSync(join(tmp, d));
+    writeFileSync(join(tmp, "GEMINI.md"), "");
+    writeFileSync(join(tmp, "CONVENTIONS.md"), "");
+
+    run(["init"], tmp);
+
+    // 1) A real manifested skill (budget 1500, lazy disclosure).
+    run(["install", `file:${join(repoRoot, "examples/skills/prereview")}`], tmp);
+
+    // 2) A bare SKILL.md-only skill — the skills.sh / Claude Skills convention,
+    //    which has no manifest and so no declared budget. Sized to a realistic
+    //    mid-size community skill.
+    const bare = join(tmp, "bare");
+    mkdirSync(bare);
+    const bareBody =
+      "Enforce this project's code-review checklist on every diff before it merges.\n\n" +
+      Array.from({ length: 40 }, (_, i) => `- Rule ${i + 1}: check the diff for issue class ${i + 1} and cite the exact line and the fix.`).join("\n") +
+      "\n";
+    writeFileSync(join(bare, "SKILL.md"), `---\nname: review-checklist\ndescription: Enforce the team code-review checklist\n---\n\n${bareBody}`);
+    run(["install", `file:${bare}`], tmp);
+
+    run(["compile"], tmp);
+    return fn(tmp);
+  } finally {
+    rmSync(tmp, { recursive: true, force: true });
+  }
+}