diff --git a/README.md b/README.md index a420ca9..478cf42 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,9 @@ compiled 1 skill for 3 targets The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target. -Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number. +Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. + +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer. Measured against `o200k_base` it is off by anywhere from 7% low to 36% high depending on what the skill contains, and it understates the gap on the 47× fixture, which is really 68×. The per-target *story* survives that error; the exact figures should be read as estimates. `npm run tokencheck` regenerates the comparison, and [the benchmark](docs/benchmarks/README.md) carries it.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 089aad5..65cfef0 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (`prereview`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered `review-checklist` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and `--strict` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: `prereview`'s 14x is **14.7x** under a real tokenizer, and `review-checklist`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. Regenerate that comparison with `npm run tokencheck` inside `packages/cli`: it measures the same artifacts this script emits against `o200k_base` and prints the range and both ratios. The tokenizer is a devDependency and runs in no shipped code path — kitbash's runtime dependency count is still zero. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/package-lock.json b/packages/cli/package-lock.json index 3760e86..6e1efd2 100644 --- a/packages/cli/package-lock.json +++ b/packages/cli/package-lock.json @@ -1,18 +1,19 @@ { "name": "kitbash", - "version": "0.16.0", + "version": "0.24.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "kitbash", - "version": "0.16.0", + "version": "0.24.1", "license": "Apache-2.0", "bin": { "kitbash": "dist/index.js" }, "devDependencies": { "@types/node": "^26.1.1", + "gpt-tokenizer": "^4.0.0", "typescript": "^5.6.0" }, "engines": { @@ -29,6 +30,13 @@ "undici-types": "~8.3.0" } }, + "node_modules/gpt-tokenizer": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/gpt-tokenizer/-/gpt-tokenizer-4.0.0.tgz", + "integrity": "sha512-YAWIyzvuVUHEfW7tFfFAxH8qQb+Q3RU9nYOTy7skMNX5qzU6Q8jxTHZLyO56ug1vYvCR7wndzpd3jwD86/mhjQ==", + "dev": true, + "license": "MIT" + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", diff --git a/packages/cli/package.json b/packages/cli/package.json index b24e262..c87d300 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -24,10 +24,12 @@ "dev": "node --experimental-strip-types src/index.ts", "test": "npm run build && node scripts/test.mjs", "bench": "npm run build && node scripts/benchmark.mjs", - "prepublishOnly": "npm test" + "prepublishOnly": "npm test", + "tokencheck": "npm run build && node scripts/tokenizer-check.mjs" }, "devDependencies": { "@types/node": "^26.1.1", + "gpt-tokenizer": "^4.0.0", "typescript": "^5.6.0" }, "keywords": [ diff --git a/packages/cli/scripts/bench-fixtures.mjs b/packages/cli/scripts/bench-fixtures.mjs new file mode 100644 index 0000000..23a485a --- /dev/null +++ b/packages/cli/scripts/bench-fixtures.mjs @@ -0,0 +1,117 @@ +/** + * The benchmark corpus, built once and shared. + * + * Both `benchmark.mjs` (which publishes the token table) and + * `tokenizer-check.mjs` (which measures how wrong the estimator is) have to + * look at *the same bytes*, or the error bar the table quotes is an error bar + * for some other corpus. Extracted here so there is one definition of the + * fixtures and one extractor per target, not two that drift. + */ +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { standingStub } from "../dist/ksf.js"; +import { ADAPTERS } from "../dist/adapters.js"; + +const here = dirname(fileURLToPath(import.meta.url)); +export const repoRoot = resolve(here, "../../.."); +const cli = join(here, "../dist/index.js"); + +// agent-plugins is an opt-in publishing target (agent-plugins.org): it is not part +// of a repo's auto-detected fan-out and would not fire in this fixture's compile +// (no plugin.json). It is lazy, so it carries the same stub cost as every other +// lazy target and adds no standing-tax story — this benchmark measures the +// always-on tax across the targets a repo compiles to by default, so it is excluded. +const BENCH_ADAPTERS = ADAPTERS.filter((a) => a.id !== "agent-plugins"); + +// How each target loads a skill, read from the adapters themselves rather than +// restated here — a second copy of this map is exactly how the published numbers +// drift away from what the compiler actually emits. +export const LOADING = Object.fromEntries(BENCH_ADAPTERS.map((a) => [a.id, a.loading])); + +export const SKILLS = [ + { name: "prereview", kind: "manifested (budget 1500, lazy)", standing: 60 }, + { name: "review-checklist", kind: "bare / unmanifested (no budget)", standing: null }, +]; + +function run(args, cwd) { + const r = spawnSync("node", [cli, ...args], { cwd, encoding: "utf8" }); + if (r.status !== 0) throw new Error(`kitbash ${args.join(" ")} failed:\n${r.stdout}${r.stderr}`); + return `${r.stdout}${r.stderr}`; +} + +/** The compiled text of one skill's contribution to a target's output. */ +export function artifactText(tmp, target, skillName) { + const read = (rel) => readFileSync(join(tmp, rel), "utf8"); + switch (target) { + case "claude-code": + return read(`.claude/skills/${skillName}/SKILL.md`); + case "cursor": + return read(`.cursor/rules/${skillName}.mdc`); + // Every one of these is served by the vendor-neutral path and reads the same + // bytes, so they measure identically. Zed and cline compile there directly; + // copilot and gemini also read it, so their own skills directory is not + // written when it is present (see VENDOR_NEUTRAL_ALIASES). Listed separately + // rather than folded together, because a reader looking up "what does Zed + // cost" must find a row. + case "agents": + case "zed": + case "cline": + case "copilot": + case "gemini": + return read(`.agents/skills/${skillName}/SKILL.md`); + case "windsurf": + return read(`.windsurf/rules/${skillName}.md`); + case "aider": + case "agentsmd": { + const file = target === "aider" ? "CONVENTIONS.md" : "AGENTS.md"; + const m = read(file).match(new RegExp(`[\\s\\S]*?`)); + return m ? m[0] : ""; + } + default: + return ""; + } +} + +/** The standing stub an agent keeps in context for a lazy target. */ +export function stubText(tmp, skillName) { + const body = readFileSync(join(tmp, ".kitbash/skills", skillName, "SKILL.md"), "utf8").replace(/^---[\s\S]*?---\n/, ""); + return standingStub(body); +} + +/** + * Build the corpus in a temp repo with every target present so every adapter + * fires, hand it to `fn`, and clean up. `fn` receives the workspace path. + */ +export function withBenchWorkspace(fn) { + const tmp = mkdtempSync(join(tmpdir(), "kitbash-bench-")); + try { + for (const d of [".claude", ".cursor", ".agents", ".zed", ".clinerules", ".windsurf", ".github"]) mkdirSync(join(tmp, d)); + writeFileSync(join(tmp, "GEMINI.md"), ""); + writeFileSync(join(tmp, "CONVENTIONS.md"), ""); + + run(["init"], tmp); + + // 1) A real manifested skill (budget 1500, lazy disclosure). + run(["install", `file:${join(repoRoot, "examples/skills/prereview")}`], tmp); + + // 2) A bare SKILL.md-only skill — the skills.sh / Claude Skills convention, + // which has no manifest and so no declared budget. Sized to a realistic + // mid-size community skill. + const bare = join(tmp, "bare"); + mkdirSync(bare); + const bareBody = + "Enforce this project's code-review checklist on every diff before it merges.\n\n" + + Array.from({ length: 40 }, (_, i) => `- Rule ${i + 1}: check the diff for issue class ${i + 1} and cite the exact line and the fix.`).join("\n") + + "\n"; + writeFileSync(join(bare, "SKILL.md"), `---\nname: review-checklist\ndescription: Enforce the team code-review checklist\n---\n\n${bareBody}`); + run(["install", `file:${bare}`], tmp); + + run(["compile"], tmp); + return fn(tmp); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +} diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 3aed266..08b73bf 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -9,111 +9,25 @@ * standing stub on lazy targets and its whole body on eager ones. That gap is * the hidden tax of hand-maintaining a copy per agent. Here it is, in numbers. * + * The corpus lives in bench-fixtures.mjs, shared with tokenizer-check.mjs so the + * table and its published error bar describe the same bytes. + * * Reproduce: node packages/cli/scripts/benchmark.mjs * Writes: docs/benchmarks/README.md */ -import { spawnSync } from "node:child_process"; -import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { estimateTokens, standingStub } from "../dist/ksf.js"; -import { ADAPTERS } from "../dist/adapters.js"; - -const here = dirname(fileURLToPath(import.meta.url)); -const repoRoot = resolve(here, "../../.."); -const cli = join(here, "../dist/index.js"); -const fixture = join(repoRoot, "examples/skills/prereview"); - -// agent-plugins is an opt-in publishing target (agent-plugins.org): it is not part -// of a repo's auto-detected fan-out and would not fire in this fixture's compile -// (no plugin.json). It is lazy, so it carries the same stub cost as every other -// lazy target and adds no standing-tax story — this benchmark measures the -// always-on tax across the targets a repo compiles to by default, so it is excluded. -const BENCH_ADAPTERS = ADAPTERS.filter((a) => a.id !== "agent-plugins"); - -// How each target loads a skill, read from the adapters themselves rather than -// restated here — a second copy of this map is exactly how the published numbers -// drift away from what the compiler actually emits. -const LOADING = Object.fromEntries(BENCH_ADAPTERS.map((a) => [a.id, a.loading])); - -function run(args, cwd) { - const r = spawnSync("node", [cli, ...args], { cwd, encoding: "utf8" }); - if (r.status !== 0) throw new Error(`kitbash ${args.join(" ")} failed:\n${r.stdout}${r.stderr}`); - return `${r.stdout}${r.stderr}`; -} - -/** Token cost of one skill's contribution to a target's output. */ -function measure(tmp, target, skillName) { - const read = (rel) => readFileSync(join(tmp, rel), "utf8"); - switch (target) { - case "claude-code": - return estimateTokens(read(`.claude/skills/${skillName}/SKILL.md`)); - case "cursor": - return estimateTokens(read(`.cursor/rules/${skillName}.mdc`)); - // Every one of these is served by the vendor-neutral path and reads the same - // bytes, so they measure identically. Zed and cline compile there directly; - // copilot and gemini also read it, so their own skills directory is not - // written when it is present (see VENDOR_NEUTRAL_ALIASES). Listed separately - // rather than folded together, because a reader looking up "what does Zed - // cost" must find a row. - case "agents": - case "zed": - case "cline": - case "copilot": - case "gemini": - return estimateTokens(read(`.agents/skills/${skillName}/SKILL.md`)); - case "windsurf": - return estimateTokens(read(`.windsurf/rules/${skillName}.md`)); - case "aider": - case "agentsmd": { - const file = target === "aider" ? "CONVENTIONS.md" : "AGENTS.md"; - const m = read(file).match(new RegExp(`[\\s\\S]*?`)); - return m ? estimateTokens(m[0]) : 0; - } - default: - return 0; - } -} - -const tmp = mkdtempSync(join(tmpdir(), "kitbash-bench-")); -const skills = []; -try { - // Every target present so every adapter fires. - for (const d of [".claude", ".cursor", ".agents", ".zed", ".clinerules", ".windsurf", ".github"]) mkdirSync(join(tmp, d)); - writeFileSync(join(tmp, "GEMINI.md"), ""); - writeFileSync(join(tmp, "CONVENTIONS.md"), ""); - - run(["init"], tmp); - - // 1) A real manifested skill (budget 1500, lazy disclosure). - run(["install", `file:${fixture}`], tmp); - skills.push({ name: "prereview", kind: "manifested (budget 1500, lazy)", standing: 60 }); +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { estimateTokens } from "../dist/ksf.js"; +import { LOADING, SKILLS, artifactText, repoRoot, stubText, withBenchWorkspace } from "./bench-fixtures.mjs"; - // 2) A bare SKILL.md-only skill — the skills.sh / Claude Skills convention, - // which has no manifest and so no declared budget. Sized to a realistic - // mid-size community skill. - const bare = join(tmp, "bare"); - mkdirSync(bare); - const bareBody = - "Enforce this project's code-review checklist on every diff before it merges.\n\n" + - Array.from({ length: 40 }, (_, i) => `- Rule ${i + 1}: check the diff for issue class ${i + 1} and cite the exact line and the fix.`).join("\n") + - "\n"; - writeFileSync(join(bare, "SKILL.md"), `---\nname: review-checklist\ndescription: Enforce the team code-review checklist\n---\n\n${bareBody}`); - run(["install", `file:${bare}`], tmp); - skills.push({ name: "review-checklist", kind: "bare / unmanifested (no budget)", standing: null }); - - run(["compile"], tmp); - - const targets = Object.keys(LOADING); +withBenchWorkspace((tmp) => { const rows = []; - for (const s of skills) { - const stubTokens = estimateTokens(standingStub(readFileSync(join(tmp, ".kitbash/skills", s.name, "SKILL.md"), "utf8").replace(/^---[\s\S]*?---\n/, ""))); + for (const s of SKILLS) { + const stubTokens = estimateTokens(stubText(tmp, s.name)); const perTarget = {}; - for (const t of targets) { - const loaded = measure(tmp, t, s.name); - const standing = LOADING[t] === "lazy" ? stubTokens : loaded; - perTarget[t] = { loaded, standing }; + for (const t of Object.keys(LOADING)) { + const loaded = estimateTokens(artifactText(tmp, t, s.name)); + perTarget[t] = { loaded, standing: LOADING[t] === "lazy" ? stubTokens : loaded }; } rows.push({ ...s, stubTokens, perTarget }); } @@ -147,7 +61,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (\`prereview\`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered \`review-checklist\` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and \`--strict\` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: \`prereview\`'s 14x is **14.7x** under a real tokenizer, and \`review-checklist\`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. Regenerate that comparison with \`npm run tokencheck\` inside \`packages/cli\`: it measures the same artifacts this script emits against \`o200k_base\` and prints the range and both ratios. The tokenizer is a devDependency and runs in no shipped code path — kitbash's runtime dependency count is still zero. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) { @@ -171,6 +85,4 @@ No other format has a field for this. Kitbash measures it at compile time, warns mkdirSync(outDir, { recursive: true }); writeFileSync(join(outDir, "README.md"), md); console.log(`wrote docs/benchmarks/README.md`); -} finally { - rmSync(tmp, { recursive: true, force: true }); -} +}); diff --git a/packages/cli/scripts/tokenizer-check.mjs b/packages/cli/scripts/tokenizer-check.mjs new file mode 100644 index 0000000..6d8364b --- /dev/null +++ b/packages/cli/scripts/tokenizer-check.mjs @@ -0,0 +1,64 @@ +/** + * How wrong is `length / 4`? + * + * The compiler budgets skills with a character-count estimate, and the + * benchmark publishes the numbers it produces. This measures those exact + * artifacts against a real tokenizer (`o200k_base`) so the error bar quoted in + * docs/benchmarks/README.md is a regenerated number rather than a remembered + * one — the previous published figure ("about 2-11% high") was wrong in the + * flattering direction, and nothing in the repo could have caught it. + * + * gpt-tokenizer is a devDependency: it runs here and in no shipped code path. + * Kitbash's runtime dependency count stays zero. + * + * Reproduce: npm run tokencheck (inside packages/cli) + */ +import { encode } from "gpt-tokenizer/encoding/o200k_base"; +import { estimateTokens } from "../dist/ksf.js"; +import { LOADING, SKILLS, artifactText, stubText, withBenchWorkspace } from "./bench-fixtures.mjs"; + +const pct = (r) => `${r >= 1 ? "+" : ""}${((r - 1) * 100).toFixed(0)}%`; + +withBenchWorkspace((tmp) => { + const rows = []; + for (const s of SKILLS) { + rows.push({ label: `${s.name} stub`, text: stubText(tmp, s.name) }); + // One row per distinct artifact; targets that emit identical bytes collapse. + const seen = new Map(); + for (const t of Object.keys(LOADING)) { + const text = artifactText(tmp, t, s.name); + if (!text || seen.has(text)) continue; + seen.set(text, t); + rows.push({ label: `${s.name} body (${t})`, text }); + } + } + + console.log("estimator (length / 4) vs o200k_base, over the benchmark artifacts\n"); + console.log(` ${"artifact".padEnd(34)} ${"est".padStart(6)} ${"real".padStart(6)} ${"error".padStart(7)}`); + console.log(` ${"-".repeat(34)} ${"-".repeat(6)} ${"-".repeat(6)} ${"-".repeat(7)}`); + + const ratios = []; + for (const r of rows) { + const est = estimateTokens(r.text); + const real = encode(r.text).length; + if (!real) continue; + ratios.push(est / real); + console.log(` ${r.label.padEnd(34)} ${String(est).padStart(6)} ${String(real).padStart(6)} ${pct(est / real).padStart(7)}`); + } + + const lo = Math.min(...ratios); + const hi = Math.max(...ratios); + console.log(`\n range: ${pct(lo)} to ${pct(hi)}`); + console.log(` ${lo < 1 ? "NOT uniformly conservative — the estimator under-counts somewhere, so a skill can pass a budget it exceeds" : "conservative throughout — every artifact estimates at or above its real cost"}`); + + console.log("\n standing-tax ratio, estimated vs real:"); + for (const s of SKILLS) { + const stub = stubText(tmp, s.name); + const eagerTarget = Object.keys(LOADING).find((t) => LOADING[t] === "eager"); + const body = artifactText(tmp, eagerTarget, s.name); + const e = estimateTokens(body) / estimateTokens(stub); + const a = encode(body).length / encode(stub).length; + console.log(` ${s.name.padEnd(18)} ${e.toFixed(1)}x estimated ${a.toFixed(1)}x real`); + } + console.log("\n docs/benchmarks/README.md quotes these; update it when they move."); +});