From 38454356d6e58f3ceb7fc2877e2b7c31b50b94d8 Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Sat, 22 Aug 2026 23:56:50 +0530 Subject: [PATCH 1/3] docs: name the prior art on token measurement, and state the estimator's error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README claimed no other tool surfaces a skill's token cost. That is no longer true: skills-check (npx skills-check budget) counts tokens per skill with a real cl100k_base tokenizer, breaks them down per section, detects redundancy between skills, and fails a build over a ceiling. Leaving the claim up would be the same credibility problem the 0.11.0 pass existed to fix, on the one number this project is known for. Corrected to what is still specific here: the per-target half. The same skill has a different standing cost on every agent because each loads it differently, and that falls out of a compile step that already knows each target's loading mode — skills-check measures the file, not the pair. Also states the estimator's real error rather than "a few percent". Measured against o200k_base: length/4 runs 2-11% high on this corpus (prereview 567 vs 515, stub 40 vs 35), biased toward failing budgets early, and the 14x ratio is 14.7x under a real tokenizer. --- README.md | 4 +++- docs/benchmarks/README.md | 2 +- packages/cli/scripts/benchmark.mjs | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a420ca9..506a236 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,9 @@ compiled 1 skill for 3 targets The identical instructions cost ~40 standing tokens on a target that lazy-loads and ~560 on one that can't — a **14× per-session tax**, charged before the skill is ever invoked, on every agent that has no lazy mode. A separate unmanifested fixture measures 19 against 885, a 47× gap. One real third-party skill from the [skills.sh](https://www.skills.sh) convention carries ~5,044 tokens of instructions, which compile to ~5,101 standing tokens on an eager target. -Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. I have not found another tool that surfaces that number. +Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. + +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer — measured against `o200k_base` it runs about 2-11% high on this corpus, and the ratio it reports is within half a multiple of the real one.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index 089aad5..e22c020 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` on this repo's corpus that estimator runs **about 2-11% high** — `prereview`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 3aed266..6a14d21 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -147,7 +147,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (~4 chars/token), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Absolute counts will differ by a few percent against a model-specific tokenizer; the lazy-vs-eager *ratio* is what the argument rests on. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` on this repo's corpus that estimator runs **about 2-11% high** — \`prereview\`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) { From 0939290cd89561d4ac0bb9c7f43d9584a811b99f Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Sat, 29 Aug 2026 19:40:20 +0530 Subject: [PATCH 2/3] docs(benchmark): state the estimator's measured error, not a flattering one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous wording said length/4 runs 'about 2-11% high' and called that the safe direction for budget enforcement. Measuring it against o200k_base over the artifacts the benchmark emits, neither half holds: prereview stub 40 est 35 real 1.14x prereview body 567 est 515 real 1.10x review-checklist stub 19 est 14 real 1.36x review-checklist body 880 est 942 real 0.93x The range is 7% low to 36% high, not 2-11% high, and the direction flips with content: prose over-counts, repetitive numbered lists under-count. A skill shaped like review-checklist can therefore pass a budget it actually exceeds, which is the opposite of the claimed bias, and --strict will not catch it. The ratio is more robust but not exact either: prereview's 14x is 14.7x, and review-checklist's 47x is 68x — the estimator understates that gap by a third, so 'within half a multiple' was wrong too. Says how the comparison was taken and that it is a one-off, since kitbash ships no tokenizer and cannot regenerate it. --- README.md | 2 +- docs/benchmarks/README.md | 2 +- packages/cli/scripts/benchmark.mjs | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 506a236..109e7d9 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The identical instructions cost ~40 standing tokens on a target that lazy-loads Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. -Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer — measured against `o200k_base` it runs about 2-11% high on this corpus, and the ratio it reports is within half a multiple of the real one.) +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer. Measured against `o200k_base` it is off by anywhere from 7% low to 36% high depending on what the skill contains, and it understates the gap on the 47× fixture, which is really 68×. The per-target *story* survives that error; the exact figures should be read as estimates, and [the benchmark](docs/benchmarks/README.md) says so.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index e22c020..fedf7c6 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` on this repo's corpus that estimator runs **about 2-11% high** — `prereview`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (`prereview`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered `review-checklist` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and `--strict` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: `prereview`'s 14x is **14.7x** under a real tokenizer, and `review-checklist`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. That comparison was made with `tiktoken`'s `o200k_base` over the exact artifacts this script emits; kitbash ships no tokenizer, so it is a one-off measurement rather than a regenerated number, and it should be re-taken when the fixtures change. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 6a14d21..3b157ba 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -147,7 +147,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` on this repo's corpus that estimator runs **about 2-11% high** — \`prereview\`'s body is 567 here and 515 in real tokens, its stub 40 against 35 — which is the safe direction for budget enforcement, since a skill fails its budget slightly early rather than late. The bias very largely cancels in the *ratio* the argument rests on: the 14x gap above is **14.7x** under a real tokenizer. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (\`prereview\`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered \`review-checklist\` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and \`--strict\` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: \`prereview\`'s 14x is **14.7x** under a real tokenizer, and \`review-checklist\`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. That comparison was made with \`tiktoken\`'s \`o200k_base\` over the exact artifacts this script emits; kitbash ships no tokenizer, so it is a one-off measurement rather than a regenerated number, and it should be re-taken when the fixtures change. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) { From 27c25caaa4c9fb23e00db4c51cd369e4f623a628 Mon Sep 17 00:00:00 2001 From: Harsh Singh Date: Sat, 29 Aug 2026 20:05:05 +0530 Subject: [PATCH 3/3] build(bench): regenerate the estimator's error bar instead of remembering it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The error bar the benchmark quotes was a number someone measured once and typed in. It was wrong — 'about 2-11% high' against a real range of -7% to +36% — and nothing in the repo could have caught that, which is the same class of problem as a hand-maintained copy of the loading-mode table. Adds scripts/tokenizer-check.mjs (npm run tokencheck): measures the artifacts the benchmark emits against o200k_base and prints the range and both standing-tax ratios, so the figures in docs/benchmarks/README.md are regenerable. gpt-tokenizer is a devDependency and runs in no shipped code path; runtime dependency count is still zero. Cross-checked against tiktoken on four artifacts — identical counts. Extracts the corpus into scripts/bench-fixtures.mjs so the table and its error bar describe the same bytes rather than two copies of the fixture setup. benchmark.mjs regenerates docs/benchmarks/README.md byte-identically across the refactor. --- README.md | 2 +- docs/benchmarks/README.md | 2 +- packages/cli/package-lock.json | 12 ++- packages/cli/package.json | 4 +- packages/cli/scripts/bench-fixtures.mjs | 117 ++++++++++++++++++++++ packages/cli/scripts/benchmark.mjs | 118 +++-------------------- packages/cli/scripts/tokenizer-check.mjs | 64 ++++++++++++ 7 files changed, 211 insertions(+), 108 deletions(-) create mode 100644 packages/cli/scripts/bench-fixtures.mjs create mode 100644 packages/cli/scripts/tokenizer-check.mjs diff --git a/README.md b/README.md index 109e7d9..478cf42 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ The identical instructions cost ~40 standing tokens on a target that lazy-loads Those numbers are measured, not asserted: the method and the full per-target table are in [docs/benchmarks/README.md](docs/benchmarks/README.md), and `npm run bench` inside `packages/cli` regenerates them. A converter would translate the format and stop. Kitbash reads the skill and tells you what it will cost you. -Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer. Measured against `o200k_base` it is off by anywhere from 7% low to 36% high depending on what the skill contains, and it understates the gap on the 47× fixture, which is really 68×. The per-target *story* survives that error; the exact figures should be read as estimates, and [the benchmark](docs/benchmarks/README.md) says so.) +Counting a skill's tokens is no longer unusual — [`skills-check`](https://www.skillscheck.ai/commands/budget) does it with a real `cl100k_base` tokenizer, per section, and will fail a build over a ceiling. What is still specific to Kitbash is the *per-target* half: the same skill has a different standing cost on every agent, because each one loads it differently, and that number falls out of the compile step that already knows each target's loading mode. `~40 on a lazy target, ~560 on an eager one` is a fact about the pair, not about the file. (Kitbash's own estimate is `length / 4`, not a tokenizer. Measured against `o200k_base` it is off by anywhere from 7% low to 36% high depending on what the skill contains, and it understates the gap on the 47× fixture, which is really 68×. The per-target *story* survives that error; the exact figures should be read as estimates. `npm run tokencheck` regenerates the comparison, and [the benchmark](docs/benchmarks/README.md) carries it.) Kitbash always compiles to the cheapest loading mode a target actually supports — nine of the eleven lazy-load; Aider's `CONVENTIONS.md` and the `AGENTS.md` floor cannot, and carry the whole body every session. (Aider does not read `CONVENTIONS.md` on its own — until you add `read: CONVENTIONS.md` to `.aider.conf.yml`, it costs nothing and does nothing, and `compile` says so.) `--strict` turns budget overruns and degradation warnings into build failures. diff --git a/docs/benchmarks/README.md b/docs/benchmarks/README.md index fedf7c6..65cfef0 100644 --- a/docs/benchmarks/README.md +++ b/docs/benchmarks/README.md @@ -11,7 +11,7 @@ Two costs matter: One caveat on the eager rows: `agentsmd` is loaded by the agent automatically, but **aider does not read `CONVENTIONS.md` on its own** — it is loaded with `aider --read CONVENTIONS.md` or a `read:` entry in `.aider.conf.yml`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and `compile` says which case a repo is in. -Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (`prereview`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered `review-checklist` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and `--strict` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: `prereview`'s 14x is **14.7x** under a real tokenizer, and `review-checklist`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. That comparison was made with `tiktoken`'s `o200k_base` over the exact artifacts this script emits; kitbash ships no tokenizer, so it is a one-off measurement rather than a regenerated number, and it should be re-taken when the fixtures change. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. +Token counts are estimates (`length / 4`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against `o200k_base` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (`prereview`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered `review-checklist` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and `--strict` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: `prereview`'s 14x is **14.7x** under a real tokenizer, and `review-checklist`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. Regenerate that comparison with `npm run tokencheck` inside `packages/cli`: it measures the same artifacts this script emits against `o200k_base` and prints the range and both ratios. The tokenizer is a devDependency and runs in no shipped code path — kitbash's runtime dependency count is still zero. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with `node packages/cli/scripts/benchmark.mjs`. ## `prereview` — manifested (budget 1500, lazy) diff --git a/packages/cli/package-lock.json b/packages/cli/package-lock.json index 3760e86..6e1efd2 100644 --- a/packages/cli/package-lock.json +++ b/packages/cli/package-lock.json @@ -1,18 +1,19 @@ { "name": "kitbash", - "version": "0.16.0", + "version": "0.24.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "kitbash", - "version": "0.16.0", + "version": "0.24.1", "license": "Apache-2.0", "bin": { "kitbash": "dist/index.js" }, "devDependencies": { "@types/node": "^26.1.1", + "gpt-tokenizer": "^4.0.0", "typescript": "^5.6.0" }, "engines": { @@ -29,6 +30,13 @@ "undici-types": "~8.3.0" } }, + "node_modules/gpt-tokenizer": { + "version": "4.0.0", + "resolved": "https://registry.npmjs.org/gpt-tokenizer/-/gpt-tokenizer-4.0.0.tgz", + "integrity": "sha512-YAWIyzvuVUHEfW7tFfFAxH8qQb+Q3RU9nYOTy7skMNX5qzU6Q8jxTHZLyO56ug1vYvCR7wndzpd3jwD86/mhjQ==", + "dev": true, + "license": "MIT" + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", diff --git a/packages/cli/package.json b/packages/cli/package.json index b24e262..c87d300 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -24,10 +24,12 @@ "dev": "node --experimental-strip-types src/index.ts", "test": "npm run build && node scripts/test.mjs", "bench": "npm run build && node scripts/benchmark.mjs", - "prepublishOnly": "npm test" + "prepublishOnly": "npm test", + "tokencheck": "npm run build && node scripts/tokenizer-check.mjs" }, "devDependencies": { "@types/node": "^26.1.1", + "gpt-tokenizer": "^4.0.0", "typescript": "^5.6.0" }, "keywords": [ diff --git a/packages/cli/scripts/bench-fixtures.mjs b/packages/cli/scripts/bench-fixtures.mjs new file mode 100644 index 0000000..23a485a --- /dev/null +++ b/packages/cli/scripts/bench-fixtures.mjs @@ -0,0 +1,117 @@ +/** + * The benchmark corpus, built once and shared. + * + * Both `benchmark.mjs` (which publishes the token table) and + * `tokenizer-check.mjs` (which measures how wrong the estimator is) have to + * look at *the same bytes*, or the error bar the table quotes is an error bar + * for some other corpus. Extracted here so there is one definition of the + * fixtures and one extractor per target, not two that drift. + */ +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { standingStub } from "../dist/ksf.js"; +import { ADAPTERS } from "../dist/adapters.js"; + +const here = dirname(fileURLToPath(import.meta.url)); +export const repoRoot = resolve(here, "../../.."); +const cli = join(here, "../dist/index.js"); + +// agent-plugins is an opt-in publishing target (agent-plugins.org): it is not part +// of a repo's auto-detected fan-out and would not fire in this fixture's compile +// (no plugin.json). It is lazy, so it carries the same stub cost as every other +// lazy target and adds no standing-tax story — this benchmark measures the +// always-on tax across the targets a repo compiles to by default, so it is excluded. +const BENCH_ADAPTERS = ADAPTERS.filter((a) => a.id !== "agent-plugins"); + +// How each target loads a skill, read from the adapters themselves rather than +// restated here — a second copy of this map is exactly how the published numbers +// drift away from what the compiler actually emits. +export const LOADING = Object.fromEntries(BENCH_ADAPTERS.map((a) => [a.id, a.loading])); + +export const SKILLS = [ + { name: "prereview", kind: "manifested (budget 1500, lazy)", standing: 60 }, + { name: "review-checklist", kind: "bare / unmanifested (no budget)", standing: null }, +]; + +function run(args, cwd) { + const r = spawnSync("node", [cli, ...args], { cwd, encoding: "utf8" }); + if (r.status !== 0) throw new Error(`kitbash ${args.join(" ")} failed:\n${r.stdout}${r.stderr}`); + return `${r.stdout}${r.stderr}`; +} + +/** The compiled text of one skill's contribution to a target's output. */ +export function artifactText(tmp, target, skillName) { + const read = (rel) => readFileSync(join(tmp, rel), "utf8"); + switch (target) { + case "claude-code": + return read(`.claude/skills/${skillName}/SKILL.md`); + case "cursor": + return read(`.cursor/rules/${skillName}.mdc`); + // Every one of these is served by the vendor-neutral path and reads the same + // bytes, so they measure identically. Zed and cline compile there directly; + // copilot and gemini also read it, so their own skills directory is not + // written when it is present (see VENDOR_NEUTRAL_ALIASES). Listed separately + // rather than folded together, because a reader looking up "what does Zed + // cost" must find a row. + case "agents": + case "zed": + case "cline": + case "copilot": + case "gemini": + return read(`.agents/skills/${skillName}/SKILL.md`); + case "windsurf": + return read(`.windsurf/rules/${skillName}.md`); + case "aider": + case "agentsmd": { + const file = target === "aider" ? "CONVENTIONS.md" : "AGENTS.md"; + const m = read(file).match(new RegExp(`[\\s\\S]*?`)); + return m ? m[0] : ""; + } + default: + return ""; + } +} + +/** The standing stub an agent keeps in context for a lazy target. */ +export function stubText(tmp, skillName) { + const body = readFileSync(join(tmp, ".kitbash/skills", skillName, "SKILL.md"), "utf8").replace(/^---[\s\S]*?---\n/, ""); + return standingStub(body); +} + +/** + * Build the corpus in a temp repo with every target present so every adapter + * fires, hand it to `fn`, and clean up. `fn` receives the workspace path. + */ +export function withBenchWorkspace(fn) { + const tmp = mkdtempSync(join(tmpdir(), "kitbash-bench-")); + try { + for (const d of [".claude", ".cursor", ".agents", ".zed", ".clinerules", ".windsurf", ".github"]) mkdirSync(join(tmp, d)); + writeFileSync(join(tmp, "GEMINI.md"), ""); + writeFileSync(join(tmp, "CONVENTIONS.md"), ""); + + run(["init"], tmp); + + // 1) A real manifested skill (budget 1500, lazy disclosure). + run(["install", `file:${join(repoRoot, "examples/skills/prereview")}`], tmp); + + // 2) A bare SKILL.md-only skill — the skills.sh / Claude Skills convention, + // which has no manifest and so no declared budget. Sized to a realistic + // mid-size community skill. + const bare = join(tmp, "bare"); + mkdirSync(bare); + const bareBody = + "Enforce this project's code-review checklist on every diff before it merges.\n\n" + + Array.from({ length: 40 }, (_, i) => `- Rule ${i + 1}: check the diff for issue class ${i + 1} and cite the exact line and the fix.`).join("\n") + + "\n"; + writeFileSync(join(bare, "SKILL.md"), `---\nname: review-checklist\ndescription: Enforce the team code-review checklist\n---\n\n${bareBody}`); + run(["install", `file:${bare}`], tmp); + + run(["compile"], tmp); + return fn(tmp); + } finally { + rmSync(tmp, { recursive: true, force: true }); + } +} diff --git a/packages/cli/scripts/benchmark.mjs b/packages/cli/scripts/benchmark.mjs index 3b157ba..08b73bf 100644 --- a/packages/cli/scripts/benchmark.mjs +++ b/packages/cli/scripts/benchmark.mjs @@ -9,111 +9,25 @@ * standing stub on lazy targets and its whole body on eager ones. That gap is * the hidden tax of hand-maintaining a copy per agent. Here it is, in numbers. * + * The corpus lives in bench-fixtures.mjs, shared with tokenizer-check.mjs so the + * table and its published error bar describe the same bytes. + * * Reproduce: node packages/cli/scripts/benchmark.mjs * Writes: docs/benchmarks/README.md */ -import { spawnSync } from "node:child_process"; -import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { dirname, join, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { estimateTokens, standingStub } from "../dist/ksf.js"; -import { ADAPTERS } from "../dist/adapters.js"; - -const here = dirname(fileURLToPath(import.meta.url)); -const repoRoot = resolve(here, "../../.."); -const cli = join(here, "../dist/index.js"); -const fixture = join(repoRoot, "examples/skills/prereview"); - -// agent-plugins is an opt-in publishing target (agent-plugins.org): it is not part -// of a repo's auto-detected fan-out and would not fire in this fixture's compile -// (no plugin.json). It is lazy, so it carries the same stub cost as every other -// lazy target and adds no standing-tax story — this benchmark measures the -// always-on tax across the targets a repo compiles to by default, so it is excluded. -const BENCH_ADAPTERS = ADAPTERS.filter((a) => a.id !== "agent-plugins"); - -// How each target loads a skill, read from the adapters themselves rather than -// restated here — a second copy of this map is exactly how the published numbers -// drift away from what the compiler actually emits. -const LOADING = Object.fromEntries(BENCH_ADAPTERS.map((a) => [a.id, a.loading])); - -function run(args, cwd) { - const r = spawnSync("node", [cli, ...args], { cwd, encoding: "utf8" }); - if (r.status !== 0) throw new Error(`kitbash ${args.join(" ")} failed:\n${r.stdout}${r.stderr}`); - return `${r.stdout}${r.stderr}`; -} - -/** Token cost of one skill's contribution to a target's output. */ -function measure(tmp, target, skillName) { - const read = (rel) => readFileSync(join(tmp, rel), "utf8"); - switch (target) { - case "claude-code": - return estimateTokens(read(`.claude/skills/${skillName}/SKILL.md`)); - case "cursor": - return estimateTokens(read(`.cursor/rules/${skillName}.mdc`)); - // Every one of these is served by the vendor-neutral path and reads the same - // bytes, so they measure identically. Zed and cline compile there directly; - // copilot and gemini also read it, so their own skills directory is not - // written when it is present (see VENDOR_NEUTRAL_ALIASES). Listed separately - // rather than folded together, because a reader looking up "what does Zed - // cost" must find a row. - case "agents": - case "zed": - case "cline": - case "copilot": - case "gemini": - return estimateTokens(read(`.agents/skills/${skillName}/SKILL.md`)); - case "windsurf": - return estimateTokens(read(`.windsurf/rules/${skillName}.md`)); - case "aider": - case "agentsmd": { - const file = target === "aider" ? "CONVENTIONS.md" : "AGENTS.md"; - const m = read(file).match(new RegExp(`[\\s\\S]*?`)); - return m ? estimateTokens(m[0]) : 0; - } - default: - return 0; - } -} - -const tmp = mkdtempSync(join(tmpdir(), "kitbash-bench-")); -const skills = []; -try { - // Every target present so every adapter fires. - for (const d of [".claude", ".cursor", ".agents", ".zed", ".clinerules", ".windsurf", ".github"]) mkdirSync(join(tmp, d)); - writeFileSync(join(tmp, "GEMINI.md"), ""); - writeFileSync(join(tmp, "CONVENTIONS.md"), ""); - - run(["init"], tmp); - - // 1) A real manifested skill (budget 1500, lazy disclosure). - run(["install", `file:${fixture}`], tmp); - skills.push({ name: "prereview", kind: "manifested (budget 1500, lazy)", standing: 60 }); +import { mkdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import { estimateTokens } from "../dist/ksf.js"; +import { LOADING, SKILLS, artifactText, repoRoot, stubText, withBenchWorkspace } from "./bench-fixtures.mjs"; - // 2) A bare SKILL.md-only skill — the skills.sh / Claude Skills convention, - // which has no manifest and so no declared budget. Sized to a realistic - // mid-size community skill. - const bare = join(tmp, "bare"); - mkdirSync(bare); - const bareBody = - "Enforce this project's code-review checklist on every diff before it merges.\n\n" + - Array.from({ length: 40 }, (_, i) => `- Rule ${i + 1}: check the diff for issue class ${i + 1} and cite the exact line and the fix.`).join("\n") + - "\n"; - writeFileSync(join(bare, "SKILL.md"), `---\nname: review-checklist\ndescription: Enforce the team code-review checklist\n---\n\n${bareBody}`); - run(["install", `file:${bare}`], tmp); - skills.push({ name: "review-checklist", kind: "bare / unmanifested (no budget)", standing: null }); - - run(["compile"], tmp); - - const targets = Object.keys(LOADING); +withBenchWorkspace((tmp) => { const rows = []; - for (const s of skills) { - const stubTokens = estimateTokens(standingStub(readFileSync(join(tmp, ".kitbash/skills", s.name, "SKILL.md"), "utf8").replace(/^---[\s\S]*?---\n/, ""))); + for (const s of SKILLS) { + const stubTokens = estimateTokens(stubText(tmp, s.name)); const perTarget = {}; - for (const t of targets) { - const loaded = measure(tmp, t, s.name); - const standing = LOADING[t] === "lazy" ? stubTokens : loaded; - perTarget[t] = { loaded, standing }; + for (const t of Object.keys(LOADING)) { + const loaded = estimateTokens(artifactText(tmp, t, s.name)); + perTarget[t] = { loaded, standing: LOADING[t] === "lazy" ? stubTokens : loaded }; } rows.push({ ...s, stubTokens, perTarget }); } @@ -147,7 +61,7 @@ Two costs matter: One caveat on the eager rows: \`agentsmd\` is loaded by the agent automatically, but **aider does not read \`CONVENTIONS.md\` on its own** — it is loaded with \`aider --read CONVENTIONS.md\` or a \`read:\` entry in \`.aider.conf.yml\`. Its standing figure is what the file costs *once wired in*; unconfigured it costs nothing, and \`compile\` says which case a repo is in. -Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (\`prereview\`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered \`review-checklist\` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and \`--strict\` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: \`prereview\`'s 14x is **14.7x** under a real tokenizer, and \`review-checklist\`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. That comparison was made with \`tiktoken\`'s \`o200k_base\` over the exact artifacts this script emits; kitbash ships no tokenizer, so it is a one-off measurement rather than a regenerated number, and it should be re-taken when the fixtures change. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. +Token counts are estimates (\`length / 4\`), the same estimator the compiler enforces budgets with, so the benchmark and the build agree by construction. Measured against \`o200k_base\` across the artifacts in this table, that estimator is off by **7% low to 36% high**, and the direction depends on what the skill contains: prose over-counts (\`prereview\`'s body 567 here against 515 real, its stub 40 against 35), while the repetitive, heavily numbered \`review-checklist\` body **under**-counts (880 against 942). So it is not uniformly conservative — a skill whose body looks like that one can pass a budget it actually exceeds, and \`--strict\` will not catch it. The lazy-vs-eager *ratio* the argument rests on is more robust than the absolute numbers but is not exact either: \`prereview\`'s 14x is **14.7x** under a real tokenizer, and \`review-checklist\`'s 47x is **68x** — the estimator understates that gap by a third. Treat every number here as an estimate with that error bar, not as a token count. Regenerate that comparison with \`npm run tokencheck\` inside \`packages/cli\`: it measures the same artifacts this script emits against \`o200k_base\` and prints the range and both ratios. The tokenizer is a devDependency and runs in no shipped code path — kitbash's runtime dependency count is still zero. Loading modes are read from the adapters themselves, not restated here, so this table cannot drift from what the compiler emits. Reproduce with \`node packages/cli/scripts/benchmark.mjs\`. `; for (const r of rows) { @@ -171,6 +85,4 @@ No other format has a field for this. Kitbash measures it at compile time, warns mkdirSync(outDir, { recursive: true }); writeFileSync(join(outDir, "README.md"), md); console.log(`wrote docs/benchmarks/README.md`); -} finally { - rmSync(tmp, { recursive: true, force: true }); -} +}); diff --git a/packages/cli/scripts/tokenizer-check.mjs b/packages/cli/scripts/tokenizer-check.mjs new file mode 100644 index 0000000..6d8364b --- /dev/null +++ b/packages/cli/scripts/tokenizer-check.mjs @@ -0,0 +1,64 @@ +/** + * How wrong is `length / 4`? + * + * The compiler budgets skills with a character-count estimate, and the + * benchmark publishes the numbers it produces. This measures those exact + * artifacts against a real tokenizer (`o200k_base`) so the error bar quoted in + * docs/benchmarks/README.md is a regenerated number rather than a remembered + * one — the previous published figure ("about 2-11% high") was wrong in the + * flattering direction, and nothing in the repo could have caught it. + * + * gpt-tokenizer is a devDependency: it runs here and in no shipped code path. + * Kitbash's runtime dependency count stays zero. + * + * Reproduce: npm run tokencheck (inside packages/cli) + */ +import { encode } from "gpt-tokenizer/encoding/o200k_base"; +import { estimateTokens } from "../dist/ksf.js"; +import { LOADING, SKILLS, artifactText, stubText, withBenchWorkspace } from "./bench-fixtures.mjs"; + +const pct = (r) => `${r >= 1 ? "+" : ""}${((r - 1) * 100).toFixed(0)}%`; + +withBenchWorkspace((tmp) => { + const rows = []; + for (const s of SKILLS) { + rows.push({ label: `${s.name} stub`, text: stubText(tmp, s.name) }); + // One row per distinct artifact; targets that emit identical bytes collapse. + const seen = new Map(); + for (const t of Object.keys(LOADING)) { + const text = artifactText(tmp, t, s.name); + if (!text || seen.has(text)) continue; + seen.set(text, t); + rows.push({ label: `${s.name} body (${t})`, text }); + } + } + + console.log("estimator (length / 4) vs o200k_base, over the benchmark artifacts\n"); + console.log(` ${"artifact".padEnd(34)} ${"est".padStart(6)} ${"real".padStart(6)} ${"error".padStart(7)}`); + console.log(` ${"-".repeat(34)} ${"-".repeat(6)} ${"-".repeat(6)} ${"-".repeat(7)}`); + + const ratios = []; + for (const r of rows) { + const est = estimateTokens(r.text); + const real = encode(r.text).length; + if (!real) continue; + ratios.push(est / real); + console.log(` ${r.label.padEnd(34)} ${String(est).padStart(6)} ${String(real).padStart(6)} ${pct(est / real).padStart(7)}`); + } + + const lo = Math.min(...ratios); + const hi = Math.max(...ratios); + console.log(`\n range: ${pct(lo)} to ${pct(hi)}`); + console.log(` ${lo < 1 ? "NOT uniformly conservative — the estimator under-counts somewhere, so a skill can pass a budget it exceeds" : "conservative throughout — every artifact estimates at or above its real cost"}`); + + console.log("\n standing-tax ratio, estimated vs real:"); + for (const s of SKILLS) { + const stub = stubText(tmp, s.name); + const eagerTarget = Object.keys(LOADING).find((t) => LOADING[t] === "eager"); + const body = artifactText(tmp, eagerTarget, s.name); + const e = estimateTokens(body) / estimateTokens(stub); + const a = encode(body).length / encode(stub).length; + console.log(` ${s.name.padEnd(18)} ${e.toFixed(1)}x estimated ${a.toFixed(1)}x real`); + } + console.log("\n docs/benchmarks/README.md quotes these; update it when they move."); +});