Skip to content

feat: support custom tokenizer modules in the CLI - #32

Merged
royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
Gambit-Checkmate:feat/cli-tokenizer-module
Sep 17, 2026
Merged

royalpinto007 merged 1 commit into
AgentPostmortem:mainfrom
Gambit-Checkmate:feat/cli-tokenizer-module

Conversation

@Gambit-Checkmate

Copy link
Copy Markdown
Contributor

What & why

Closes #6: add --tokenizer <module> so CLI analysis and compaction can use the existing custom-counter API instead of always using the built-in estimate.

Changes

  • Load a caller-relative or absolute local module and pass its named count(text) export to both CLI modes, leaving the default counter unchanged.
  • Report missing flag values, module-load errors, and missing/non-callable count exports without a stack trace.
  • Document a gpt-tokenizer adapter, trusted-module execution, and the remaining message-framing/image/protected-turn limitations without adding a runtime dependency.
  • Add tests for custom counts, actual budget-driven message removal, path handling, default behavior, and invalid modules.

Validation

The tokenizer integration/error tests were added first and failed on the original CLI, then passed after implementation.

  • node --test: 41 passed, 0 failed (Node 24.15.0, Windows).
  • node --check bin/tokencut.mjs: passed.
  • git diff --check: passed.

Checklist

  • Focused change with a clear description
  • Ran project tests and syntax checks
  • Updated docs and CHANGELOG
  • Linked the related issue

AI assistance was used to implement and validate this contribution.

@royalpinto007
royalpinto007 merged commit 7c2291b into AgentPostmortem:main Sep 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The CLI cannot use a real tokenizer, only the built-in estimate

2 participants