Cut 95% of useless framework noise from error logs before feeding them to AI agents.
When your backend crashes, runtimes dump hundreds of lines of node_modules and site-packages junk.
Tokenectomy runs locally in Rust to cut out the noise, redact leaked API keys, and give Cursor & Claude only the code that caused the crash.
Saves up to 99% tokens on Next.js, Node, and Python errors β’ <0.2ms speed β’ 100% offline & local
Website β’ Documentation β’ Quick Start β’ Benchmarks β’ Architecture β’ Sentinel Tier
// Add to ~/.cursor/mcp.json or claude_desktop_config.json
{
"mcpServers": {
"tokenectomy": {
"command": "npx",
"args": ["-y", "tokenectomy-razor", "--mcp"]
}
}
}When an autonomous coding agent runs a failing build or test, the terminal dumps tens of thousands of tokens of internal framework stack frames and potentially leaks production secrets directly into the LLM context window.
Raw Terminal Crash (45,820 tokens + Leaked Secrets)
β
βΌ <0.2ms Zero-Allocation Rust DFA Excision
[Redact Secrets Locally] βββΊ [Filter Framework Frames] βββΊ [Extract Source Context]
β
βΌ
Sanitized Agent Context (118 tokens β’ Zero Secrets β’ Sub-millisecond)
TypeError: Cannot read properties of undefined (reading 'digest')
at Object.<anon> (/node_modules/next/bundle5.js:142:31)
at __webpack_require__ (/node_modules/next/bundle5.js:198:12)
at Object.execute (/node_modules/next/dev-server.js:412:19)
at processTicksAndRejections (task_queues:95:5)
Database connection failed: postgresql://admin:super_secret_password@db.prod.internal:5432/primary
API key leaked: sk-ant-api03-abcdef1234567890abcdef1234567890
[... 480 internal dependency frames flooding LLM context ...]
[:TOKENECTOMY:M2M_CONTROL_PLANE:v1.3.0]
[ADVISORY_ONLY=true]
[STATE=FRAMEWORK_NOISE_PURGED]
[STRATEGY_APPLIED=AGGRESSIVE]
[ORIGINAL_BYTES=45820 | CLEAN_BYTES=118 | REDUCTION=99%]
[PRIMARY_CRASH_COORDINATES=src/components/Header.tsx:42]
[SUGGESTED_NEXT_FRAME=src/components/Header.tsx:42]
[:END_CONTROL_PLANE]
src/components/Header.tsx:42:15 - SyntaxError
42 | const user = useSession( ;
| ^ Expected ')'
π‘οΈ [CONNECTION_STRING_REDACTED]
π‘οΈ [REDACTED] ANTHROPIC_API_KEY=[REDACTED_SECRET_KEY]
Result: 99.7% context token reduction, zero credential leakage, prompt cache preserved. Provides a deterministic M2M control envelope that directs agents toward primary crash coordinates and root-cause patching without narrative ambiguity.
All performance claims are hardware-grounded and independently reproducible on physical hardware (measured on 10-Core Intel Core i5-1235U @ 15W running Arch Linux, Kernel 6.13):
Hardware Dependency Notice: Performance is hardware-dependent; reported throughput represents measured results on the specified test hardware (10-Core Intel Core i5-1235U @ 15W TDP). Throughput scales with higher TDP desktop/server CPUs and faster memory buses. Developers are encouraged to independently audit performance using the reproduction command below.
$ cargo test --release --test stress_benchmark -- --nocapture
=====================================================================================
π§ͺ TOKENECTOMY OSS VERIFIABLE HEAVY STRESS BENCHMARK (100% REPRODUCIBLE IN OSS)
Hardware: 10-Core / 12-Thread Intel Core i5-1235U | OS: Arch Linux | Kernel Telemetry Active
Initial Baseline Process Memory (VmRSS): 3.45 MB
=====================================================================================
π₯ [TEST 1/3] QUARTER-MILLION LINES LOG REDACTION TORTURE (250,000 LINES / 25MB+ BUFFER)
βββ Buffer Size: 24.44 MB (250000 lines)
βββ Redaction Latency: 471.05ms (51.9 MB/sec)
βββ Line Throughput: 530,735 lines/sec
βββ Peak Memory (VmRSS): 76.05 MB (Delta: +72.60 MB)
βββ Status: β
PASSED (100% of 250,000 lines sanitized, zero memory balloon)
π₯ [TEST 2/3] REDOS CATASTROPHIC BACKTRACKING TORTURE (50,000 CHARS PAYLOAD)
βββ Attack Payload Size: 50,082 characters
βββ Execution Latency: 1.165 ms
βββ Status: β
PASSED (Linear O(N) evaluation, ReDoS-resistant on tested payloads)
π₯ [TEST 3/3] HIGH-CONCURRENCY TORTURE (100 PARALLEL OS THREADS)
βββ Thread Concurrency: 100 concurrent OS threads
βββ Successful Operations: 100/100 (100.0%)
βββ Total Elapsed: 11.31ms
βββ Concurrency Throughput: 17,688 ops/sec
βββ Final VmRSS: 78.99 MB
βββ Status: β
PASSED (Zero race condition, zero deadlock)
=====================================================================================
π TOKENECTOMY OSS STRESS BENCHMARK: 3/3 PASSED (100% GREEN)
Total Suite Duration: 580.83ms
Bounded Final VmRSS: 78.99 MB
=====================================================================================
test test_oss_heavy_stress_benchmark ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.58s
| Benchmark Target | Workload Under Test | Verified Measurement | Result |
|---|---|---|---|
| Log Redaction Throughput | 250,000 lines (24.44 MB) enterprise dump with API keys & connection URIs | 530,735 lines/sec (471.0 ms, 51.9 MB/s) | Pass |
| ReDoS Resilience | 50,000-character pathological backtracking regex payload |
1.16 ms (Deterministic Linear |
Pass |
| Thread Concurrency | 100 concurrent OS threads executing simultaneous redaction | 17,688 ops/sec (100/100 completed in 11.31 ms) | Pass |
| Memory Footprint | Peak Resident Memory during 250k-line continuous stress test |
76.05 MB VmRSS via /proc/self/status
|
Pass |
| Release Test Suite | Full integration test matrix across extractors, filters, and analyzers | 57 / 57 Verified Green (Zero panics, zero leaks) | Pass |
Automated evaluation across 12 polyglot crash traces (Rust, Python, TypeScript, Go, YAML) containing 22 ground-truth credentials and clean negative controls. Evaluated head-to-head against Gitleaks v8.30.1.
| Secret Category | Ground Truth | Razor Recall | Razor F1 | Gitleaks Recall | Gitleaks F1 | Sanitization Advantage |
|---|---|---|---|---|---|---|
Anthropic Claude API Key (sk-ant-...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
AWS Access Key ID (AKIA...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| AWS Secret Access Key | 1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| Database URI (PostgreSQL, MySQL, Redis, Mongo) | 4 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| Generic Passwords / Auth Secrets (YAML/JSON) | 3 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
GitHub Personal Access Token (ghp_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
GitLab Personal Access Token (glpat-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
HuggingFace API Token (hf_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| JSON Web Token (RFC 7519 / Truncated) | 2 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
npm Registry Access Token (npm_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
OpenAI API Key (sk-..., sk-proj-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
| PEM Private RSA Key Block | 1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
PyPI Package Upload Token (pypi-AgEI...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
SendGrid API Key (SG...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
Slack Bot/User Token (xoxb-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
Stripe Live/Test Secret Key (sk_live_...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
| Dimension | Tokenectomy Razor (--scrub) |
Gitleaks v8.30.1 | Architectural Rationale |
|---|---|---|---|
| Overall Secret Recall | 100.0% (22/22) | 36.4% (8/22) | Razor captures unquoted URIs, DB ports & AI keys missed by diff rules |
| Overall Precision | 100.0% (0 False Positives) | 88.9% | Zero false triggers on compiler errors & minified traces |
| Overall F1-Score | 100.0% | 51.6% | Comprehensive coverage engineered specifically for crash context |
| Execution Engine | Zero-allocation Rust DFA ($O(N)$) | Go regex scanner + Git tree crawler | Sub-millisecond latency for agent streaming backtraces |
| ReDoS Resilience | Deterministic Linear Time ($O(N)$) | Engine dependent | Non-backtracking DFA regex prevents catastrophic backtracking on tested dumps |
| Sanitization Action | Inline token redaction ([KEY_REDACTED]) |
Warning log only (No scrub) | Directly sanitizes text before ingestion by LLM cortex |
| Metric | Measured Value | Operational Impact for AI Coding Agents |
|---|---|---|
| Mean Token Reduction | 41.67% | Consistently shrinks raw crash trace token footprint |
| Median Reduction (P50) | 42.95% | Typical credential and connection dump reduction |
| 90th Percentile (P90) | 61.42% | Eliminates long multi-line keys and credentials |
| Min / Max Spread | 0.00% β 81.36% | 0% on clean negative controls (zero distortion), up to 81.4% on leaks |
| Total Tokens Preserved / Saved | 920 tokens (44.02% net) | Prevents context window saturation and reduces LLM billing |
Tokenectomy Razor operates natively over JSON-RPC 2.0 stdio, compliant with the official Model Context Protocol specification.
Add to .cursor/mcp.json in your workspace root:
{
"mcpServers": {
"tokenectomy": {
"command": "npx",
"args": ["-y", "tokenectomy-razor", "--mcp"]
}
}
}Add to claude_desktop_config.json:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json - Linux:
~/.config/Claude/claude_desktop_config.json
{
"mcpServers": {
"tokenectomy": {
"command": "npx",
"args": ["-y", "tokenectomy-razor", "--mcp"]
}
}
}Add to ~/.codeium/windsurf/mcp_config.json:
{
"mcpServers": {
"tokenectomy": {
"command": "npx",
"args": ["-y", "tokenectomy-razor", "--mcp"]
}
}
}In Cline or Roo Code settings (cline_mcp_settings.json):
{
"mcpServers": {
"tokenectomy": {
"command": "npx",
"args": ["-y", "tokenectomy-razor", "--mcp"],
"disabled": false,
"autoApprove": ["get_error_context", "analyze_code"]
}
}
}agy mcp add tokenectomy-razor -- npx -y tokenectomy-razor --mcpWhen debugging or piping terminal output directly:
# Pipe terminal test failures through the surgical redactor:
npm test 2>&1 | razor --scrub
# Sanitize a specific raw log file:
razor --scrub --file /var/log/app/error.log > sanitized.log
# Offline local air-gapped mode (zero external network calls):
cat failure.log | razor --scrub --local-onlyTokenectomy Razor can operate as a high-throughput local HTTP reverse proxy on 127.0.0.1:8080. It intercepts outbound prompt streams, performs real-time token excision and credential sanitization, and forwards clean requests upstream to OpenAI, Anthropic, or Ollama.
# Start local gateway proxy forwarding to OpenAI:
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url https://api.openai.com/v1
# Start local gateway forwarding to Ollama:
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url http://127.0.0.1:11434/v1Point any standard SDK client to the local proxy:
export OPENAI_BASE_URL="http://127.0.0.1:8080/v1"Open http://127.0.0.1:8080/dashboard in any browser to monitor real-time token savings, dollar savings (blended LLM pricing), total requests, and active security redactions.
Tokenectomy Razor complies with Glama Grade A Tool Definition Quality Score (TDQS) with explicit parameter boundaries:
| Tool Name | Capability Description |
|---|---|
get_error_context |
Performs trace surgery on error dumps, removes framework noise, redacts credentials, and extracts relevant local source context bounded to the workspace. |
analyze_code |
Performs static AST code analysis to detect resource leaks, unclosed handles, and syntax vulnerabilities with bounded execution limits and precise LSP UTF-16 coordinates. |
apply_code_patch |
Applies atomic file modifications with post-write language syntax verification (cargo check, py_compile, node --check) and automated rollback on failure. |
search_stack_overflow |
Queries Stack Exchange API for relevant error signatures using sanitized, redacted search terms. |
audit_context_health |
Audits raw logs, traces, or prompt payloads for token bloat, framework noise, and credentials. Returns M2M telemetry, savings metrics, and context health grades. |
| Language | Frameworks Supported | Excluded Framework Internals |
|---|---|---|
| Rust | Tokio, Actix-web, Axum | .cargo/registry, .rustup, target/debug/build |
| TypeScript / JS | Next.js, Express, NestJS, Vite | node_modules, .next, dist, webpack internals |
| Python | Django, FastAPI, Flask, PyTorch | site-packages, dist-packages, venv, __pycache__ |
| Golang | Gin, Fiber, Stdlib Panics | go/src (stdlib), go/pkg/mod, vendor |
| Java / Kotlin | Spring Boot 3, Tomcat, Netty | .m2/repository, .gradle/caches, internal bytecode |
| C / C++ | AddressSanitizer, GDB / LLDB | /usr/include, /usr/lib, vcpkg_installed |
| C# (.NET) | ASP.NET Core, .NET Runtime | System.Private.CoreLib, Microsoft.AspNetCore |
| Ruby on Rails | Rails, Sinatra, Bundler | /gems/, ruby/gems, internal rack handlers |
| PHP | Laravel, Symfony | vendor/composer, vendor/symfony |
npx -y tokenectomy-razor --mcpcargo install tokenectomyZero-dependency, standalone release binaries available on GitHub Releases:
- Linux:
x86_64-unknown-linux-gnu,x86_64-unknown-linux-musl,aarch64-unknown-linux-gnu - macOS:
aarch64-apple-darwin(Apple Silicon M1/M2/M3/M4),x86_64-apple-darwin(Intel) - Windows:
x86_64-pc-windows-msvc.exe
docker pull ghcr.io/tokenectomy-labs/razor:latest
docker run -i ghcr.io/tokenectomy-labs/razor:latest --mcp| Capability | Razor (Community OSS) | Sentinel (Commercial Tier) |
|---|---|---|
| Polyglot Stack Trace Surgery | Yes (4 Languages) | Yes (All 7 Languages) |
| O(N) ReDoS-Safe Secret Redaction | Yes | Yes |
| JSON-RPC 2.0 MCP Server | Yes | Yes |
AI Gateway Reverse Proxy (--proxy) |
Yes | Yes |
| SHA-256 Idempotency Cache (24h TTL) | Yes | Yes |
| FinOps Metrics Dashboard | Yes | Yes |
| Tree-sitter AST Syntax Healing | β | Yes |
| Anti-Hallucination Scope Guard | β | Yes |
| Automated Test Rollback (0 Dirty Diff) | β | Yes |
| Multi-File Atomic Transactions | β | Yes |
Time Machine Undo Engine (--undo) |
β | Yes |
| Autonomous Healing State Machine | β | Yes |
Need Enterprise AST Self-Healing? Explore the Sentinel Tier
| Milestone / Capability | Status | Target |
|---|---|---|
| Core Polyglot Log Surgery & |
β Complete | v1.0.0 |
AI Gateway Reverse Proxy (--proxy) & Idempotency Cache |
β Complete | v1.1.0 |
| Multi-arch Docker & GitHub Actions Marketplace Action | β Complete | v1.1.3 |
Static AST Analysis Engine (analyze_code) & UTF-16 LSP |
β Complete | v1.1.5 |
| Glama.ai Tool Definition Quality Score (TDQS Grade A) | β Complete | v1.1.5 |
| Precompiled Standalone Binaries (Linux, macOS, Windows) | β Complete | v1.1.6 |
Official MCP Registry Listing (io.github.Tokenectomy-Labs/razor) |
β Complete | v1.1.7 |
mcpservers.org Directory Listing |
β Complete | v1.1.7 |
| Java/Kotlin (Spring Boot 3) & C/C++ (ASan) Extractors | β Complete | v1.2.0 |
| C# (.NET) & Ruby on Rails Deep Stack Surgery | β Complete | v1.2.2 |
User-defined custom redaction & noise rules (~/.tokenectomy.toml) |
β Complete | v1.2.2 |
Autonomous Context Health Audit (audit_context_health) & M2M Advisory |
β Complete | v1.2.2 |
| Automated Redaction Benchmark & CI Gate | β Complete | v1.2.2 |
Declarative Advisory M2M Control Plane ([:TOKENECTOMY:M2M_CONTROL_PLANE:v1.3.0]) |
β Complete | v1.3.0 |
Interactive Multi-Strategy Budgeting (aggressive, conservative, lossless_compact) |
β Complete | v1.2.3 |
| GitHub Actions OIDC Official Registry Publishing Gate | β Complete | v1.2.3 |
awesome-mcp-servers Community Catalog Listing |
β Complete | v1.2.4 |
Inline Dropped Frame Identities ([DROPPED_FRAMES: ...]) & Anti-Silent Truncation Audit |
β Complete | v1.3.1 |
Content-Addressable Raw Log Cache & Verification Hash (--diff-verify) |
β Complete | v1.3.1 |
| Native VS Code & JetBrains companion extensions | π Planned | v1.4.0 |
| Server-Sent Events (SSE) remote MCP transport | π Planned | v1.4.0 |
- Zero-Knowledge Architecture: All parsing, filtering, and secret redaction execute on physical local hardware. No logs are ever transmitted to third-party telemetry servers.
-
Deterministic Linear-Time Pattern Matching: All pattern matchers utilize finite automaton evaluation (Rust non-backtracking DFA regex engine and Aho-Corasick) providing deterministic
$O(N)$ linear time guarantees on tested adversarial inputs. -
Path Traversal Boundary Isolation: File operations are strictly locked within the active workspace root (
CWD). Path traversals (../) and unauthorized symlinks are blocked. -
Safe Rust Implementation: Core execution paths enforce safe Rust memory guarantees with bounded stream readers (
.take()) preventing resource exhaustion.
For vulnerability disclosures, please review our Security Policy.
Tokenectomy Razor is provided strictly for lawful developer productivity, observability, log surgery, and defensive credential redaction. Any downstream forks, clones, redistributions, or private deployments operate completely independently of the original authors. Tokenectomy Labs and its maintainers assume zero liability for unlawful, malicious, or unauthorized actions committed by third parties using this codebase or derivatives thereof. All downstream operators bear 100% individual responsibility for compliance with local and international cybersecurity laws. See DISCLAIMER.md for full legal terms.
- π Official Website
- π Documentation
- ποΈ System Architecture
- π Changelog
- π€ Contributing Guidelines
- π Security Policy
- βοΈ Disclaimer & Liability
Tokenectomy Labs β’ Autonomous M2M Sub-Cortex
Engineered with precision by Daffa (@daffa2555)
Licensed under the MIT License
