Reduction

Five-layer token-optimization pipeline for AI agents. Attacks token waste at every layer — from the shell to the serialized response — on every input and output. Compression is reversible: the agent can retrieve any dropped detail on demand (CCR). Connect via the in-process SDK, zero-touch client adapters, an MCP server, a CLI, or a shared HTTP gateway.

flowchart LR
    subgraph inputs [Inputs]
        TOOL[tool / command output]
        CTX[context docs]
        USER[user turn]
        SYS[system prompt]
    end

    subgraph pipeline ["Reduction pipeline — prepare()"]
        L1["L1 · shell filter<br/>content-aware routing<br/>(JSON/diff/log/code)"]
        L2["L2 · LLMLingua-2<br/>context compress<br/>(optional)"]
        NORM["normalize<br/>whitespace + dedupe"]
        L5IN["L5 · caveman +<br/>TOON/YAML contract"]
        L4["L4 · stable-prefix ordering<br/>+ cache_control breakpoint"]
    end

    L3{{"L3 · semantic cache<br/>hit? skip generation<br/>(optional)"}}
    PROVIDER[[Provider<br/>Anthropic / OpenAI]]

    subgraph outputs [Outputs]
        L5OUT["L5 · decode<br/>TOON/YAML → objects"]
        METRICS[(metrics<br/>+ savings)]
    end

    CCR[("CCR store<br/>originals by ref<br/>reduction_retrieve")]

    TOOL --> L1
    CTX --> L2
    USER --> NORM
    SYS --> L5IN

    L1 --> NORM
    L2 --> NORM
    NORM --> L4
    L5IN --> L4

    L1 -. store originals .-> CCR
    L2 -. store originals .-> CCR

    L4 --> L3
    L3 -- miss --> PROVIDER
    L3 -- hit --> L5OUT
    PROVIDER --> L5OUT
    L5OUT --> METRICS
    CCR -. retrieve on demand .-> PROVIDER

    classDef opt stroke-dasharray: 4 4;
    class L2,L3 opt;

Loading

Entry points into this pipeline: in-process SDK (TokenOptimizer), zero-touch client adapters, reduction.install(), an MCP server, a CLI, an OpenAI/Anthropic-compatible proxy, and a shared HTTP gateway.

Quickstart — one line, zero code change

The fastest way to start saving tokens. Add two lines at startup; nothing else in your code changes. Every anthropic / openai client in the process — sync or async — is patched in place to route through the pipeline. (reduction init prints this snippet and tells you which SDKs it detects.)

import reduction
reduction.install()          # the entire integration

# ... your existing code, untouched ...
import anthropic
client = anthropic.Anthropic()
client.messages.create(model="claude-sonnet-4-6", max_tokens=512,
                       system="You are a planner.", messages=[...])

print(reduction.report())    # token-savings summary

No constructor swap, no base-URL change, no proxy process. reduction.install() is idempotent and reduction.uninstall() restores the original SDKs. Configure entirely with env vars (e.g. REDUCTION_OUTPUT_FORMAT=toon) — see Configuration. Prefer an explicit wrapper or a shared HTTP service instead? Those paths are below.

The five layers

#	Layer	Technique	Default	Savings
1	Shell	content-aware tool-output compression (JSON/diff/log routing) + reversible CCR; zap/RTK or heuristic fallback	on	60–97% on tool output
2	Context	LLMLingua-2 compresses retrieved docs (never instructions)	off*	2–5× on context
3	Cache	in-process LocalSemanticCache (SDK) or LiteLLM semantic cache (gateway, Redis VSS / Qdrant)	off*	skips generation on hit
4	Provider	Native prompt caching via stable-prefix ordering + cache_control	on	90% input discount (Anthropic)
5	Output	Caveman persona + TOON/YAML serialization	on	~45% output; 30–60% structured

* Layers 2 and 3 pull heavy optional dependencies (torch, litellm/Redis) and only pay off on large or reused context, so they default off. Everything else is dependency-free and on by default.

Content-aware compression + CCR (reversible)

Layer 1 doesn't just truncate lines. Tool/command output is classified (JSON / diff / log / code / text) and routed to a specialized compressor:

Content	Compressor	Example result
JSON	SmartCrusher-lite — sample large uniform arrays, render as TOON	300-row scan: 9,006 → 231 tokens (97%)
diff	per-file +/- shortstat	5,000-line diff → 3 files, +142/-89
log	dedupe + keep error/warn lines + head/tail	noisy build log → signal only
code / text	lossless dedupe + whitespace	safe, no data dropped

Because lossy compression occasionally hides the one row the model needs, every compression is reversible via CCR (Compress-Cache-Retrieve): the original is stored under a short content hash and the compressed text carries a marker — [reduction: json compressed 97% ..., ref=14a9cd0d]. The agent calls the reduction_retrieve tool (or opt.retrieve(ref)) to get the original back.

opt = TokenOptimizer()
small = opt.filter_tool_output(huge_json_scan_output)   # 97% smaller, carries a ref
# ...later, if the model needs everything:
original = opt.retrieve("14a9cd0d")

Install

pip install -e .                # core SDK (zero heavy deps)
pip install -e ".[gateway]"     # + FastAPI/LiteLLM HTTP gateway
pip install -e ".[compress]"    # + LLMLingua-2 (Layer 2)
pip install -e ".[mcp]"         # + MCP server (reduction_compress/retrieve/stats)
pip install -e ".[proxy]"       # + OpenAI/Anthropic compression proxy
pip install -e ".[code]"        # + tree-sitter AST code compression
pip install -e ".[memory]"      # + sentence-transformers + hnswlib vector memory
pip install -e ".[tokenizer]"   # + tiktoken (accurate token counts)
pip install -e ".[dev]"         # + test/lint tooling

Use it — in-process SDK

from reduction import TokenOptimizer

opt = TokenOptimizer()
req = opt.prepare(
    system="You are a security planner.",
    user=target_profile_json,          # per-target / volatile → user turn
    static_context=[taxonomy, schema], # reused → cacheable prefix
    output_format="toon",
)
resp = client.messages.create(
    model="claude-sonnet-4-6",
    system=req.system_blocks,          # cache_control already attached
    messages=req.messages,
    max_tokens=2048,
)
opt.record_usage(resp.usage)
data = opt.decode_output(resp.content[0].text, req.output_format)
print(opt.render())                    # token-savings report

Use it — zero-touch adapters

Wrap an existing client and change nothing else:

from reduction.adapters import OptimizedAnthropic
client = OptimizedAnthropic(api_key=...)      # same ctor as anthropic.Anthropic
resp = client.messages.create(model="claude-sonnet-4-6",
                              system="You plan.", messages=[...],
                              max_tokens=1024, output_format="toon")
print(client.optimizer.render())

OptimizedOpenAI does the same for openai.OpenAI.

Connect it to an agent — the odysseus example

The odysseus security agent (Agent_security_testing/Security_module) routes every call through ClaudeClient.message(...). One line wraps it — the whole scan gets caveman output, TOON serialization, normalized inputs, and savings metrics, with no change to the planner/synthesizer/triager call sites:

from reduction import TokenOptimizer
from reduction.adapters import wrap_message_fn

client = ClaudeClient()
opt = TokenOptimizer()
client.message = wrap_message_fn(client.message, opt, output_format="toon")

See examples/odysseus_integration.py (python examples/odysseus_integration.py runs an offline demo).

Use it — HTTP gateway

For non-Python agents or one shared service:

docker compose up --build         # gateway + redis-stack (semantic cache)
curl localhost:8000/v1/pipeline/chat -H 'content-type: application/json' \
  -d '{"user_message":"summarize failures","output_format":"toon"}'

Endpoints: /v1/pipeline/chat, /v1/optimize, /v1/encode/toon, /v1/metrics, /healthz.

Use it — MCP server (any MCP host)

The most universal "connect to an agent" path: instead of wrapping a client, the agent calls compression tools directly. Works with Claude Code, Cursor, or any MCP host.

pip install -e ".[mcp]"
reduction mcp                       # runs the stdio MCP server

// Claude Code / Cursor MCP config
{ "mcpServers": { "reduction": { "command": "reduction", "args": ["mcp"] } } }

Exposes reduction_compress (content-aware + CCR), reduction_retrieve (expand a ref), reduction_compress_history (shrink old conversation turns), reduction_fit_context (pack chunks into a token budget), reduction_route_effort (recommend a thinking budget), and reduction_stats (savings summary).

Use it — CLI

reduction init                          # print the one-line install snippet + detect SDKs
reduction compress scan.json            # content-aware compress, prints CCR ref
reduction retrieve 14a9cd0d             # expand a ref back to the original
reduction fit a.md b.md --budget 4000 --query "deploy"   # pack files into a budget
reduction history convo.json --keep-last 4               # compress old turns
reduction effort "debug why latency spiked"              # recommend a thinking budget
reduction simulate --daily-input-tokens 5000000
reduction wrap anthropic                # print a copy-paste integration snippet
reduction demo                          # compress a sample and show savings
reduction serve / reduction mcp         # gateway / MCP server
reduction proxy --port 8788             # OpenAI/Anthropic-compatible compression proxy
reduction memory add "..." / search "..."   # persistent vector memory
reduction learn --log f.jsonl --write CLAUDE.md   # failure-learning corrections

Advanced subsystems

These close the gap with full context-optimization platforms. All have dependency-free fallbacks, so they work before you install any extras.

Compression proxy (reduction/proxy.py)

A drop-in OpenAI- and Anthropic-compatible HTTP proxy. Point any client at it; it compresses large message content, injects the reduction_retrieve tool, and transparently satisfies retrieval tool calls from the CCR store so the client never sees the round-trip. Streaming (SSE) is supported: content tokens forward as they arrive, while reduction_retrieve tool-call events are buffered, resolved mid-stream, and the turn continues — all transparent to the client. Non-retrieval tool calls pass straight through.

pip install -e ".[proxy]"
OPENAI_BASE_URL=https://api.openai.com reduction proxy --port 8788
# point your client's base_url at http://127.0.0.1:8788

AST-aware code compression (reduction/layers/codecrush.py)

CODE content keeps imports, decorators, and class/function signatures while eliding bodies (... (12 lines)) — the agent sees the shape, retrieves a body via CCR when it needs one. tree-sitter ([code] extra) for language-exact parsing; robust Python/JS/TS/Go/Java/C++/Rust heuristic otherwise.

Conversation-history compression (reduction/layers/history.py)

For long-horizon agents the biggest token sink is not any single tool output but the accumulation of past turns — every tool result, file dump, and log is re-sent on every subsequent call. compress_messages shrinks the older messages (same content-aware compressors as Layer 1, CCR-reversible) while keeping system messages and the last keep_last turns verbatim, so the model retains full recent context and a retrievable history of everything before it. Handles both OpenAI string content and Anthropic block content (recursing into tool_result).

opt = TokenOptimizer()
messages = opt.compress_messages(conversation, keep_last=4)   # old turns shrunk + CCR
resp = client.messages.create(model="claude-sonnet-4-6", messages=messages, ...)

Inspired by ACON (context compression for long-horizon agents). Off by default; enable globally with REDUCTION_HISTORY=true.

Budget-aware context fitting (reduction/layers/contextfit.py)

RAG and agent loops assemble more candidate context than is worth sending. fit_context packs a list of chunks into a fixed token budget: it scores each chunk for relevance to a query (lexical, length-normalized), includes them in priority order while they fit, compresses (content-aware + CCR) any that don't, and truncates/drops the rest — reporting exactly what was trimmed so a budget-fit context is never mistaken for the full set.

opt = TokenOptimizer()
chunks = opt.fit_context(retrieved_docs, token_budget=4000, query=user_question)

Dependency-free analogue of Headroom's score-based context fitting.

In-process semantic cache (reduction/layers/semantic_cache.py)

Layer 3 without infrastructure: a prompt semantically close to a previous one (cosine ≥ REDUCTION_SEMANTIC_THRESHOLD) returns the cached response and skips the generation entirely. Embeddings use sentence-transformers when the [memory] extra is installed, else a dependency-free hashing embedding (lexical overlap — near-duplicate prompts hit, true paraphrases may not). Hits are counted in the metrics (semantic_cache_hits).

opt = TokenOptimizer(OptimizerConfig(semantic_cache=True))
text = opt.cached_call(user_msg, lambda: call_model(user_msg))  # hit skips the call

For a shared cross-process cache, the gateway's LiteLLM redis-semantic backend (configure_cache) does the same against Redis VSS / Qdrant.

Truncated-JSON recovery (reduction/layers/jsoncrush.py)

Captured tool output is often JSON cut off mid-stream, which fails to parse. The detector still classifies it as JSON and the compressor recovers the longest complete prefix (closing brackets at the last complete value), samples it as usual, and flags the result lossy so the raw capture stays retrievable via CCR. Unrecoverable JSON-ish text falls back to the text path.

Persistent vector memory (reduction/memory.py)

Per-project SQLite store with semantic search for cross-turn / cross-agent recall. Namespaced so projects never bleed into each other.

from reduction.memory import Memory
mem = Memory("proj.db", namespace="my-project")
mem.add("the deploy step needs AWS_PROFILE=prod", metadata={"src": "runbook"})
hits = mem.search("how do I deploy", k=3)

Real embeddings with [memory] (sentence-transformers); a deterministic hashing embedding otherwise. When hnswlib is installed, search uses an ANN index (built from SQLite on open, updated on add) for sub-linear lookups; otherwise it falls back to an exact cosine scan.

Effort routing (reduction/effort.py)

Extended thinking is the biggest output-side cost in agent loops — a model that "thinks" for 8k tokens before reading a file wastes budget. route_effort classifies a task and recommends a reasoning level plus the concrete provider knob: Anthropic thinking.budget_tokens (0 = disabled) or OpenAI reasoning_effort. Routine verbs (read/list/grep) route to minimal; analytical verbs (debug/why/design) route to high.

opt = TokenOptimizer()
d = opt.route_effort("read config.py")     # -> level="minimal", thinking omitted
resp = client.messages.create(model="claude-sonnet-4-6", thinking=d.anthropic_thinking(), ...)

Failure-learning (reduction/learn.py)

Record agent outcomes; recurring failures become corrections written into a managed block in CLAUDE.md / AGENTS.md, so the next run starts smarter.

from reduction.learn import FailureLog, write_corrections
log = FailureLog()
log.record(context="run tests", action="pytest -k foo", outcome="fail", error="no tests ran")
write_corrections("CLAUDE.md", log.derive_corrections(min_occurrences=2))

Batch-API CCR (reduction/ccr_batch.py)

Resolves reduction_retrieve tool calls that arrive in asynchronous Batch API results, producing continuation messages — CCR stays reversible even off the live request path.

Configuration

Every knob has an env-var fallback (see reduction/config.py):

Env var	Purpose	Default
REDUCTION_CAVEMAN	inject terse-output persona	true
REDUCTION_OUTPUT_FORMAT	text / toon / yaml	text
REDUCTION_SHELL_FILTER	filter tool output (Layer 1)	true
REDUCTION_CONTENT_ROUTING	content-aware tool-output compression	true
REDUCTION_CCR	reversible compression (store + retrieve refs)	true
REDUCTION_CCR_STORE	path to persist the CCR store as JSON	(memory)
REDUCTION_HISTORY	compress old conversation turns (keep recent verbatim)	false
REDUCTION_HISTORY_KEEP_LAST	turns kept verbatim by history compression	4
REDUCTION_COMPRESS	LLMLingua-2 (Layer 2)	false
REDUCTION_SEMANTIC_CACHE	LiteLLM semantic cache (Layer 3)	false
REDUCTION_SEMANTIC_THRESHOLD	cosine hit threshold	0.92
REDUCTION_NATIVE_CACHE	stable-prefix + cache_control (Layer 4)	true

Cost simulator

python simulator/simulate.py --daily-input-tokens 5000000 --daily-output-tokens 800000

Models the compounded savings against provider pricing as a before/after waterfall.

CI/CD

• ci.yml — ruff lint + format, pytest on Python 3.11/3.12, simulator smoke test.
• docker.yml — builds the gateway image on main, publishes to GHCR on v* tags.

Accuracy evaluation (does compression keep answers correct?)

Saving tokens is only safe if the model still answers correctly. The eval harness runs each case raw and compressed through an injectable model_fn and reports answer preservation alongside token savings, so you see the trade-off instead of guessing:

reduction eval        # offline self-check (synthetic log case)
# -> Answer preservation: 100.0%   Token savings: 98.8%

from reduction.evals import EvalCase, run_evals
report = run_evals(cases, model_fn)   # model_fn(context, question) -> answer
print(report.render())                # flags any REGRESSIONS

Wire model_fn to a real client to validate on your own traffic. This is the number that actually matters — a high savings % with a low preservation % means the compression is too aggressive for that content.

Honesty notes (what the metrics do and don't claim)

• Input savings are measured; output savings are not. Caveman/TOON shrink output, but we have no counterfactual (we never see the uncompressed generation), so the metrics report observed output tokens, never "output saved." Use the eval harness to quantify the output/accuracy effect.
• Token counts for Claude are approximate. tiktoken (cl100k/o200k) is exact for OpenAI; Anthropic's tokenizer isn't bundled, so Claude counts are a close proxy, not exact billing.
• codecrush uses a real tree-sitter parser when [code] is installed and a docstring-safe heuristic otherwise — both are honest about which ran.

Caveats (read before production)

• Compounded ≠ additive — measure end-to-end, don't multiply marketing numbers.
• Semantic cache can return a wrong answer for a subtly different query; keep the threshold high (≥0.92) on high-stakes paths.
• LLMLingua is itself a model — only worth running on large, reused context.
• TOON loses to JSON on deeply nested / non-uniform data; the encoder falls back automatically.
• Caveman output reads as terse — restrict it to machine/tool legs.
• CCR refs in an in-memory store don't survive a restart — set REDUCTION_CCR_STORE to a file path (SQLite-backed, O(1) puts; a legacy JSON store at that path is migrated in place) if a later process must retrieve them.
• Tool output is only re-run through zap for a read-only allowlist of commands (git status/log/diff, ls, grep, ...). Commands with side effects are never re-executed; their captured output goes through content-aware compression instead.

Credits

The content-aware compression, CCR (Compress-Cache-Retrieve), and MCP-tool design are inspired by Headroom (Apache-2.0). Reduction is an independent Python implementation of those ideas layered onto its own caveman/TOON/native-cache pipeline.

License

Apache-2.0