diff --git a/.gitignore b/.gitignore index 8841967..35b8e79 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,7 @@ venv/ .python-version -# Secrets — never commit +# Secrets - never commit .env .env.local .env.*.local @@ -20,6 +20,11 @@ venv/ experiments/eval_results.json experiments/eval_outputs*.json +# Python build artifacts (winnow-core wheel/sdist) +/dist/ +/build/ +*.egg-info/ + # Misc .DS_Store *.log diff --git a/README.md b/README.md index bc8efdb..9abcc85 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Winnow > Voice-first, real-time token compression for LLM pipelines. -> Speak, and we transcribe, prune, and pipe — losing words, keeping meaning. +> Speak, and we transcribe, prune, and pipe - losing words, keeping meaning. Winnow is a hackathon project that compresses live speech transcripts in real time before they reach an LLM, demonstrably without losing meaning. Spoken language is verbose; LLM input tokens cost money. Winnow shows you the savings, the diff, and a side-by-side fidelity audit, then lets you talk to the compressed transcript through a Claude-Projects-style workspace. @@ -12,14 +12,14 @@ Winnow is a hackathon project that compresses live speech transcripts in real ti Two views, one pipeline. ### Compare -Watch compression happen live. Raw transcript on the left as you speak, pruned transcript on the right within milliseconds of each pause. Pruned words are struck through. A stats bar tracks running token counts, % saved, $ saved per the chosen LLM, average pipeline latency, and a sparkline of compression ratio over time. A built-in A/B Q&A box asks the same question against both transcripts in parallel — when the answers match, you have proof compression preserved the buried details. +Watch compression happen live. Raw transcript on the left as you speak, pruned transcript on the right within milliseconds of each pause. Pruned words are struck through. A stats bar tracks running token counts, % saved, $ saved per the chosen LLM, average pipeline latency, and a sparkline of compression ratio over time. A built-in A/B Q&A box asks the same question against both transcripts in parallel - when the answers match, you have proof compression preserved the buried details. ### Learn -A voice-first Claude-Projects-style workspace built around the compressed transcript. Tap the mic, talk naturally, each pause auto-sends a chat message. Claude streams answers back. One-tap **Insights** generate a summary, decision log, action items, flashcards (flippable carousel), and glossary — all backed by Anthropic prompt caching so multi-turn sessions stay cheap. +A voice-first Claude-Projects-style workspace built around the compressed transcript. Tap the mic, talk naturally, each pause auto-sends a chat message. Claude streams answers back. One-tap **Insights** generate a summary, decision log, action items, flashcards (flippable carousel), and glossary - all backed by Anthropic prompt caching so multi-turn sessions stay cheap. ### Stage-safety features - **Swappable input source.** Live mic and a pre-recorded JSON fixture both implement the same `TranscriptSource` interface. One click swaps them; downstream code can't tell which is active. If the live mic ever fails on stage, the fallback is a literal replay of you doing the demo, with original utterance timing. -- **In-app fixture recorder.** Capture a live session, download as JSON, drop into `public/fixtures/demo-transcript.json` — and your fallback becomes you. +- **In-app fixture recorder.** Capture a live session, download as JSON, drop into `public/fixtures/demo-transcript.json` - and your fallback becomes you. - **Director mode.** Keyboard shortcuts (`Space` start/stop, `R` swap source, `1/2/3` fire preset Q&A probes, `X` clear, `?` show overlay) let you drive the demo without touching a trackpad. --- @@ -66,18 +66,18 @@ flowchart TD ``` **Token-space (which words survive).** The text fans out to two compressor -families at once — LLMLingua-2 (+ LongLLMLingua's query-aware reranker/perplexity +families at once - LLMLingua-2 (+ LongLLMLingua's query-aware reranker/perplexity arm) and AttentionRAG (attention-guided sentence pruning). Both produce per-token keep decisions, which Winnow merges token-by-token over the *original* text under a -user-picked boolean rule: **intersection** (keep only if both kept — aggressive) or -**union** (keep if either kept — recall-safe). The merge aligns both methods to one +user-picked boolean rule: **intersection** (keep only if both kept - aggressive) or +**union** (keep if either kept - recall-safe). The merge aligns both methods to one canonical token sequence rather than doing string membership, and falls back to LLMLingua-only if AttentionRAG gates everything out. **Model-space (how it's answered).** The compressed prompt then goes either to a **black-box API** (Claude / GPT, provider-agnostic), or to our **self-hosted** workers: **Qwen + TurboQuant** (KV-cache quantized to ~4 bits, 3–4× smaller via a -data-free random-rotation `DynamicCache`), or **LCLM + TurboQuant** — LCLM +data-free random-rotation `DynamicCache`), or **LCLM + TurboQuant** - LCLM compresses the context into a few latent soft tokens (fewer KV *entries*) while TurboQuant compresses the *bits per entry* of the same decoder cache, so the savings multiply. @@ -138,7 +138,7 @@ multiply. **Compression** - LLMLingua-2 (`microsoft/llmlingua-2-xlm-roberta-large-meetingbank`) -- Hosted on Modal — T4 GPU, persistent HF cache volume, memory + GPU snapshots for ~1s cold start +- Hosted on Modal - T4 GPU, persistent HF cache volume, memory + GPU snapshots for ~1s cold start - FastAPI / Pydantic / Uvicorn as a thin local proxy **LLM** @@ -147,6 +147,32 @@ multiply. --- +## Installing just the compression core (no GPU, no Modal, no keys) + +The CPU-side half of Winnow ships as a standalone distribution, `winnow-core`. +It is the part an external consumer can actually use without a GPU: the +AttentionRAG selection logic (`attentionrag`), the two-compressor token merge +(`token_merge`), and the model-artifact guard (`model_guard`). + +Prerequisites: Python 3.11, 3.12 or 3.14. Nothing else - the core has zero +runtime dependencies. + +```bash +python3 -m venv .venv && . .venv/bin/activate +pip install build && python -m build # writes dist/winnow_core-0.1.0-*.whl +pip install dist/winnow_core-0.1.0-py3-none-any.whl +python -m attentionrag.test_core # 10/10 +``` + +The model-touching half (`attentionrag.hf_backend`, and reading real tensors +through `model_guard.safe_load_state_dict`) needs torch and transformers: + +```bash +pip install "dist/winnow_core-0.1.0-py3-none-any.whl[hf]" +``` + +Exact resolved versions are recorded in `requirements-hf.lock`. + ## Local setup Prerequisites: @@ -191,14 +217,14 @@ COMPRESS_BACKEND_URL=http://localhost:8000 # default, leave as-is ### 4. Run everything -In one terminal — the GPU worker proxy: +In one terminal - the GPU worker proxy: ```bash # from repo root ./run.sh # deploys (if needed) + warms + serves FastAPI on :8000 ``` Honors `SKIP_DEPLOY=1` and `SKIP_WARMUP=1` for faster restarts during dev. -In another terminal — the frontend: +In another terminal - the frontend: ```bash cd web npm run dev # http://localhost:3000 @@ -212,15 +238,15 @@ Open http://localhost:3000. 1. Click **Start** in the top right. The browser asks for mic permission. 2. Talk. Watch the raw column fill on the left, compressed on the right with strike-throughs on dropped words. Stats bar updates live. -3. Hit one of the preset probes under the **Q&A** box (or type a question) — answers fire against raw and compressed in parallel. A green "answers match" badge is the proof. +3. Hit one of the preset probes under the **Q&A** box (or type a question) - answers fire against raw and compressed in parallel. A green "answers match" badge is the proof. 4. Switch to **Learn**. The compressed transcript is auto-pinned as a source. -5. Tap the big neon mic. Talk to Claude about what was said. Each pause auto-sends. Tap any **Insight** card to generate flashcards, action items, glossary, etc. — results stay cached. +5. Tap the big neon mic. Talk to Claude about what was said. Each pause auto-sends. Tap any **Insight** card to generate flashcards, action items, glossary, etc. - results stay cached. ### Stage-safety toggles -- **Live mic ⇄ Recorded** at the top — instant swap. Recorded plays `web/public/fixtures/demo-transcript.json` with original timing. -- **Rec fixture / Save** — capture a fresh recorded fallback from your current live session and download as JSON. Replace `public/fixtures/demo-transcript.json` with it so the fallback is literally you. -- **Director** button or `?` key — pulls up the keyboard cheat sheet. +- **Live mic ⇄ Recorded** at the top - instant swap. Recorded plays `web/public/fixtures/demo-transcript.json` with original timing. +- **Rec fixture / Save** - capture a fresh recorded fallback from your current live session and download as JSON. Replace `public/fixtures/demo-transcript.json` with it so the fallback is literally you. +- **Director** button or `?` key - pulls up the keyboard cheat sheet. --- @@ -230,13 +256,13 @@ Open http://localhost:3000. Your Deepgram API key doesn't have project-admin scope, so `/v1/auth/grant` rejects it. Winnow's `/api/deepgram-token` falls back to shipping the raw key to the browser (fine for local). Make sure `DEEPGRAM_API_KEY` is set in `web/.env`. Restart `npm run dev` after editing. **Chat error: `invalid x-api-key` / `authentication_error`** -`ANTHROPIC_API_KEY` is missing, malformed, or stale. Edit `web/.env`, **then restart `npm run dev`** — Next only reads env vars at boot. +`ANTHROPIC_API_KEY` is missing, malformed, or stale. Edit `web/.env`, **then restart `npm run dev`** - Next only reads env vars at boot. **`backend unreachable` on compress** FastAPI isn't running. Start it: `./run.sh` from the repo root. **Modal call fails / cold start is slow** -First call after a deploy or a long idle period builds the GPU snapshot — that's ~30s once, then restores in ~1s. `./run.sh` automatically warms the worker after deploy. +First call after a deploy or a long idle period builds the GPU snapshot - that's ~30s once, then restores in ~1s. `./run.sh` automatically warms the worker after deploy. **Port 3000 already in use** A previous dev server is still up. `lsof -i :3000` to find the PID, then `kill `. Or use `npm run dev -- -p 3001`. diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..4f4aa5f --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,52 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "winnow-core" +version = "0.1.0" +description = "Winnow's CPU-side compression core: AttentionRAG selection, the two-compressor token merge, and the model-artifact guard." +readme = "README.md" +requires-python = ">=3.11" +license = { text = "MIT" } +authors = [{ name = "Winnow" }] +keywords = ["prompt-compression", "rag", "kv-cache", "llm"] +classifiers = [ + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.14", + "Topic :: Scientific/Engineering :: Artificial Intelligence", +] + +# The core is deliberately STDLIB ONLY. `attentionrag.core` is the paper's +# selection logic behind a Backend protocol, `token_merge` is pure string/span +# arithmetic, and `model_guard`'s validation path uses pickletools/hashlib/ +# zipfile. That is what makes this half installable and testable without a GPU. +dependencies = [] + +[project.optional-dependencies] +# Only needed for `attentionrag.hf_backend` (the model-touching half) and for +# `model_guard.safe_load_state_dict` reading real tensors. Pinned, not floated: +# the guard's whole point is that "the same model name" must not mean different +# bytes tomorrow, and the same argument applies to the loader. +# numpy is pinned to 2.4.6, not the newest release: safetensors needs numpy to +# convert torch tensors, and numpy 2.5.x publishes no cp311 wheel, which is the +# interpreter the Modal images use. Verified by installing, not by reading. +hf = [ + "torch==2.14.0", + "transformers==5.16.1", + "safetensors==0.8.0", + "huggingface-hub==1.30.0", + "accelerate==1.14.0", + "numpy==2.4.6", +] + +[project.urls] +Homepage = "https://github.com/TC960/winnow" + +[tool.setuptools] +py-modules = ["model_guard", "token_merge"] + +[tool.setuptools.packages.find] +include = ["attentionrag*"] +exclude = ["attentionrag.test_*"] diff --git a/requirements-hf.lock b/requirements-hf.lock new file mode 100644 index 0000000..3a35216 --- /dev/null +++ b/requirements-hf.lock @@ -0,0 +1,48 @@ +# Lockfile for `pip install "winnow-core[hf]"` - the model-touching half. +# +# Produced by `pip freeze` in a CLEAN python 3.11 venv after installing the +# built wheel with the [hf] extra, on 2026-09-05. Not hand-written: every line +# below is a package pip actually resolved and installed. +# +# The CORE install (`pip install winnow-core`, no extra) needs NOTHING from this +# file - it is stdlib-only, and its freeze is one line: winnow-core itself. +# +# Reproduce: +# python3.11 -m venv .venv && . .venv/bin/activate +# pip install "dist/winnow_core-0.1.0-py3-none-any.whl[hf]" +# pip freeze +accelerate==1.14.0 +annotated-doc==0.0.5 +anyio==4.15.1 +certifi==2026.7.22 +click==8.5.0 +filelock==3.32.5 +fsspec==2026.7.0 +h11==0.16.0 +hf-xet==1.6.0 +httpcore==1.0.9 +httpx==0.28.1 +huggingface_hub==1.30.0 +idna==3.19 +Jinja2==3.1.6 +markdown-it-py==4.2.0 +MarkupSafe==3.0.3 +mdurl==0.1.2 +mpmath==1.3.0 +networkx==3.6.1 +numpy==2.4.6 +packaging==26.3 +psutil==7.2.2 +Pygments==2.21.0 +PyYAML==6.0.3 +regex==2026.9.3 +rich==15.0.0 +safetensors==0.8.0 +shellingham==1.5.4 +sympy==1.14.0 +tokenizers==0.23.2 +torch==2.14.0 +tqdm==4.70.0 +transformers==5.16.1 +typer==0.27.2 +typing_extensions==4.16.0 diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..9b39034 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,43 @@ +aiohappyeyeballs==2.6.2 +aiohttp==3.14.3 +aiosignal==1.4.0 +annotated-doc==0.0.4 +annotated-types==0.7.0 +anyio==4.14.0 +attrs==26.1.0 +cbor2==6.1.2 +certifi==2026.6.17 +click==8.4.1 +fastapi==0.138.0 +frozenlist==1.8.0 +grpclib==0.4.9 +h11==0.16.0 +h2==4.3.0 +hpack==4.1.0 +httptools==0.8.0 +hyperframe==6.1.0 +idna==3.18 +markdown-it-py==4.2.0 +mdurl==0.1.2 +modal==1.5.0 +multidict==6.7.1 +propcache==0.5.2 +protobuf==6.33.6 +pydantic==2.13.4 +pydantic_core==2.46.4 +Pygments==2.20.0 +python-dotenv==1.2.2 +PyYAML==6.0.3 +rich==15.0.0 +starlette==1.3.1 +synchronicity==0.12.5 +toml==0.10.2 +types-certifi==2021.10.8.3 +types-toml==0.10.8.20260518 +typing-inspection==0.4.2 +typing_extensions==4.15.0 +uvicorn==0.49.0 +uvloop==0.22.1 +watchfiles==1.2.0 +websockets==16.0 +yarl==1.24.2