From e12662a9ee1a1625df11eee9107c315714567c0f Mon Sep 17 00:00:00 2001 From: lgoyal6 Date: Sun, 6 Sep 2026 16:11:26 -0700 Subject: [PATCH 1/2] Replace prose em dashes with hyphens in README and .gitignore Repository style is hyphens, and these two files were the last places in the tree still using em dashes in prose. Text only: 19 lines in README.md and the one comment line in .gitignore. No rules, no instructions and no code change, so the two files behave exactly as before. Split out from the packaging change on the next commit so that one is a readable diff rather than a page of punctuation. --- .gitignore | 2 +- README.md | 36 ++++++++++++++++++------------------ 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/.gitignore b/.gitignore index 8841967..79fb788 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,7 @@ venv/ .python-version -# Secrets — never commit +# Secrets - never commit .env .env.local .env.*.local diff --git a/README.md b/README.md index bc8efdb..2862558 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Winnow > Voice-first, real-time token compression for LLM pipelines. -> Speak, and we transcribe, prune, and pipe — losing words, keeping meaning. +> Speak, and we transcribe, prune, and pipe - losing words, keeping meaning. Winnow is a hackathon project that compresses live speech transcripts in real time before they reach an LLM, demonstrably without losing meaning. Spoken language is verbose; LLM input tokens cost money. Winnow shows you the savings, the diff, and a side-by-side fidelity audit, then lets you talk to the compressed transcript through a Claude-Projects-style workspace. @@ -12,14 +12,14 @@ Winnow is a hackathon project that compresses live speech transcripts in real ti Two views, one pipeline. ### Compare -Watch compression happen live. Raw transcript on the left as you speak, pruned transcript on the right within milliseconds of each pause. Pruned words are struck through. A stats bar tracks running token counts, % saved, $ saved per the chosen LLM, average pipeline latency, and a sparkline of compression ratio over time. A built-in A/B Q&A box asks the same question against both transcripts in parallel — when the answers match, you have proof compression preserved the buried details. +Watch compression happen live. Raw transcript on the left as you speak, pruned transcript on the right within milliseconds of each pause. Pruned words are struck through. A stats bar tracks running token counts, % saved, $ saved per the chosen LLM, average pipeline latency, and a sparkline of compression ratio over time. A built-in A/B Q&A box asks the same question against both transcripts in parallel - when the answers match, you have proof compression preserved the buried details. ### Learn -A voice-first Claude-Projects-style workspace built around the compressed transcript. Tap the mic, talk naturally, each pause auto-sends a chat message. Claude streams answers back. One-tap **Insights** generate a summary, decision log, action items, flashcards (flippable carousel), and glossary — all backed by Anthropic prompt caching so multi-turn sessions stay cheap. +A voice-first Claude-Projects-style workspace built around the compressed transcript. Tap the mic, talk naturally, each pause auto-sends a chat message. Claude streams answers back. One-tap **Insights** generate a summary, decision log, action items, flashcards (flippable carousel), and glossary - all backed by Anthropic prompt caching so multi-turn sessions stay cheap. ### Stage-safety features - **Swappable input source.** Live mic and a pre-recorded JSON fixture both implement the same `TranscriptSource` interface. One click swaps them; downstream code can't tell which is active. If the live mic ever fails on stage, the fallback is a literal replay of you doing the demo, with original utterance timing. -- **In-app fixture recorder.** Capture a live session, download as JSON, drop into `public/fixtures/demo-transcript.json` — and your fallback becomes you. +- **In-app fixture recorder.** Capture a live session, download as JSON, drop into `public/fixtures/demo-transcript.json` - and your fallback becomes you. - **Director mode.** Keyboard shortcuts (`Space` start/stop, `R` swap source, `1/2/3` fire preset Q&A probes, `X` clear, `?` show overlay) let you drive the demo without touching a trackpad. --- @@ -66,18 +66,18 @@ flowchart TD ``` **Token-space (which words survive).** The text fans out to two compressor -families at once — LLMLingua-2 (+ LongLLMLingua's query-aware reranker/perplexity +families at once - LLMLingua-2 (+ LongLLMLingua's query-aware reranker/perplexity arm) and AttentionRAG (attention-guided sentence pruning). Both produce per-token keep decisions, which Winnow merges token-by-token over the *original* text under a -user-picked boolean rule: **intersection** (keep only if both kept — aggressive) or -**union** (keep if either kept — recall-safe). The merge aligns both methods to one +user-picked boolean rule: **intersection** (keep only if both kept - aggressive) or +**union** (keep if either kept - recall-safe). The merge aligns both methods to one canonical token sequence rather than doing string membership, and falls back to LLMLingua-only if AttentionRAG gates everything out. **Model-space (how it's answered).** The compressed prompt then goes either to a **black-box API** (Claude / GPT, provider-agnostic), or to our **self-hosted** workers: **Qwen + TurboQuant** (KV-cache quantized to ~4 bits, 3–4× smaller via a -data-free random-rotation `DynamicCache`), or **LCLM + TurboQuant** — LCLM +data-free random-rotation `DynamicCache`), or **LCLM + TurboQuant** - LCLM compresses the context into a few latent soft tokens (fewer KV *entries*) while TurboQuant compresses the *bits per entry* of the same decoder cache, so the savings multiply. @@ -138,7 +138,7 @@ multiply. **Compression** - LLMLingua-2 (`microsoft/llmlingua-2-xlm-roberta-large-meetingbank`) -- Hosted on Modal — T4 GPU, persistent HF cache volume, memory + GPU snapshots for ~1s cold start +- Hosted on Modal - T4 GPU, persistent HF cache volume, memory + GPU snapshots for ~1s cold start - FastAPI / Pydantic / Uvicorn as a thin local proxy **LLM** @@ -191,14 +191,14 @@ COMPRESS_BACKEND_URL=http://localhost:8000 # default, leave as-is ### 4. Run everything -In one terminal — the GPU worker proxy: +In one terminal - the GPU worker proxy: ```bash # from repo root ./run.sh # deploys (if needed) + warms + serves FastAPI on :8000 ``` Honors `SKIP_DEPLOY=1` and `SKIP_WARMUP=1` for faster restarts during dev. -In another terminal — the frontend: +In another terminal - the frontend: ```bash cd web npm run dev # http://localhost:3000 @@ -212,15 +212,15 @@ Open http://localhost:3000. 1. Click **Start** in the top right. The browser asks for mic permission. 2. Talk. Watch the raw column fill on the left, compressed on the right with strike-throughs on dropped words. Stats bar updates live. -3. Hit one of the preset probes under the **Q&A** box (or type a question) — answers fire against raw and compressed in parallel. A green "answers match" badge is the proof. +3. Hit one of the preset probes under the **Q&A** box (or type a question) - answers fire against raw and compressed in parallel. A green "answers match" badge is the proof. 4. Switch to **Learn**. The compressed transcript is auto-pinned as a source. -5. Tap the big neon mic. Talk to Claude about what was said. Each pause auto-sends. Tap any **Insight** card to generate flashcards, action items, glossary, etc. — results stay cached. +5. Tap the big neon mic. Talk to Claude about what was said. Each pause auto-sends. Tap any **Insight** card to generate flashcards, action items, glossary, etc. - results stay cached. ### Stage-safety toggles -- **Live mic ⇄ Recorded** at the top — instant swap. Recorded plays `web/public/fixtures/demo-transcript.json` with original timing. -- **Rec fixture / Save** — capture a fresh recorded fallback from your current live session and download as JSON. Replace `public/fixtures/demo-transcript.json` with it so the fallback is literally you. -- **Director** button or `?` key — pulls up the keyboard cheat sheet. +- **Live mic ⇄ Recorded** at the top - instant swap. Recorded plays `web/public/fixtures/demo-transcript.json` with original timing. +- **Rec fixture / Save** - capture a fresh recorded fallback from your current live session and download as JSON. Replace `public/fixtures/demo-transcript.json` with it so the fallback is literally you. +- **Director** button or `?` key - pulls up the keyboard cheat sheet. --- @@ -230,13 +230,13 @@ Open http://localhost:3000. Your Deepgram API key doesn't have project-admin scope, so `/v1/auth/grant` rejects it. Winnow's `/api/deepgram-token` falls back to shipping the raw key to the browser (fine for local). Make sure `DEEPGRAM_API_KEY` is set in `web/.env`. Restart `npm run dev` after editing. **Chat error: `invalid x-api-key` / `authentication_error`** -`ANTHROPIC_API_KEY` is missing, malformed, or stale. Edit `web/.env`, **then restart `npm run dev`** — Next only reads env vars at boot. +`ANTHROPIC_API_KEY` is missing, malformed, or stale. Edit `web/.env`, **then restart `npm run dev`** - Next only reads env vars at boot. **`backend unreachable` on compress** FastAPI isn't running. Start it: `./run.sh` from the repo root. **Modal call fails / cold start is slow** -First call after a deploy or a long idle period builds the GPU snapshot — that's ~30s once, then restores in ~1s. `./run.sh` automatically warms the worker after deploy. +First call after a deploy or a long idle period builds the GPU snapshot - that's ~30s once, then restores in ~1s. `./run.sh` automatically warms the worker after deploy. **Port 3000 already in use** A previous dev server is still up. `lsof -i :3000` to find the PID, then `kill `. Or use `npm run dev -- -p 3001`. From 02d6e656de34817342521fff3c87d4cb3a2f16c1 Mon Sep 17 00:00:00 2001 From: lgoyal6 Date: Sun, 6 Sep 2026 16:11:39 -0700 Subject: [PATCH 2/2] Package the CPU-side core as winnow-core, and lock what the GPU half needs The compression core is the part of this repo an outside consumer can actually use: the AttentionRAG selection logic, the two-compressor token merge, and the model-artifact guard. None of it needs a GPU, a Modal account or an API key, but until now the only way to get it was to clone the repo and hope the imports resolved. pyproject.toml builds it as `winnow-core` with setuptools, exporting the attentionrag package plus the token_merge and model_guard top-level modules. dependencies is deliberately empty: attentionrag.core is the paper's selection logic behind a Backend protocol, token_merge is pure string and span arithmetic, and model_guard's validation path uses only pickletools, hashlib, json and zipfile. That is what makes this half installable and testable without a GPU. Verified rather than assumed, on this machine: python -m build -> winnow_core-0.1.0-py3-none-any.whl pip install in a clean venv -> pip freeze lists winnow-core and nothing else cd /tmp && python -m attentionrag.test_core -> 10/10 core tests passed import token_merge, model_guard, attentionrag -> all resolve from site-packages The model-touching half goes in an [hf] extra, pinned rather than floated: the guard's whole point is that "the same model name" must not mean different bytes tomorrow, and the same argument applies to the loader. numpy is held at 2.4.6 rather than the newest release because safetensors needs numpy to convert torch tensors and numpy 2.5.x publishes no cp311 wheel, which is the interpreter the Modal images use. requirements-hf.lock is the pip freeze of that extra in a clean 3.11 venv and records the reproduce command, so the resolved set is a fact rather than a wish. requirements.txt is the separate Modal/FastAPI client freeze for the local proxy in server.py. .gitignore picks up /dist/, /build/ and *.egg-info/ so a local build does not show up as untracked noise. One deliberate correction while landing this: the Homepage URL read https://github.com/winnow, which is not this repository and does not resolve. It now points at https://github.com/TC960/winnow. --- .gitignore | 5 +++++ README.md | 26 ++++++++++++++++++++++ pyproject.toml | 52 ++++++++++++++++++++++++++++++++++++++++++++ requirements-hf.lock | 48 ++++++++++++++++++++++++++++++++++++++++ requirements.txt | 43 ++++++++++++++++++++++++++++++++++++ 5 files changed, 174 insertions(+) create mode 100644 pyproject.toml create mode 100644 requirements-hf.lock create mode 100644 requirements.txt diff --git a/.gitignore b/.gitignore index 79fb788..35b8e79 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,11 @@ venv/ experiments/eval_results.json experiments/eval_outputs*.json +# Python build artifacts (winnow-core wheel/sdist) +/dist/ +/build/ +*.egg-info/ + # Misc .DS_Store *.log diff --git a/README.md b/README.md index 2862558..9abcc85 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,32 @@ multiply. --- +## Installing just the compression core (no GPU, no Modal, no keys) + +The CPU-side half of Winnow ships as a standalone distribution, `winnow-core`. +It is the part an external consumer can actually use without a GPU: the +AttentionRAG selection logic (`attentionrag`), the two-compressor token merge +(`token_merge`), and the model-artifact guard (`model_guard`). + +Prerequisites: Python 3.11, 3.12 or 3.14. Nothing else - the core has zero +runtime dependencies. + +```bash +python3 -m venv .venv && . .venv/bin/activate +pip install build && python -m build # writes dist/winnow_core-0.1.0-*.whl +pip install dist/winnow_core-0.1.0-py3-none-any.whl +python -m attentionrag.test_core # 10/10 +``` + +The model-touching half (`attentionrag.hf_backend`, and reading real tensors +through `model_guard.safe_load_state_dict`) needs torch and transformers: + +```bash +pip install "dist/winnow_core-0.1.0-py3-none-any.whl[hf]" +``` + +Exact resolved versions are recorded in `requirements-hf.lock`. + ## Local setup Prerequisites: diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..4f4aa5f --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,52 @@ +[build-system] +requires = ["setuptools>=68", "wheel"] +build-backend = "setuptools.build_meta" + +[project] +name = "winnow-core" +version = "0.1.0" +description = "Winnow's CPU-side compression core: AttentionRAG selection, the two-compressor token merge, and the model-artifact guard." +readme = "README.md" +requires-python = ">=3.11" +license = { text = "MIT" } +authors = [{ name = "Winnow" }] +keywords = ["prompt-compression", "rag", "kv-cache", "llm"] +classifiers = [ + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.14", + "Topic :: Scientific/Engineering :: Artificial Intelligence", +] + +# The core is deliberately STDLIB ONLY. `attentionrag.core` is the paper's +# selection logic behind a Backend protocol, `token_merge` is pure string/span +# arithmetic, and `model_guard`'s validation path uses pickletools/hashlib/ +# zipfile. That is what makes this half installable and testable without a GPU. +dependencies = [] + +[project.optional-dependencies] +# Only needed for `attentionrag.hf_backend` (the model-touching half) and for +# `model_guard.safe_load_state_dict` reading real tensors. Pinned, not floated: +# the guard's whole point is that "the same model name" must not mean different +# bytes tomorrow, and the same argument applies to the loader. +# numpy is pinned to 2.4.6, not the newest release: safetensors needs numpy to +# convert torch tensors, and numpy 2.5.x publishes no cp311 wheel, which is the +# interpreter the Modal images use. Verified by installing, not by reading. +hf = [ + "torch==2.14.0", + "transformers==5.16.1", + "safetensors==0.8.0", + "huggingface-hub==1.30.0", + "accelerate==1.14.0", + "numpy==2.4.6", +] + +[project.urls] +Homepage = "https://github.com/TC960/winnow" + +[tool.setuptools] +py-modules = ["model_guard", "token_merge"] + +[tool.setuptools.packages.find] +include = ["attentionrag*"] +exclude = ["attentionrag.test_*"] diff --git a/requirements-hf.lock b/requirements-hf.lock new file mode 100644 index 0000000..3a35216 --- /dev/null +++ b/requirements-hf.lock @@ -0,0 +1,48 @@ +# Lockfile for `pip install "winnow-core[hf]"` - the model-touching half. +# +# Produced by `pip freeze` in a CLEAN python 3.11 venv after installing the +# built wheel with the [hf] extra, on 2026-09-05. Not hand-written: every line +# below is a package pip actually resolved and installed. +# +# The CORE install (`pip install winnow-core`, no extra) needs NOTHING from this +# file - it is stdlib-only, and its freeze is one line: winnow-core itself. +# +# Reproduce: +# python3.11 -m venv .venv && . .venv/bin/activate +# pip install "dist/winnow_core-0.1.0-py3-none-any.whl[hf]" +# pip freeze +accelerate==1.14.0 +annotated-doc==0.0.5 +anyio==4.15.1 +certifi==2026.7.22 +click==8.5.0 +filelock==3.32.5 +fsspec==2026.7.0 +h11==0.16.0 +hf-xet==1.6.0 +httpcore==1.0.9 +httpx==0.28.1 +huggingface_hub==1.30.0 +idna==3.19 +Jinja2==3.1.6 +markdown-it-py==4.2.0 +MarkupSafe==3.0.3 +mdurl==0.1.2 +mpmath==1.3.0 +networkx==3.6.1 +numpy==2.4.6 +packaging==26.3 +psutil==7.2.2 +Pygments==2.21.0 +PyYAML==6.0.3 +regex==2026.9.3 +rich==15.0.0 +safetensors==0.8.0 +shellingham==1.5.4 +sympy==1.14.0 +tokenizers==0.23.2 +torch==2.14.0 +tqdm==4.70.0 +transformers==5.16.1 +typer==0.27.2 +typing_extensions==4.16.0 diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..9b39034 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,43 @@ +aiohappyeyeballs==2.6.2 +aiohttp==3.14.3 +aiosignal==1.4.0 +annotated-doc==0.0.4 +annotated-types==0.7.0 +anyio==4.14.0 +attrs==26.1.0 +cbor2==6.1.2 +certifi==2026.6.17 +click==8.4.1 +fastapi==0.138.0 +frozenlist==1.8.0 +grpclib==0.4.9 +h11==0.16.0 +h2==4.3.0 +hpack==4.1.0 +httptools==0.8.0 +hyperframe==6.1.0 +idna==3.18 +markdown-it-py==4.2.0 +mdurl==0.1.2 +modal==1.5.0 +multidict==6.7.1 +propcache==0.5.2 +protobuf==6.33.6 +pydantic==2.13.4 +pydantic_core==2.46.4 +Pygments==2.20.0 +python-dotenv==1.2.2 +PyYAML==6.0.3 +rich==15.0.0 +starlette==1.3.1 +synchronicity==0.12.5 +toml==0.10.2 +types-certifi==2021.10.8.3 +types-toml==0.10.8.20260518 +typing-inspection==0.4.2 +typing_extensions==4.15.0 +uvicorn==0.49.0 +uvloop==0.22.1 +watchfiles==1.2.0 +websockets==16.0 +yarl==1.24.2