Phase 2A: real local transcription via whisper.cpp with a queued local worker

.gitignore

@@ -25,6 +25,38 @@ __pycache__/
 backend/storage/recordings/**
 !backend/storage/recordings/.gitkeep
 
+# --- Phase 2A: external whisper.cpp runtime (source, build tree, binaries) ---
+# Cloned and built locally by scripts/setup_whisper.sh. Never vendored: see
+# docs/adr/0009-pinned-whisper-cpp-runtime.md.
+backend/runtime/**
+!backend/runtime/.gitkeep
+
+# --- Phase 2A: model weights (multi-hundred-MB, never committed) ---
+backend/storage/models/**
+!backend/storage/models/.gitkeep
+
+# --- Phase 2A: per-job temporary decoded PCM (your speech in the clear) ---
+backend/storage/processing/**
+!backend/storage/processing/.gitkeep
+
+# --- Phase 2A: local benchmark inputs and output (real recordings) ---
+backend/storage/benchmarks/**
+!backend/storage/benchmarks/.gitkeep
+
+# --- Audio, anywhere (no real recording is ever a tracked fixture) ---
+*.wav
+*.webm
+*.ogg
+*.oga
+*.m4a
+*.mp3
+*.flac
+
+# --- Core ML / quantized model artifacts (generated, never committed) ---
+*.mlmodel
+*.mlmodelc/
+*.mlpackage/
+
 # --- Local application logs ---
 *.log
 logs/

AGENTS.md

@@ -36,7 +36,10 @@ it to learn software architecture, so how changes are made matters as much as wh
   task, mention it or file it separately — don't fold it into an unrelated change.
 - **Follow the existing provider-interface pattern** when adding or changing AI-adjacent
   functionality (`TranscriptionProvider`, `MetricsProvider`, `FeedbackProvider`) — new
-  implementations go behind the existing interface, not as special-cased calls elsewhere.
+  implementations go behind the existing interface, not as special-cased calls elsewhere. If a
+  real implementation genuinely needs a different signature than the mock, **change the interface
+  deliberately and document why** rather than smuggling data in through the side — see the
+  correction in [docs/AI_MODEL_STRATEGY.md](docs/AI_MODEL_STRATEGY.md).
 - Match the existing test discipline: any new backend logic gets a `pytest` test, any new
   frontend logic gets a `vitest` test where practical (see [docs/TESTING.md](docs/TESTING.md) for
   what "where practical" has meant so far).
@@ -70,8 +73,32 @@ the same change. Test fixtures must be synthetic, never real personal data.
 
 ## Explicit scope guardrails
 
-Do not add, without the owner's explicit approval: real Whisper integration, ffmpeg, Ollama, RAG,
-embeddings, vector databases, agents, Tauri packaging, Docker, cloud infrastructure, user
-accounts, Playwright/e2e browser automation, or webcam processing. See
+**Approved and now present** (Phase 2A, with the owner's explicit approval): real whisper.cpp
+transcription, ffmpeg, and a local worker process with a SQLite job queue. See ADRs
+[0006](docs/adr/0006-local-worker-sqlite-queue.md),
+[0007](docs/adr/0007-ffmpeg-preprocessing-boundary.md),
+[0008](docs/adr/0008-http-polling-not-websockets.md), and
+[0009](docs/adr/0009-pinned-whisper-cpp-runtime.md).
+
+Still **not** to be added without the owner's explicit approval: Ollama, RAG, embeddings, vector
+databases, agents, Tauri packaging, Docker, cloud infrastructure, user accounts, Playwright/e2e
+browser automation, webcam processing, or a message broker (Redis/RabbitMQ/Celery/SQS — see
+[ADR 0006](docs/adr/0006-local-worker-sqlite-queue.md) for why SQLite is the queue). See
 [docs/ROADMAP.md](docs/ROADMAP.md) for what's actually planned and why each of these is
 sequenced where it is (or not planned at all).
+
+## Rules that came out of Phase 2A
+
+- **Never silently substitute a mock for a real provider.** If whisper.cpp is unavailable, the job
+  fails with an actionable error code. Putting simulated text under a UI label reading "Real
+  transcript" would be the single worst bug this system could have.
+- **Never fabricate a measurement.** No benchmark number, latency figure, accuracy claim, or
+  "verified" statement goes in this repository unless it was actually produced on real hardware.
+  If something couldn't be run, say which prerequisite was missing.
+- **All external processes go through `app/subprocess_util.py: run_command`** — argv arrays, never
+  `shell=True`, always with a timeout.
+- **Never let subprocess output or absolute paths reach the API, the job table, or a log.** Route
+  it through `app/safe_text.py` first; absolute paths contain the owner's username.
+- **New runtime artifact kinds need a `.gitignore` entry AND a
+  `scripts/check_public_safety.sh` rule in the same change.**
+- **Test fixtures stay synthetic.** Generate silent WAVs arithmetically; never commit audio.

README.md

@@ -8,20 +8,29 @@ Built as a personal project to get better at speaking *and* to practice building
 architecture: local AI inference boundaries, audio handling, a real state machine, asynchronous
 job design, and honest error handling — not just a UI over a mock API.
 
-## Status: Phase 1
+## Status: Phase 2A
 
 The full local practice workflow — track/topic/mode selection, timers, browser microphone
 recording, review, self-reflection, results, and history — is built and working, backed by SQLite
-and local file storage. **The transcript, metrics, and feedback you see are simulated**, clearly
-labeled as such throughout the UI. No real speech-to-text or speech analysis has been built yet —
-see [docs/SCORING_AND_LIMITATIONS.md](docs/SCORING_AND_LIMITATIONS.md).
+and local file storage.
 
-See [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) for exact test results, known limitations,
-and what's next.
+**The transcript is now real**: local speech-to-text via a pinned whisper.cpp build, running
+entirely on your machine with no network access. **Metrics and feedback are still simulated**, and
+each section of the results screen is labeled with its own provenance rather than a single
+page-wide disclaimer — see
+[docs/SCORING_AND_LIMITATIONS.md](docs/SCORING_AND_LIMITATIONS.md).
+
+Transcription runs in a **separate worker process** reading a durable SQLite job queue, because it
+is far too slow to hold an HTTP request open for. Uploads return `202 Accepted` and the frontend
+polls for progress.
+
+See [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) for exact test results and known limitations
+— including, honestly, that no whisper.cpp build or real-model run has yet happened in this
+repository's build environment.
 
 ## Quick start
 
-Requires Python 3.11+ and Node 20+. Two processes, both local, nothing cloud-hosted:
+Requires Python 3.11+ and Node 20+. **Three** local processes, nothing cloud-hosted:
 
 ```bash
 git clone <repository-url>
@@ -34,6 +43,12 @@ pip install -r requirements.txt
 uvicorn app.main:app --reload --host 127.0.0.1 --port 8000
 ```
 
+```bash
+# Transcription worker (separate terminal)
+cd backend && source .venv/bin/activate
+python -m app.worker
+```
+
 ```bash
 # Frontend (separate terminal)
 cd frontend
@@ -44,16 +59,38 @@ npm run dev
 Open the URL Vite prints (typically `http://127.0.0.1:5173`). See
 [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) for the production build and its limitations.
 
+### Setting up transcription
+
+Transcription needs two things this repository deliberately does **not** ship: `ffmpeg`, and a
+locally-built whisper.cpp runtime plus model weights. Neither is vendored (they are large, and
+this is a public repo — see [docs/adr/0009-pinned-whisper-cpp-runtime.md](docs/adr/0009-pinned-whisper-cpp-runtime.md)).
+
+```bash
+brew install cmake ffmpeg      # your call -- the setup script never installs packages for you
+scripts/setup_whisper.sh       # clone pinned v1.9.1, build whisper-cli, download base.en
+scripts/check_whisper_runtime.sh   # read-only diagnosis of what's missing
+```
+
+The setup script builds into `backend/runtime/whisper/` and downloads models into
+`backend/storage/models/` — both gitignored. If a dependency is missing it prints the exact
+install command and exits rather than installing anything itself.
+
+Without this, recording and reflection still work; sessions will fail with a message naming the
+missing piece and the command that fixes it, and your recordings are always preserved.
+
 ## Architecture, in one paragraph
 
 A SvelteKit frontend talks to a FastAPI backend over a small JSON/multipart REST API (relative
 URLs only, proxied by Vite in dev — see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)). The
-backend owns SQLite (WAL mode, real migrations, no ORM) and local recording storage, and
-processes each recording through three provider interfaces
-(`TranscriptionProvider` / `MetricsProvider` / `FeedbackProvider`) — today backed entirely by
-deterministic mocks, designed so a real whisper.cpp / real analytics / optional local-Ollama
-implementation can be swapped in later without touching the rest of the app. See
-[docs/AI_MODEL_STRATEGY.md](docs/AI_MODEL_STRATEGY.md).
+backend owns SQLite (WAL mode, real migrations, no ORM) and local recording storage. Uploading a
+recording stores it, queues a row in a `processing_jobs` table, and returns `202` immediately; a
+separate worker process claims that job atomically, decodes the audio to 16 kHz mono PCM with
+ffmpeg, runs a pinned local whisper.cpp binary, and writes a transcript with full model/runtime
+provenance. Analysis still goes through three provider interfaces
+(`TranscriptionProvider` / `MetricsProvider` / `FeedbackProvider`) — transcription is now real,
+metrics and feedback remain deterministic mocks. See
+[docs/AI_MODEL_STRATEGY.md](docs/AI_MODEL_STRATEGY.md), which also documents honestly where the
+Phase 1 plan for this swap turned out to be wrong.
 
 ## Documentation map
 
@@ -66,14 +103,14 @@ implementation can be swapped in later without touching the rest of the app. See
 | [docs/DATA_MODEL.md](docs/DATA_MODEL.md) | Schema, and the topic seed-vs-runtime split |
 | [docs/TOPIC_TAXONOMY.md](docs/TOPIC_TAXONOMY.md) | The 50-topic bank and selection logic |
 | [docs/SCORING_AND_LIMITATIONS.md](docs/SCORING_AND_LIMITATIONS.md) | What's simulated, honestly |
-| [docs/AI_MODEL_STRATEGY.md](docs/AI_MODEL_STRATEGY.md) | Mock → real provider swap plan |
+| [docs/AI_MODEL_STRATEGY.md](docs/AI_MODEL_STRATEGY.md) | Real transcription, and what the swap actually cost |
 | [docs/PRIVACY_AND_SECURITY.md](docs/PRIVACY_AND_SECURITY.md) | What stays local, upload validation, deletion |
 | [docs/TESTING.md](docs/TESTING.md) | Automated / mocked / integration / manual test tiers |
 | [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) | Running locally; why no Docker/cloud |
-| [docs/ROADMAP.md](docs/ROADMAP.md) | Recommended Phase 2+ scope |
+| [docs/ROADMAP.md](docs/ROADMAP.md) | What shipped in Phase 2A; recommended Phase 2B scope |
 | [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) | Exact current state, test results, known gaps |
 | [docs/LEARNING_NOTES.md](docs/LEARNING_NOTES.md) | Plain-language architecture walkthrough |
-| [docs/adr/](docs/adr/) | Five architectural decision records |
+| [docs/adr/](docs/adr/) | Nine architectural decision records |
 | [AGENTS.md](AGENTS.md) | Rules for any coding agent working in this repo |
 | [SECURITY.md](SECURITY.md) | How to report a security/privacy issue |
 
@@ -87,20 +124,31 @@ upload validation does and doesn't guarantee, and how deletion works.
 ## Testing
 
 ```bash
-cd backend && source .venv/bin/activate && pytest -q     # 49 passed
-cd frontend && npx vitest run                              # 26 passed
+cd backend && source .venv/bin/activate && pytest -q     # 133 passed, 2 skipped
+cd frontend && npx vitest run                              # 60 passed
 cd frontend && npm run check && npm run lint               # type-check + lint
 cd frontend && npm run build                                # production build
+bash scripts/check_public_safety.sh                         # pre-publication safety audit
 ```
 
+The 2 skips are the opt-in real-whisper.cpp integration test, which runs only when a built
+runtime, a model, ffmpeg and your own sample recording are all genuinely present — and names the
+specific missing prerequisite when they aren't. Every ffmpeg/whisper interaction in the regular
+suite is mocked, so the suite runs on a machine with neither installed.
+
 See [docs/TESTING.md](docs/TESTING.md) for what each tier actually covers, and
 [docs/PROJECT_STATUS.md](docs/PROJECT_STATUS.md) for exact current results.
 
 ## What I learned building this
 
-Provider interfaces are worth their extra boilerplate the moment you can name the second
-implementation that's coming — they turned the eventual whisper.cpp swap from "rewrite the app"
-into "write one class." Honest status machines with explicit failure/cleanup paths (see
+Provider interfaces are worth their boilerplate — but not for the reason I first wrote here. I
+claimed they would turn the whisper.cpp swap into "write one class." They didn't: the Phase 1
+interface never received any audio, the result type had nowhere to record which model produced it,
+and the call site had to leave the HTTP request entirely for a worker process. What the interface
+genuinely bought was *containment* — everything that had to change was findable from one
+definition. An abstraction designed only against a fake encodes the fake's assumptions, and
+"we have an interface, so this will be easy" is a claim worth distrusting. Honest status machines
+with explicit failure/cleanup paths (see
 [docs/DATA_FLOW.md](docs/DATA_FLOW.md)) catch real bugs that unit tests with idealized fixtures
 miss — a manual pass through the real UI against the real backend found a genuine MIME-type
 handling bug that 48 passing automated tests hadn't (see