Phase 2A: real local transcription via whisper.cpp with a queued local worker - #1
Draft
abdo2006-dev wants to merge 9 commits into
Draft
Phase 2A: real local transcription via whisper.cpp with a queued local worker#1abdo2006-dev wants to merge 9 commits into
abdo2006-dev wants to merge 9 commits into
Conversation
Everything the local transcription runtime needs lives outside Git: backend/runtime/whisper/ for the cloned source and built binary, backend/storage/models/ for weights, backend/storage/processing/ for per-job decoded audio, backend/storage/benchmarks/ for benchmark input and output. setup_whisper.sh clones and builds the pinned v1.9.1 release (commit f049fff95a089aa9969deb009cdd4892b3e74916), downloads a model with upstream's own downloader, verifies both, and records the tag, commit, build options, binary version and model SHA-256 in a manifest the application reads. It never installs system packages, never touches git or shell configuration: a missing cmake/ffmpeg is reported with the exact command to run, and the script exits. Core ML, quantization and VAD are deliberately left off. They change output and need measuring before adoption. check_whisper_runtime.sh diagnoses the same set-up read-only. The public-safety check now rejects runtime binaries, models, processing and benchmark artifacts, audio of any kind, and transcript dumps; it also fails if the local username appears in tracked content, or if anything under the runtime directories is neither tracked nor ignored.
Migration 0002 adds the durable job queue and everything a real
transcript needs to stay identifiable years later.
processing_jobs carries status, stage, attempt/max-attempt counts, a safe
error code and message, worker id, timestamps and runtime provenance. A
partial unique index on (session_id, job_type) WHERE status IN
('PENDING','RUNNING') enforces one active job per session in the
database, rather than via a check-then-insert that would race.
transcripts gains detected language, runtime version, model name and
SHA-256, parameters, audio and processing durations, and real-time
factor. transcript_segments stores ordered segment timestamps with a
UNIQUE (transcript_id, segment_index) constraint, CHECKs for ordering,
and cascading deletion. Word-level timestamps are deliberately out of
scope for this phase.
sessions gains audio_duration_seconds: the authoritative duration decoded
from the media, kept separate from client_reported_duration_seconds so
the browser's own timer stays available for comparison rather than being
overwritten.
Every added column is nullable or defaulted, so this applies cleanly to
an existing Phase 1 database without rewriting a single row.
Job claiming uses BEGIN IMMEDIATE on an autocommit connection, so
get_connection now takes an autocommit flag. The claim's UPDATE re-asserts
status = 'PENDING' so a lost race updates zero rows instead of
double-claiming, and a rejected duplicate rolls back rather than leaving
its connection holding a write lock.
Browsers record Opus-in-WebM or AAC-in-MP4; whisper.cpp reads neither. The decode step between them is a real boundary, so it gets an interface: AudioPreprocessor, with FfmpegAudioPreprocessor behind it. The preprocessor produces 16 kHz mono signed-16-bit PCM WAV, measures the authoritative duration from the decoded WAV header rather than trusting the client, writes only inside a per-job temporary directory, and removes that directory on success and on failure alike. The original recording is opened read-only and never modified. A decode that silently ignores the conversion flags is treated as a failure rather than handed to the model. run_command is now the only place this application starts a process. It takes argv arrays and rejects strings outright, so there is no shell and no quoting class of bug; a timeout is mandatory; output is captured rather than inherited; stdin is closed. ffmpeg and whisper stderr are full of absolute paths, which on a personal machine contain the owner's username. safe_text redacts absolute paths and home references before anything is logged, and raw subprocess output never reaches an exception message or the API.
The Phase 1 docs called this a one-line dependency swap. The provider boundary was right; the signature was not, and three things had to change. transcribe() received session metadata and no audio at all -- a mock can write a sentence from a topic title, a speech model cannot. It now takes a TranscriptionRequest carrying the decoded audio path, the measured duration, the requested language, and explicit model and runtime configuration, rather than reaching for global settings itself. TranscriptResult carried a name, a version, text and is_mock. It now also carries the detected language, ordered segments with millisecond timestamps, model name and SHA-256, runtime version, inference parameters, processing duration and real-time factor. Without those, two transcripts produced by different models are indistinguishable after the fact. WhisperCppTranscriptionProvider parses --output-json rather than scraping console text, so a formatting change upstream cannot silently corrupt a transcript. Missing, unreadable, or structurally wrong output is a provider failure; an empty transcription list is a legitimate "no speech" result and is not. Each failure carries a stable code and says whether a retry could plausibly help. MockTranscriptionProvider stays, implements the new contract, and reports no model or runtime provenance -- but it is no longer the default and is never a silent fallback. If whisper.cpp is unavailable the job fails visibly, because simulated text under a "real transcript" label is worse than an error.
The upload request now stores the recording, transitions the session to
PROCESSING, creates a job row, and returns 202 Accepted. It no longer
decodes audio or runs any provider: real transcription of a two-minute
clip takes far longer than any reasonable request timeout, and a request
that died mid-inference would leave no record that work was owed. The job
row is that record.
The worker (python -m app.worker) polls, claims one job at a time,
releases the database transaction before touching audio, and handles its
own migrations so it can start before the API ever has. It sweeps stale
RUNNING jobs on startup and periodically, so a worker that was SIGKILLed
does not strand a session forever. SIGINT/SIGTERM finish the current job
and stop; a second signal still kills.
Failures are sorted into retryable (timeouts, transient subprocess
crashes) and terminal (no model, no binary, no ffmpeg, undecodable audio).
Retrying a missing model file forever would bury the one instruction the
owner needs to read. A terminal failure moves the session to FAILED and
always preserves the original recording.
The session becomes COMPLETED only after the transcript transaction
commits, and a retry clears any partial analysis output first so the
UNIQUE constraint on transcripts.session_id cannot fail a job for a reason
unrelated to the audio.
Adds GET /api/sessions/{id}/status as a narrow polling endpoint -- polling
the full session detail once a second would drag the transcript across the
wire to read one field -- and GET /api/runtime/capabilities, which reports
which prerequisite is missing without exposing paths, usernames, home
directories, environment variables or transcript content.
The session-service tests now assert the queueing contract: an upload leaves the session in PROCESSING with a PENDING job and no analysis output, the status-event log stops at PROCESSING, a second upload is rejected, deleting a session cascades to its jobs, and reflection works while transcription is still running. The Phase 1 partial-failure tests are kept as-is, plus a new one covering a job row that cannot be written. API tests cover the 202 response shape, the narrow status endpoint, and the capabilities endpoint, asserting that neither leaks an absolute path. The end-to-end test completes the job the way the worker would, so it exercises real-provenance transcript data over HTTP. Test fixtures generate synthetic silent PCM WAVs arithmetically and a synthetic whisper.cpp JSON payload matching the documented shape. No real recording is ever a fixture. benchmark_models.py compares models on a recording the owner supplies, reporting model size, audio duration, processing time, real-time factor, detected language, runtime and model version, normalized transcript, and word error rate against a reference the owner writes. It measures rather than estimating: a missing model or binary is an error, not a plausible number. Input and output stay under storage/benchmarks, which is gitignored, because both contain the owner's voice and words.
Migration tests apply 0001, populate it with Phase 1 shaped rows, and only then apply 0002 -- testing migrations against a fresh database only is how a migration that works in CI destroys the one database that matters. They assert existing rows survive untouched, the one-active-job index is enforced by SQLite, and segment ordering and cascade constraints hold. Worker tests cover the guarantee the queue exists for: two workers on separate connections cannot claim the same job. Also oldest-first ordering, the success path writing a real transcript with ordered segments and full provenance, retryable failure requeuing while the session stays PROCESSING, terminal failure failing the session while preserving the recording, a finite retry budget, stale-RUNNING recovery, restart behavior, graceful shutdown, temp-file cleanup after both success and failure, and that an unexpected error cannot escape and kill the loop. Preprocessor tests assert argv arrays with the required conversion flags, that a string command is refused outright, that shell metacharacters are never interpreted, mandatory timeouts, and that ffmpeg stderr containing an absolute path never reaches the exception a caller might surface. whisper provider tests cover argv construction and every way its JSON can be wrong: missing file, unparseable, missing transcription key, missing or non-numeric offsets. An empty transcription list stays a legitimate empty result rather than an error. The real whisper.cpp integration test is opt-in and skips with the exact missing prerequisite named. On this machine it skips: ffmpeg is not installed, so no real-model run has occurred.
The frontend now waits for transcription with ordinary HTTP polling: no WebSocket, no SSE. One user on one machine waiting on one job does not need a persistent connection, and a connection would bring reconnection logic and a proxying story the dev setup does not have. The poller takes its clock, timers and fetch as injected dependencies, so its whole lifecycle is unit-testable with no real time and no real network. It starts near one second, backs off to a cap, and stops on COMPLETED, FAILED, a 404 when the session was deleted, an overall timeout, or explicit cancellation from route teardown. Calling start twice cannot create a second loop, and a response arriving after cancel changes nothing -- navigating away mid-transcription must not touch a destroyed component or corrupt the session. Results presentation changed to match what is now true. Before, every section was simulated, so one page-level banner was honest; now the transcript is real while metrics and feedback are still mocks, so a global banner would be false. Each section reads is_mock from its own row: the transcript shows "Real transcript" with its model, runtime, language and speed, while metrics and feedback keep "Simulated" and say plainly that they are not derived from the transcript above. A Phase 1 session still shows "Simulated" on all three, because its rows still say so. A failed session explains which prerequisite is missing and gives the exact command that fixes it, and always states that the recording itself survived. A session that stays queued past the timeout says the worker is probably not running, rather than spinning forever. Reflection is reachable while transcription runs and shows the current stage without ever blocking the form -- reflecting while it is fresh is the point of that screen.
Four new ADRs: the separate worker and SQLite job queue (including why no broker and why not an in-process background task), the ffmpeg preprocessing boundary, HTTP polling over WebSockets/SSE, and pinned external whisper.cpp runtime management. AI_MODEL_STRATEGY.md now corrects, rather than quietly edits away, the Phase 1 claim that this would be a one-line dependency swap. The provider boundary was right; the signature was not. The interface received no audio, the result had nowhere to record provenance, the call site had to leave the request entirely, and a decode step had to exist at all. An interface localises where a change lands -- it does not make the change small, and it cannot anticipate what a real implementation needs. SCORING_AND_LIMITATIONS.md now describes three different truths on one page instead of one blanket disclaimer, and states plainly that the simulated metrics are still not derived from the real transcript sitting directly above them. It also records that transcription accuracy is itself unmeasured here. LEARNING_NOTES.md explains why transcription cannot run in the request, why a worker is not an AI agent, how SQLite serves as a durable queue, what atomic claiming means and the double-transcription bug it prevents, why browser audio must be decoded, weights versus runtime, why versions must be stored, what real-time factor means, why the original recording and the temporary PCM are treated oppositely, and why polling suffices. PROJECT_STATUS.md records exact results and is explicit about what did NOT happen: no whisper.cpp build, no real-model run, and no benchmark has ever executed here, because this machine has neither cmake nor ffmpeg. No latency, real-time-factor or accuracy figure is published anywhere, because none was measured. AGENTS.md moves whisper.cpp, ffmpeg and the local worker from prohibited to approved, keeps message brokers prohibited, and adds the rules this phase produced -- never silently substitute a mock for a real provider, never fabricate a measurement, all subprocesses through one argv-only helper, and new runtime artifacts need both a gitignore entry and a safety-check rule in the same change. The safety check flagged this change's own test fixtures for containing literal home-directory paths. Rather than weaken the check, the fixtures now assemble that path at runtime.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the simulated transcript with real local speech-to-text via whisper.cpp. Metrics and feedback stay simulated and are now labeled per section rather than behind one page-wide banner.
The correction this phase forced
Phase 1 documented this as "write one class, change one line in
dependencies.py". That was wrong, anddocs/AI_MODEL_STRATEGY.mdnow records why rather than quietly editing it away:transcribe()never received any audio — it took a topic title and a duration. A mock can write a sentence from that; a model cannot.TranscriptResulthad nowhere to record which model produced the text, so historical transcripts would be indistinguishable after the fact. That needed a migration, not a new class.SessionServiceran providers inside the HTTP request. Real transcription cannot, so processing had to move to a separate process.The provider boundary was genuinely valuable: it made the change contained. It did not make it small.
Architecture
Upload → store audio →
PROCESSING→ insert job row → 202 Accepted. A separate worker (python -m app.worker) claims the job, decodes with ffmpeg, runs whisper.cpp, and writes the transcript. The frontend polls.Job claiming is atomic via
BEGIN IMMEDIATEplus aWHERE status = 'PENDING'guard on theUPDATE; the transaction commits before any audio work, so inference never holds the write lock. Failures are sorted into retryable (timeouts) and terminal (missing model/binary/ffmpeg) — retrying a missing model file forever would bury the one instruction the owner needs. The recording is preserved on every failure path.Migration 0002
Applies to a populated Phase 1 database — every added column is nullable or defaulted, no row is rewritten.
tests/test_migrations.pyapplies 0001, inserts Phase 1 shaped rows, and only then applies 0002.processing_jobs, with a partial unique index enforcing one active job per session in the database rather than via a racy check-then-inserttranscript_segments(ordering + cascade constraints; no word-level timestamps this phase)sessions.audio_duration_seconds— authoritative, decoded — alongside the preservedclient_reported_duration_secondsRuntime
scripts/setup_whisper.shbuilds pinned v1.9.1 (f049fff95a089aa9969deb009cdd4892b3e74916, verified against the upstream releases API at the start of this work) into gitignoredbackend/runtime/whisper/, downloads a model intobackend/storage/models/, and records tag, actual commit, build options, binary version and model SHA-256 in a manifest the app reads. It never installs system packages — missing dependencies get the exact command and the script exits. Core ML, quantization and VAD are deliberately off; Metal is on by default upstream.Verification — exact results
backend $ pytest -q133 passed, 2 skipped, 1 warning in 2.28s(baseline was 49)frontend $ npx vitest run6 files, 60 passed(baseline was 26)frontend $ npm run check256 FILES 0 ERRORS 0 WARNINGSfrontend $ npm run lintfrontend $ npm run build✔ done(adapter-node)scripts/check_public_safety.shOK(exit 0)What was NOT verified — please read this part
This build environment has neither
cmakenorffmpeginstalled. Therefore:setup_whisper.shwas syntax-checked only.ffmpeg is not installed (brew install ffmpeg).backend/scripts/benchmark_models.pyis ready to produce them.Every ffmpeg/whisper interaction in the passing suite is mocked, and all audio fixtures are synthetic silent WAVs generated arithmetically.
A real bug found while building
Creating a duplicate job (correctly rejected by the partial unique index) left the connection inside an open transaction holding a write lock, so the worker's next
BEGIN IMMEDIATEfailed withdatabase is locked. Fixed by rolling back before re-raising. Every unit test passed both before and after — it only appeared once two connections interacted, which is precisely what the queue exists to handle.The safety script also flagged this change's own test fixtures for containing literal home-directory paths. Rather than weaken the check, the fixtures now assemble that path at runtime.
Recommended Phase 2B
base.envssmall.enbenchmark.Draft — do not merge. Please review the contract revision (
transcription_provider.py), the claim logic (job_repository.py), and migration 0002 most closely.