Whole-video questions, the fold that makes them affordable, and a container to run it all in - #30
Merged
Merged
Conversation
VRAG-018 asks for a concept primer on chunking vs recall, embeddings and the cost of overlap. Both claims it has to make are measurable on evals/dev, so they are measured first and the prose is written against the result. tools/sweep_chunking.py sweeps a 12-point cross through chunk.window_s x chunk.overlap_s -- the window walked at the shipped overlap, the overlap walked at the shipped window -- scoring recall@1/@3/@5 by the QA_SPEC section 2 rule and recording chunk count, duplication factor, longest chunk, chunks past the +/-30 s tolerance, embed wall-time and store size per point. It re-chunks from the cached transcripts, so twelve points make zero ASR calls, and it never writes config.toml or touches ./chroma: Phase 1 was graded at 25.0/8.0 and a sweep that moved the levers in place would re-tune a gate already passed. 12 points in 1334.9s, $0.0000 window cut: recall@5 = 0.9167 at 12, 15, 20, 25, 30 and 45 s; 0.8333 at 60. 1300 chunks/9.8 MB down to 114/1.9 MB across that range. overlap cut: 0.8333 at overlap 0, 0.9167 from 2 s up. Duplication 1.22x-3.34x. docs/learning/primer-chunking-embeddings.md is the prose; docs/learning/coach.html is the same numbers with the levers movable, built by tools/build_coach.py. The page inlines the sweep JSON at build time because a file:// page cannot fetch its own data directory, and it refuses to interpolate: an unswept combination reads "not measured", and an impossible one (overlap >= window) says the config raises. tests/unit/test_primer_numbers.py parses the tables back out of the primer and asserts every cell against the sweep JSON. It caught five hand-rounded word counts in the first draft. Three things the sweep found, all corrected in place: - The window_s ceiling derivation was generalised from one video. 181's longest whisper segment is 4.16 s; 611's is 29.98 s, so the bound is 85 s, not 33 s, and 120 of 346 dev chunks are longer than the tolerance. The formula was right, the input was one video. Costs no recall on dev, so 25.0 stands -- but not for the reason written next to it. README and config.toml now say so. - make gate-phase1 prints 0.8333 on this working copy with no lever moved: ./chroma has acquired two videos that are not in the corpus, and 521 extra chunks take top-5 slots. Filtered to the corpus videos it is 0.9167 again. Written up as section 6 -- recall@k is a property of the whole index. - make index-dev INDEX_FLAGS=--reset silently swallowed the variable. Wired up. uv run pytest tests/unit -q -> 645 passed, 1 skipped uv run pytest tests/gates/*.py -q -> 51 passed make leakage-check -> PASS, overlap 0
`local_file` knew one naming layout: `make sample-real` writes
`<video_id>_<youtube_id>.<ext>`, so a corpus id was matched as a prefix. A
video ingested locally has no youtube id to append and lands as
`<video_id>.<ext>` — `bob-video.mp4` — which that glob never matched.
The symptom was not a crash. Such a video indexes, answers and cites
correctly, then `media_url` finds no file, `stream_url` comes back null, and
the frontend draws "no playable copy" over a video that is right there on
disk.
Both globs keep the separator explicit. A bare `f"{video_id}*"` would let
`61` match `611_H8fGd3fCJbg.mp4` and serve one video under another's id.
"What is this video about?" and "who is taking part?" have no answer in any
one 25-second chunk, so the extractive path declines them — correctly, under
answer_v1.md rules 2 and 3, which are the rules that make the abstention
selective on d013-d015. The synthesis happens here instead: once per video,
over the whole transcript, into runs/<stem>/overview.json.
Every claim carries seconds off a real chunk, so `as_chunks` hands them to
the existing `ground()` and the player seeks to them through the existing
citation path. `speaker_labels` is `Literal[False]` and cannot be set true:
nothing here diarizes, so `people` is who the transcript *names*, which is a
different claim from who is speaking.
Why it folds
------------
The first build produced a document whose `people` held "Baroque period",
"Victorian England" and "Rome", every span crammed into the last 300s of a
1805s video. That was not the prompt. It had run on the 3B local model,
because the groq arm fell back and the warning was above the tail.
The cause is a 413, and the old code could not tell it from a 429:
413 {'code': 'rate_limit_exceeded', 'type': 'tokens'}
"Request too large ... tokens per minute (TPM): Limit 8000, Requested 17152"
x-ratelimit-remaining-tokens: 8000 <- a FULL bucket
x-should-retry: false
Groq reports both capacity failures under `rate_limit_exceeded`, so matching
the string read a permanent condition as throttling. The bucket was never
empty; the request cannot fit. So `_GroqRequestTooLargeError` is now separate
from `_GroqRateLimitError`, classification switches on HTTP status rather
than message text, and a 413 never falls back whatever `answer.fallback`
says — a smaller model answering a question the configured one refused to be
asked is how this went unnoticed in the first place.
No real transcript fits one call on this tier, so `build` cuts the
transcript into windows, summarises each, and merges. `people` and `topics`
are merged in code and never by a model: every span has to be a span off a
real chunk, a model asked to reconcile two documents will adjust a timestamp
to make them line up, and concatenation cannot invent one. Only the abstract
needs synthesis, and it is given no timestamps at all.
Three levers, each found by a failed run rather than estimated
--------------------------------------------------------------
overview.max_context_chars 180000 -> 10000 180000 measured the model's
context window; the binding limit is the tier's throughput. Measured
3.531 chars/token on transcript, 2528 chars of fixed overhead.
overview.window_max_tokens new, 4000 a window is a quarter of a video but
the cap is charged whether used or not. At 2500 the reply was cut off
before `topics` and strict mode rejected the whole thing.
answer.reasoning_effort new, "low" gpt-oss reasoning tokens are charged
against the completion cap. The window that failed finished in 1539
tokens at "low".
Measured: `make overview VIDEO=611 OVERVIEW_FLAGS=--refresh`, 6 windows,
3m16s, openai/gpt-oss-120b — 24 people, 0 whose evidence is not an exact
chunk range; 72 topics, 0 starting off a real chunk or ending past 1804.8s.
gate_phase2a re-run because answer.reasoning_effort is global and that gate
scores the extractive path: schema-valid 1.0000 (15/15), abstentions 3/3,
abstention rate 0.1667 on 12 answerable against a 0.25 ceiling, 10 citations
all grounded, 0 repairs, $0.0000. Unmoved.
`make setup` has to work from a clean clone, and on this project that has meant a list of things to install by hand and one that is easy to get subtly wrong: ffmpeg installed but off PATH silently skips ~37 ingest tests and leaves a green suite that tested nothing. The image is a toolbox rather than a server, and that is the deliberate choice here. This repo's discipline is that a phase is done when its number is printed, and that a supervisor re-runs the gate command and compares output. An image that could only serve /ask could not be handed to one, so `make docker-gate` runs every phase gate inside the thing that ships. Ollama is a service, not something in the app image. src/embed.py and src/retrieve.py both call `ollama.embed` and there is no hosted embedding arm, so nothing retrieves without a reachable daemon — it is a hard dependency of the system, not a convenience of local development. The app finds it over OLLAMA_HOST, which the python client and src/doctor.py already read, so this needed no code change. docker/ollama-init.sh pulls the model with the `:F16` tag written out in full. An untagged `ollama pull hf.co/<repo>` takes the repo's smallest file, which here is Q2_K, and config.toml records what that costs: recall@5 0.9167 -> 0.4167 on 2-bit weights. It also refuses to run if the tag it pulls has drifted from the [embed] lever, because an index built with weights nothing queries with fails as bad retrieval rather than as an error. `make doctor` is the HEALTHCHECK — it is the only command that answers "is this container able to work" — which needed one fix to be true inside one: the ollama *binary* is not present when the daemon is a sibling service, and FAILing on that made the healthcheck red on a container that works. A missing CLI is now a WARN when the daemon answers, and still a FAIL when it does not. .dockerignore is a licence boundary before it is an optimisation. An image layer is a copy and a pushed image is redistribution, so samples/, runs/, chroma/ and every media extension stay out; they are volumes instead. `.env` stays out too — secrets reach the container as environment variables, which src/env.py already prefers over any file. uv.lock is no longer ignored. The build installs with `uv sync --frozen`, so the image gets exactly the resolution that was tested and a stale lock fails the build instead of quietly resolving something new. A lockfile that is not in the repo pins nothing for anybody else. Verified: `docker compose config` exits 0. The image is NOT yet built — Docker Desktop's daemon is not running on this machine — so `make docker-build && make docker-up && make docker-doctor` is unrun and is the first thing a reviewer should try.
ritika371
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three separable changes, one per commit — read it commit by commit.
1. Resolve a locally ingested video to the file in
samples/local_fileknew only themake sample-reallayout (<video_id>_<youtube_id>.<ext>). A locally ingested video lands as<video_id>.<ext>and never matched. It indexed, answered and cited fine, then the frontend drew "no playable copy" over a video sitting on disk. Both globs keep the separator explicit so61cannot match611_H8fGd3fCJbg.mp4.2. Answer whole-video questions, and fold the transcript so the tier can
"What is this video about?" has no answer in any one 25-second chunk, so the extractive path declines it — correctly, under
answer_v1.mdrules 2 and 3. The synthesis happens once per video intoruns/<stem>/overview.json, and every claim carries seconds off a real chunk so the existingground()and the existing player handle it unchanged.The 413 is the finding in this PR. The first build produced a document whose
peopleheld "Baroque period", "Victorian England" and "Rome", every span crammed into the last 300s of a 1805s video. That was not the prompt — it had run on the 3B local model, because the groq arm fell back:Groq reports both capacity failures under
rate_limit_exceeded, so matching the string read a permanent condition as throttling. The bucket was never empty; the request cannot fit. A 413 now never falls back, whateveranswer.fallbacksays.No real transcript fits one call on this tier, so
buildwindows the transcript and merges.peopleandtopicsare merged in code, never by a model — every span has to be a span off a real chunk, and concatenation cannot invent one. Only the abstract needs synthesis, and it is given no timestamps at all.Three levers, each found by a failed run rather than estimated:
overview.max_context_charsoverview.window_max_tokenstopics.answer.reasoning_effort"low""low".3. Run the pipeline, the gates and the API in a container
A toolbox image, not a server: a supervisor re-runs the gate command and compares output, and an image that could only serve
/askcould not be handed to one. Ollama is a sibling service becausesrc/embed.pyandsrc/retrieve.pycallollama.embedand there is no hosted embedding arm.docker/ollama-init.shpulls the:F16tag written out in full — an untagged pull takesQ2_Kandconfig.tomlrecords what that costs (recall@5 0.9167 → 0.4167) — and refuses to run if that tag has drifted from the[embed]lever..dockerignoreis a licence boundary: an image layer is a copy and a pushed image is redistribution, so media and.envstay out.uv.lockis no longer ignored, because a lockfile that is not in the repo pins nothing.Numbers, with the commands
gate_phase2awas re-run specifically becauseanswer.reasoning_effortis a global lever and that gate scores the extractive path. It did not move: schema-valid 1.0000 (15/15), abstentions 3/3, abstention rate 0.1667 on 12 answerable against a 0.25 ceiling, 10 citations all grounded, 0 repairs, $0.0000.What a reviewer should push on
make docker-build && make docker-up && make docker-doctoris unrun. That is the first thing to try.config.deploy.tomlis missing.src/config.pyreads one file and refuses defaults, so a deploy config is either a 900-line copy that drifts or a small overlay needingload(path, overlay=). Left undone rather than done badly; only needed for a public host, since local compose already binds0.0.0.0through the existingHOSToverride.docs/plan-whole-video-questions.mdare still open — theindex_videohook,modeonAskRequest/Provenance, themention._labelfallback, the frontend control, andevals/overview/+tests/gates/gate_overview.py. There is no gate on the overview yet, so its quality is argued from one run, not measured.