Skip to content

Whole-video questions, the fold that makes them affordable, and a container to run it all in - #30

Merged
ritika371 merged 6 commits into
devfrom
feat/vrag-018-primer-coach
Aug 31, 2026
Merged

Whole-video questions, the fold that makes them affordable, and a container to run it all in#30
ritika371 merged 6 commits into
devfrom
feat/vrag-018-primer-coach

Conversation

@vimal-55

Copy link
Copy Markdown
Collaborator

Three separable changes, one per commit — read it commit by commit.

1. Resolve a locally ingested video to the file in samples/

local_file knew only the make sample-real layout (<video_id>_<youtube_id>.<ext>). A locally ingested video lands as <video_id>.<ext> and never matched. It indexed, answered and cited fine, then the frontend drew "no playable copy" over a video sitting on disk. Both globs keep the separator explicit so 61 cannot match 611_H8fGd3fCJbg.mp4.

2. Answer whole-video questions, and fold the transcript so the tier can

"What is this video about?" has no answer in any one 25-second chunk, so the extractive path declines it — correctly, under answer_v1.md rules 2 and 3. The synthesis happens once per video into runs/<stem>/overview.json, and every claim carries seconds off a real chunk so the existing ground() and the existing player handle it unchanged.

The 413 is the finding in this PR. The first build produced a document whose people held "Baroque period", "Victorian England" and "Rome", every span crammed into the last 300s of a 1805s video. That was not the prompt — it had run on the 3B local model, because the groq arm fell back:

413  {'code': 'rate_limit_exceeded', 'type': 'tokens'}
"Request too large ... tokens per minute (TPM): Limit 8000, Requested 17152"
x-ratelimit-remaining-tokens: 8000      <- a FULL bucket
x-should-retry: false

Groq reports both capacity failures under rate_limit_exceeded, so matching the string read a permanent condition as throttling. The bucket was never empty; the request cannot fit. A 413 now never falls back, whatever answer.fallback says.

No real transcript fits one call on this tier, so build windows the transcript and merges. people and topics are merged in code, never by a model — every span has to be a span off a real chunk, and concatenation cannot invent one. Only the abstract needs synthesis, and it is given no timestamps at all.

Three levers, each found by a failed run rather than estimated:

lever was now why
overview.max_context_chars 180000 10000 180000 measured the model's context window; the binding limit is the tier's throughput. Measured 3.531 chars/token, 2528 chars fixed overhead.
overview.window_max_tokens 4000 A window is a quarter of a video but the cap is charged whether used or not. At 2500 the reply was cut off before topics.
answer.reasoning_effort "low" gpt-oss reasoning tokens are charged against the completion cap. The failing window finished in 1539 tokens at "low".

3. Run the pipeline, the gates and the API in a container

A toolbox image, not a server: a supervisor re-runs the gate command and compares output, and an image that could only serve /ask could not be handed to one. Ollama is a sibling service because src/embed.py and src/retrieve.py call ollama.embed and there is no hosted embedding arm.

docker/ollama-init.sh pulls the :F16 tag written out in full — an untagged pull takes Q2_K and config.toml records what that costs (recall@5 0.9167 → 0.4167) — and refuses to run if that tag has drifted from the [embed] lever. .dockerignore is a licence boundary: an image layer is a copy and a pushed image is redistribution, so media and .env stay out. uv.lock is no longer ignored, because a lockfile that is not in the repo pins nothing.

Numbers, with the commands

uv run pytest tests/unit -q                                671 passed, 1 skipped   (was 645, 1)
make leakage-check                                         PASS, overlap 0
uv run pytest tests/gates/gate_phase0.py gate_phase1.py    17 passed
uv run pytest tests/gates/gate_phase2a.py -q -s            10 passed
make overview VIDEO=611 OVERVIEW_FLAGS=--refresh           6 windows, 3m16s, openai/gpt-oss-120b
  people 24  evidence not an exact chunk range: 0
  topics 72  start not a real chunk start, or end past 1804.8s: 0
docker compose config                                      exit 0

gate_phase2a was re-run specifically because answer.reasoning_effort is a global lever and that gate scores the extractive path. It did not move: schema-valid 1.0000 (15/15), abstentions 3/3, abstention rate 0.1667 on 12 answerable against a 0.25 ceiling, 10 citations all grounded, 0 repairs, $0.0000.

What a reviewer should push on

  • The image is not built. Docker Desktop's daemon is not running on this machine, so make docker-build && make docker-up && make docker-doctor is unrun. That is the first thing to try.
  • config.deploy.toml is missing. src/config.py reads one file and refuses defaults, so a deploy config is either a 900-line copy that drifts or a small overlay needing load(path, overlay=). Left undone rather than done badly; only needed for a public host, since local compose already binds 0.0.0.0 through the existing HOST override.
  • Items 4, 7, 8, 9, 10 of docs/plan-whole-video-questions.md are still open — the index_video hook, mode on AskRequest/Provenance, the mention._label fallback, the frontend control, and evals/overview/ + tests/gates/gate_overview.py. There is no gate on the overview yet, so its quality is argued from one run, not measured.
  • 72 topics for a 30-minute video is granular (~25s each). Defensible, since each window's topics are contiguous, but worth an opinion.

VRAG-018 asks for a concept primer on chunking vs recall, embeddings and the
cost of overlap. Both claims it has to make are measurable on evals/dev, so
they are measured first and the prose is written against the result.

tools/sweep_chunking.py sweeps a 12-point cross through chunk.window_s x
chunk.overlap_s -- the window walked at the shipped overlap, the overlap walked
at the shipped window -- scoring recall@1/@3/@5 by the QA_SPEC section 2 rule
and recording chunk count, duplication factor, longest chunk, chunks past the
+/-30 s tolerance, embed wall-time and store size per point. It re-chunks from
the cached transcripts, so twelve points make zero ASR calls, and it never
writes config.toml or touches ./chroma: Phase 1 was graded at 25.0/8.0 and a
sweep that moved the levers in place would re-tune a gate already passed.

  12 points in 1334.9s, $0.0000
  window cut:  recall@5 = 0.9167 at 12, 15, 20, 25, 30 and 45 s; 0.8333 at 60.
               1300 chunks/9.8 MB down to 114/1.9 MB across that range.
  overlap cut: 0.8333 at overlap 0, 0.9167 from 2 s up. Duplication 1.22x-3.34x.

docs/learning/primer-chunking-embeddings.md is the prose; docs/learning/coach.html
is the same numbers with the levers movable, built by tools/build_coach.py. The
page inlines the sweep JSON at build time because a file:// page cannot fetch its
own data directory, and it refuses to interpolate: an unswept combination reads
"not measured", and an impossible one (overlap >= window) says the config raises.

tests/unit/test_primer_numbers.py parses the tables back out of the primer and
asserts every cell against the sweep JSON. It caught five hand-rounded word
counts in the first draft.

Three things the sweep found, all corrected in place:

- The window_s ceiling derivation was generalised from one video. 181's longest
  whisper segment is 4.16 s; 611's is 29.98 s, so the bound is 85 s, not 33 s,
  and 120 of 346 dev chunks are longer than the tolerance. The formula was right,
  the input was one video. Costs no recall on dev, so 25.0 stands -- but not for
  the reason written next to it. README and config.toml now say so.
- make gate-phase1 prints 0.8333 on this working copy with no lever moved:
  ./chroma has acquired two videos that are not in the corpus, and 521 extra
  chunks take top-5 slots. Filtered to the corpus videos it is 0.9167 again.
  Written up as section 6 -- recall@k is a property of the whole index.
- make index-dev INDEX_FLAGS=--reset silently swallowed the variable. Wired up.

  uv run pytest tests/unit -q            -> 645 passed, 1 skipped
  uv run pytest tests/gates/*.py -q      -> 51 passed
  make leakage-check                     -> PASS, overlap 0
`local_file` knew one naming layout: `make sample-real` writes
`<video_id>_<youtube_id>.<ext>`, so a corpus id was matched as a prefix. A
video ingested locally has no youtube id to append and lands as
`<video_id>.<ext>` — `bob-video.mp4` — which that glob never matched.

The symptom was not a crash. Such a video indexes, answers and cites
correctly, then `media_url` finds no file, `stream_url` comes back null, and
the frontend draws "no playable copy" over a video that is right there on
disk.

Both globs keep the separator explicit. A bare `f"{video_id}*"` would let
`61` match `611_H8fGd3fCJbg.mp4` and serve one video under another's id.
"What is this video about?" and "who is taking part?" have no answer in any
one 25-second chunk, so the extractive path declines them — correctly, under
answer_v1.md rules 2 and 3, which are the rules that make the abstention
selective on d013-d015. The synthesis happens here instead: once per video,
over the whole transcript, into runs/<stem>/overview.json.

Every claim carries seconds off a real chunk, so `as_chunks` hands them to
the existing `ground()` and the player seeks to them through the existing
citation path. `speaker_labels` is `Literal[False]` and cannot be set true:
nothing here diarizes, so `people` is who the transcript *names*, which is a
different claim from who is speaking.

Why it folds
------------
The first build produced a document whose `people` held "Baroque period",
"Victorian England" and "Rome", every span crammed into the last 300s of a
1805s video. That was not the prompt. It had run on the 3B local model,
because the groq arm fell back and the warning was above the tail.

The cause is a 413, and the old code could not tell it from a 429:

    413  {'code': 'rate_limit_exceeded', 'type': 'tokens'}
    "Request too large ... tokens per minute (TPM): Limit 8000, Requested 17152"
    x-ratelimit-remaining-tokens: 8000      <- a FULL bucket
    x-should-retry: false

Groq reports both capacity failures under `rate_limit_exceeded`, so matching
the string read a permanent condition as throttling. The bucket was never
empty; the request cannot fit. So `_GroqRequestTooLargeError` is now separate
from `_GroqRateLimitError`, classification switches on HTTP status rather
than message text, and a 413 never falls back whatever `answer.fallback`
says — a smaller model answering a question the configured one refused to be
asked is how this went unnoticed in the first place.

No real transcript fits one call on this tier, so `build` cuts the
transcript into windows, summarises each, and merges. `people` and `topics`
are merged in code and never by a model: every span has to be a span off a
real chunk, a model asked to reconcile two documents will adjust a timestamp
to make them line up, and concatenation cannot invent one. Only the abstract
needs synthesis, and it is given no timestamps at all.

Three levers, each found by a failed run rather than estimated
--------------------------------------------------------------
  overview.max_context_chars  180000 -> 10000  180000 measured the model's
      context window; the binding limit is the tier's throughput. Measured
      3.531 chars/token on transcript, 2528 chars of fixed overhead.
  overview.window_max_tokens  new, 4000  a window is a quarter of a video but
      the cap is charged whether used or not. At 2500 the reply was cut off
      before `topics` and strict mode rejected the whole thing.
  answer.reasoning_effort     new, "low"  gpt-oss reasoning tokens are charged
      against the completion cap. The window that failed finished in 1539
      tokens at "low".

Measured: `make overview VIDEO=611 OVERVIEW_FLAGS=--refresh`, 6 windows,
3m16s, openai/gpt-oss-120b — 24 people, 0 whose evidence is not an exact
chunk range; 72 topics, 0 starting off a real chunk or ending past 1804.8s.

gate_phase2a re-run because answer.reasoning_effort is global and that gate
scores the extractive path: schema-valid 1.0000 (15/15), abstentions 3/3,
abstention rate 0.1667 on 12 answerable against a 0.25 ceiling, 10 citations
all grounded, 0 repairs, $0.0000. Unmoved.
`make setup` has to work from a clean clone, and on this project that has
meant a list of things to install by hand and one that is easy to get
subtly wrong: ffmpeg installed but off PATH silently skips ~37 ingest tests
and leaves a green suite that tested nothing.

The image is a toolbox rather than a server, and that is the deliberate
choice here. This repo's discipline is that a phase is done when its number
is printed, and that a supervisor re-runs the gate command and compares
output. An image that could only serve /ask could not be handed to one, so
`make docker-gate` runs every phase gate inside the thing that ships.

Ollama is a service, not something in the app image. src/embed.py and
src/retrieve.py both call `ollama.embed` and there is no hosted embedding
arm, so nothing retrieves without a reachable daemon — it is a hard
dependency of the system, not a convenience of local development. The app
finds it over OLLAMA_HOST, which the python client and src/doctor.py already
read, so this needed no code change.

docker/ollama-init.sh pulls the model with the `:F16` tag written out in
full. An untagged `ollama pull hf.co/<repo>` takes the repo's smallest file,
which here is Q2_K, and config.toml records what that costs: recall@5
0.9167 -> 0.4167 on 2-bit weights. It also refuses to run if the tag it
pulls has drifted from the [embed] lever, because an index built with
weights nothing queries with fails as bad retrieval rather than as an error.

`make doctor` is the HEALTHCHECK — it is the only command that answers "is
this container able to work" — which needed one fix to be true inside one:
the ollama *binary* is not present when the daemon is a sibling service, and
FAILing on that made the healthcheck red on a container that works. A
missing CLI is now a WARN when the daemon answers, and still a FAIL when it
does not.

.dockerignore is a licence boundary before it is an optimisation. An image
layer is a copy and a pushed image is redistribution, so samples/, runs/,
chroma/ and every media extension stay out; they are volumes instead.
`.env` stays out too — secrets reach the container as environment
variables, which src/env.py already prefers over any file.

uv.lock is no longer ignored. The build installs with `uv sync --frozen`,
so the image gets exactly the resolution that was tested and a stale lock
fails the build instead of quietly resolving something new. A lockfile that
is not in the repo pins nothing for anybody else.

Verified: `docker compose config` exits 0. The image is NOT yet built —
Docker Desktop's daemon is not running on this machine — so
`make docker-build && make docker-up && make docker-doctor` is unrun and is
the first thing a reviewer should try.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants