VRAG-023: read the text off slide-heavy keyframes, cost both arms, and record the notetaker roadmap

[vimal-55]

Two commits on top of what PR #30 already merged into dev.

VRAG-023 — read the text off slide-heavy keyframes, and cost both arms

Ingest has always sampled frames; nothing ever read them. Five of the answerable
held-out pairs turn on something on screen rather than something said, and one is on a
video with no speech at all, so a transcript-only index cannot reach them. This measures
what closing that gap would cost and closes nothing.

The cost lever is the selection, not the model. Captioning every sampled frame of the
91-minute client meeting is 1091 vision calls whether or not those frames were worth
reading. A slide is worth reading and it holds still, so one extra ffmpeg pass over the
already-extracted frames uses select as a measuring instrument rather than a filter —
gte(scene,0) is always true, so every frame prints its own scene score. Stretches of low
scores collapse to one keyframe each. The same two levers pick a far smaller share of the
Bernini documentary, which is what makes "slide-heavy" a measured property rather than a
label.

New: src/keyframes.py, src/caption.py, schemas/caption.py, prompts/caption_v1.md,
tools/caption_arms.py, [caption] in config.toml, make captions / make caption-arms.

The table

From the committed artifact docs/learning/data/caption_arms.json — recorded when the tool
was run, not re-measured in this PR. Reproduce with
make caption-arms VIDEO=samples/vector7-21aug-client-meeting.mp4.

Selection over vector7-21aug-client-meeting (5454.7 s): 1091 frames considered → 64
still-stretches found, covering 90.6% of the video; scored in 0.65 s.

arm	model	calls	tokens/call	s/call	cost	text yield
hosted (NVIDIA NIM)	meta-llama/Llama-3.2-11B-Vision-Instruct	10	3761.2	9.42	$0.00	10/10
local (Ollama)	ggml-org/Qwen2.5-VL-3B-Instruct-GGUF:Q4_K_M	10	1598.4	19.58	$0.00	10/10

cost_usd is 0.00 on both rows because both run on a free tier or locally; the telemetry
logger records rate 0.0 rather than inventing a paid rate. The projected_* columns in the
artifact are arithmetic from the measured per-call rate over 64 stretches — projected, not
measured.

Two facts that had to be checked rather than derived

• Groq serves no vision model on our key, so the hosted arm is NVIDIA NIM, whose wire id
  has a different owner from the HF repo id. The lookup raises on a miss — this is the third
  distinct naming rule in the repo, after Whisper's and nomic's.
• The local repo was chosen because it carries an mmproj projector. Without one, Ollama
  loads a vision model that cannot see the image and captions every frame from the prompt
  alone.

Deliberate deviations, flagged for review

• Nothing is wired in. caption.index ships false, src/index.py is untouched, and no
  caption reaches the vrag collection.
• Neither arm gets a JSON schema, which breaks this repo's usual rule on purpose: the two
  providers do not offer the same structured-output guarantees, and the deliverable is a table
  in which the arm is the only variable. The structure actually needed is one sentinel,
  NO_TEXT, and schemas/caption.py refuses a caption whose has_text disagrees with its
  text, so a sentinel the model stops emitting fails loudly instead of silently reporting
  full yield.

Record the notetaker roadmap and the 2026-08-28 standup

docs/plan-teams-notetaker.md — the plan lived outside the repo where a teammate could not
read it. Ordered by what is actually unblocked rather than by what is interesting:
containerize, build, integrate. Two findings in it are worth having written down whatever
happens to the plan:

• Groq reports a 413 and a 429 under the same rate_limit_exceeded code, and they need
  opposite handling — one is throttling, the other is a request that can never fit.
• The real-time meeting bot cannot be Python or run here at all: Microsoft requires C# on
  Windows Server in Azure, and states the bot cannot be deployed even as an Azure web app.
  That decides an architecture, so it should not live in a chat log.

The STANDUP.md entry is dated 08-28 and inserted before the 08-31 block, per the
"newest at the bottom" rule at the top of that file.

Verification run for this PR

$ uv run pytest tests/unit/test_caption.py tests/unit/test_keyframes.py -q
51 passed in 0.59s

The wider suite and the phase gates were not re-run in this session — flagging that rather
than quoting a number I did not produce.

Review

Per CLAUDE.md: no self-merge. Needs the other person's review before it lands.