Skip to content

Sync videos nightly with GitHub Actions - #486

Merged
jeroenjanssens merged 20 commits into
mainfrom
videos/nightly-sync
Nov 12, 2026
Merged

jeroenjanssens merged 20 commits into
mainfrom
videos/nightly-sync

Conversation

@jeroenjanssens

@jeroenjanssens jeroenjanssens commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Note

Merge #487 first (catch-up of the videos since September plus the fixed transcripts). Then the first nightly run starts from an up-to-date site and should produce a small PR, the best test that the churn fix works. After merging this PR, run the workflow once by hand with refresh_stats (view counts were last updated in July, so that PR touches nearly every video page).

Prerequisite check

  • I have linked an approved GitHub Issue below. (No issue; this automates the manual video update.)

Related Issue

None.

Description of changes

Moves the manual video update (four scripts run by hand, in the right order) to a nightly GitHub Actions workflow that opens a pull request, like the course sync.

Workflow (.github/workflows/sync-videos.yml), 05:00 UTC:

  1. update-youtube-videos.py: YouTube metadata → data/videos.toml
  2. sync-videos.py: video pages and thumbnails
  3. transcribe.py: audio download (yt-dlp) + Whisper → transcription.vtt
  4. format-transcriptions.py: Claude → transcription.txt

It then opens or updates an automation/sync-videos PR with the posit-open-source-website-sync app. If last night's PR is still open, the run builds on it instead of redoing its work. Logs and machine-readable run summaries are attached to every run.

Transcripts stay a (small) local step. YouTube blocks audio downloads from GitHub's runners: both dry runs on this PR got "Sign in to confirm you're not a bot", the second one with the bgutil PO token provider loaded and working. The block is based on the runner's IP. So the nightly run skips step 3 (a manual run can still try), and the PR description lists the videos waiting for a transcript with the commands to make them on the PR branch (just transcribe, then push). The next run formats them with Claude and updates the PR.

Script changes

  • Only real changes: a video's entry and last_updated only change when a field changes; view counts are refreshed on Mondays only. Before, every run rewrote all ~1,700 entries and pages. Against the live API, a second run in a row now changes nothing.
  • Robust transcription: transcribe: false respected, --limit, temporary download dirs, yt-dlp errors classified (bot check, PO token, 403, …) and logged in full, a circuit breaker after three blocked downloads, and detection and retry of Whisper output with long stretches of empty cues.
  • Better formatted transcripts (found by checking all 1,710 published ones):
    • 520 fail validation, mostly 45–90 minute videos that lost more than 10% of their words: they were formatted in one request, which ran into the output limit or got condensed. Videos longer than 20 minutes are now formatted in 15-minute chunks, cut-off responses are rejected, and the prompt says not to skip anything; chunks that still lose words are retried.
    • Only 64% of the timestamps were within a second of the right moment. Timestamps are now taken from the VTT instead of the model: 97% exact.
    • Output is validated (allowed elements, timestamps, ≥ 90% of the spoken words) before it's written.
    • New --check-existing, --repair-timestamps, and --reformat-invalid modes for the published transcripts (the fixes themselves are in Sync videos from YouTube and fix the published transcripts #487).
  • Logging everywhere: per-video log lines with slug and YouTube ID, tool and package versions, timings, token usage, request IDs, and estimated cost; --summary-json and exit codes 0 (ok), 1 (fatal), 2 (partial).

Tests: 336 tests in 8 files (all services faked, plus a real ffmpeg test and an end-to-end test that checks a second run changes nothing), run by just test and the new test-scripts.yml workflow.

Repository secrets YOUTUBE_API_KEY, OPENAI_API_KEY, and ANTHROPIC_API_KEY have been added.

Before merging

  • Check the dry runs of the workflow on this PR: downloads are blocked on the runners, so transcription is local (see above).
  • Remove the temporary pull_request trigger from sync-videos.yml.

Screenshots or previews (if applicable)

Not applicable: no layout changes.

Checklist

  • My code/content builds locally without errors or warnings. (Hugo 0.158.0, the CI version, with the content of a local run.)
  • I have verified that all links and code blocks function properly.
  • This contribution aligns with the project's CONTRIBUTING.md guidelines.

- scripts/videolib.py: frontmatter and VTT parsing, logging setup (Rich in a
  terminal, plain UTC-timestamped lines in CI), RunSummary with exit codes
  0/1/2 and --summary-json, atomic writes, and retries with backoff
- scripts/videotesting.py: fakes for YouTube, yt-dlp, Whisper, and Anthropic
- scripts/run-tests.sh and 'just test' run every scripts/test-*.py
- Remove the stale download-youtube-audio recipe, pass arguments through to
  the video scripts, and add 'just update-videos' for local runs
- Fetch all tracked videos every run, but only change an entry (and its
  last_updated) when a field actually changed
- Keep stored view/like/comment counts unless --refresh-stats is passed
- Report videos that went missing or private instead of silently keeping them
- Resolve duplicate slugs deterministically, preferring the stored entry
- Treat an unreadable videos.toml or a failure of every source as fatal
  (exit 1, nothing written); a single failed source is partial (exit 2)
- Retry API calls, log quota use, and add --dry-run and --summary-json
- Write videos.toml once at the end, atomically, and only when it changed

Against the live API a second run in a row now changes nothing; before,
every run rewrote all ~1,700 entries.
- Compare the rendered _index.md with the file on disk and only write it when
  it differs; log and record which (external.) keys changed
- Download the thumbnail again when its URL changes, retry once, and record
  a failed download as an item error instead of only printing it
- Record renames, merges, updated references, and orphaned directories in
  the run summary
- Write files atomically; add --dry-run and --summary-json; exit codes 0/1/2

On the current repo, a run now leaves 1,692 of 1,710 pages untouched; the 18
updates pick up people and software pages added since the last sync.
- Respect transcribe: false and skip videos without a duration (live
  streams, premieres); process newest first
- Add --limit, --max-audio-minutes, --video, --cookies, and --dry-run
- Download into a temporary directory and split chunks there (replacing the
  insecure tempfile.mktemp), so no audio is left behind
- Classify yt-dlp failures (bot_check, po_token, http_403, private, ...),
  log yt-dlp's full output, and retry network errors and 403s once
- Stop downloading after three blocked downloads in a row
- Retry Whisper calls with logged backoff; reject empty transcripts and warn
  when a transcript ends well before the video does
- Log tool and package versions, timings, sizes, and estimated cost; write
  a run summary and use exit codes 0/1/2
Summarizes the four pipeline steps in a table, lists new, renamed, and
changed videos and the new transcripts, and collects problems (blocked or
failed downloads, failed formatting, crashed steps) with the command to fix
each one locally. With --annotations it also prints ::warning:: and
::error:: workflow commands for the Actions run page.
sync-videos.yml runs the four video scripts at 05:00 UTC (stats refreshed on
Mondays) and opens or updates an automation/sync-videos pull request with
the posit-open-source-website-sync app, like sync-courses.yml. It first
restores the files added by a still-open sync PR, so videos aren't
transcribed (and paid for) twice. Transcription and formatting never fail
the job; video-sync-report.py writes the PR description and step summary,
and the logs and run summaries are uploaded as an artifact.

A temporary pull_request trigger runs dry runs (no PR, diff attached) on the
PR that adds the workflow; it has to be removed before merging.

test-scripts.yml runs scripts/run-tests.sh when scripts change.
Checking the 1,710 published transcripts showed that 84-93% of the videos
of 45-90 minutes lost more than 10% of their words: they were formatted in
one request, which ran into the output token limit or got condensed, and a
response cut off at max_tokens was written anyway.

- Reject responses that stop at max_tokens
- Validate the HTML: only h3/p/a/blockquote, only data-t attributes, data-t
  within the video (a small overshoot is a warning), and at least 90% of
  the spoken words outside pull quotes; invalid output is never written
- Chunk videos longer than 20 minutes into 15-minute segments (was 90 and
  30 minutes) and request the chunks concurrently
- Add --check-existing (report invalid published transcripts, no API
  calls) and --reformat-invalid
- Log model, tokens, stop reason, latency, and request ID per call; record
  estimated cost; add --limit, --dry-run, $TRANSCRIPT_MODEL, exit codes
  0/1/2, and atomic writes
The end-to-end test runs the four scripts and the report against a fixture
repo with all services faked, and checks that a second run changes nothing,
that a renamed video keeps its transcripts, that stats only change on a
refresh, and that blocked downloads are retried the next run.
The model copies each cue's text verbatim but often gets its timestamp
wrong. Across the 1,706 published transcripts only 64% of the data-t values
are within a second of their cue (p90: 60 s off); 271 transcripts have
timestamps beyond the end of the video.

repair_timestamps() walks the anchors in order, finds each one's first
words in the VTT (searching forward from the previous match), and sets
data-t to that cue's start. Pull quotes reuse the value of their prose
anchor, and anchors split by inject_links stay together. Unmatched anchors
keep the model's value, capped at the last cue.

On the published transcripts this brings exact timestamps from 64% to 97%
(p90: 0 s) without changing anything but data-t values. New transcripts
are repaired before validation, and --repair-timestamps fixes existing
ones without API calls.
…ugging

Found in a local end-to-end run: Whisper returned only "." cues for the
last 22 minutes of a video, and the transcript passed every check. One of
the published transcripts has the same problem.

- transcribe.py: retry a chunk once when it has more than 2 minutes of
  cues without words, and reject the transcript (degenerate_transcript)
  when it still does
- transcribe.py: aim audio chunks at 90% of the API limit, so a chunk that
  comes out slightly larger isn't split again into a few seconds
- format-transcriptions.py: log and record the coverage per chunk, and save
  rejected output with --rejected-dir (the workflow keeps it in the artifact)
- videolib.py: also quiet the httpx2 logger
In a local run the model skipped small talk, logistics ("can you see my
slides?"), and parts of the Q&A, leaving chunks at 78-82% of the spoken
words. The prompt said to keep the text verbatim but never said not to
leave things out. With an explicit instruction to include every cue, the
same video came back at 97-100% per chunk.

Chunks that still lose more than 10% of their words are requested once
more, and the better answer is kept; both requests count toward the usage
and cost in the summary.
A short anchor such as "For us," could match a later occurrence hundreds
of words ahead, after which every following anchor was behind the search
position: in one real transcript 565 of 1,298 anchors went unmatched.
Anchors shorter than the 4-word key now only match within the next 40
words. That video now has 2 unmatched anchors, and exact timestamps across
the published transcripts go from 97.1% to 97.4%.
@netlify

netlify Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for posit-open-source canceled.

Name Link
🔨 Latest commit b40a0ed
🔍 Latest deploy log https://app.netlify.com/projects/posit-open-source/deploys/6af5ffe69c26dc89753befe8

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The first dry run on the runners failed both downloads with "Sign in to
confirm you're not a bot". Run the bgutil provider as a service container
and add its yt-dlp plugin, with debug logging for transcribe.py, to see
whether PO tokens get the downloads through.
yt-dlp's verbose output shows the JavaScript runtime, PO token providers,
and player clients it uses, which is what we need to see when downloads
are blocked.
Two dry runs on the runners, the second with the bgutil PO token provider
(loaded and working, with Deno for the JS challenges), both got "Sign in
to confirm you're not a bot": the block is based on the runner's IP. So:

- The nightly run no longer transcribes (a manual run can still try, with
  the new transcribe input); the PO token provider is removed again
- It still formats transcripts with Claude, which works fine on the runners
- format-transcriptions.py records the videos without a transcription.vtt,
  and the PR description lists them with the commands to transcribe them
  on the PR branch; the next run formats them and updates the PR
The model occasionally writes '<a a data-t="623">' or '<a data-t="="1876">'
(4 of the published transcripts, and again during the re-format). Repair
those instead of rejecting an otherwise complete transcript.
It ran dry runs on the PR that adds the workflow, which settled that
YouTube blocks downloads from GitHub's runners. From now on the workflow
runs on its schedule and by hand from main.
A pull quote whose text matches no prose anchor kept the model's value
unchecked; one re-formatted transcript had data-t="46039" in a 1-hour
video.
A ten-second network interruption failed 15 videos in a row: the backoff
(1, 2, 4, 8 s) gave up before the connection was back. Now 6 attempts
starting at 2 s, each wait capped at 60 s.
- inject_links: in plain text (such as an <h3>), a shorter name could match
  inside the URL of a link inserted just before, e.g. "Shiny" in
  /software/shiny-python/, producing broken HTML (once on the live site).
  Insert placeholders first and the links at the end.
- coverage: count a cue whose text repeats an earlier cue once. Whisper
  sometimes loops on a line, and the model rightly leaves the copies out;
  three short videos were rejected for that.
@jeroenjanssens
jeroenjanssens marked this pull request as ready for review November 12, 2026 18:09
@jeroenjanssens
jeroenjanssens merged commit dbdbe8c into main Nov 12, 2026
13 checks passed
@jeroenjanssens
jeroenjanssens deleted the videos/nightly-sync branch November 12, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant