Repository navigation
Sync videos nightly with GitHub Actions - #486
Merged
Merged
Conversation
- scripts/videolib.py: frontmatter and VTT parsing, logging setup (Rich in a terminal, plain UTC-timestamped lines in CI), RunSummary with exit codes 0/1/2 and --summary-json, atomic writes, and retries with backoff - scripts/videotesting.py: fakes for YouTube, yt-dlp, Whisper, and Anthropic - scripts/run-tests.sh and 'just test' run every scripts/test-*.py - Remove the stale download-youtube-audio recipe, pass arguments through to the video scripts, and add 'just update-videos' for local runs
- Fetch all tracked videos every run, but only change an entry (and its last_updated) when a field actually changed - Keep stored view/like/comment counts unless --refresh-stats is passed - Report videos that went missing or private instead of silently keeping them - Resolve duplicate slugs deterministically, preferring the stored entry - Treat an unreadable videos.toml or a failure of every source as fatal (exit 1, nothing written); a single failed source is partial (exit 2) - Retry API calls, log quota use, and add --dry-run and --summary-json - Write videos.toml once at the end, atomically, and only when it changed Against the live API a second run in a row now changes nothing; before, every run rewrote all ~1,700 entries.
- Compare the rendered _index.md with the file on disk and only write it when it differs; log and record which (external.) keys changed - Download the thumbnail again when its URL changes, retry once, and record a failed download as an item error instead of only printing it - Record renames, merges, updated references, and orphaned directories in the run summary - Write files atomically; add --dry-run and --summary-json; exit codes 0/1/2 On the current repo, a run now leaves 1,692 of 1,710 pages untouched; the 18 updates pick up people and software pages added since the last sync.
- Respect transcribe: false and skip videos without a duration (live streams, premieres); process newest first - Add --limit, --max-audio-minutes, --video, --cookies, and --dry-run - Download into a temporary directory and split chunks there (replacing the insecure tempfile.mktemp), so no audio is left behind - Classify yt-dlp failures (bot_check, po_token, http_403, private, ...), log yt-dlp's full output, and retry network errors and 403s once - Stop downloading after three blocked downloads in a row - Retry Whisper calls with logged backoff; reject empty transcripts and warn when a transcript ends well before the video does - Log tool and package versions, timings, sizes, and estimated cost; write a run summary and use exit codes 0/1/2
Summarizes the four pipeline steps in a table, lists new, renamed, and changed videos and the new transcripts, and collects problems (blocked or failed downloads, failed formatting, crashed steps) with the command to fix each one locally. With --annotations it also prints ::warning:: and ::error:: workflow commands for the Actions run page.
sync-videos.yml runs the four video scripts at 05:00 UTC (stats refreshed on Mondays) and opens or updates an automation/sync-videos pull request with the posit-open-source-website-sync app, like sync-courses.yml. It first restores the files added by a still-open sync PR, so videos aren't transcribed (and paid for) twice. Transcription and formatting never fail the job; video-sync-report.py writes the PR description and step summary, and the logs and run summaries are uploaded as an artifact. A temporary pull_request trigger runs dry runs (no PR, diff attached) on the PR that adds the workflow; it has to be removed before merging. test-scripts.yml runs scripts/run-tests.sh when scripts change.
Checking the 1,710 published transcripts showed that 84-93% of the videos of 45-90 minutes lost more than 10% of their words: they were formatted in one request, which ran into the output token limit or got condensed, and a response cut off at max_tokens was written anyway. - Reject responses that stop at max_tokens - Validate the HTML: only h3/p/a/blockquote, only data-t attributes, data-t within the video (a small overshoot is a warning), and at least 90% of the spoken words outside pull quotes; invalid output is never written - Chunk videos longer than 20 minutes into 15-minute segments (was 90 and 30 minutes) and request the chunks concurrently - Add --check-existing (report invalid published transcripts, no API calls) and --reformat-invalid - Log model, tokens, stop reason, latency, and request ID per call; record estimated cost; add --limit, --dry-run, $TRANSCRIPT_MODEL, exit codes 0/1/2, and atomic writes
The end-to-end test runs the four scripts and the report against a fixture repo with all services faked, and checks that a second run changes nothing, that a renamed video keeps its transcripts, that stats only change on a refresh, and that blocked downloads are retried the next run.
The model copies each cue's text verbatim but often gets its timestamp wrong. Across the 1,706 published transcripts only 64% of the data-t values are within a second of their cue (p90: 60 s off); 271 transcripts have timestamps beyond the end of the video. repair_timestamps() walks the anchors in order, finds each one's first words in the VTT (searching forward from the previous match), and sets data-t to that cue's start. Pull quotes reuse the value of their prose anchor, and anchors split by inject_links stay together. Unmatched anchors keep the model's value, capped at the last cue. On the published transcripts this brings exact timestamps from 64% to 97% (p90: 0 s) without changing anything but data-t values. New transcripts are repaired before validation, and --repair-timestamps fixes existing ones without API calls.
…ugging Found in a local end-to-end run: Whisper returned only "." cues for the last 22 minutes of a video, and the transcript passed every check. One of the published transcripts has the same problem. - transcribe.py: retry a chunk once when it has more than 2 minutes of cues without words, and reject the transcript (degenerate_transcript) when it still does - transcribe.py: aim audio chunks at 90% of the API limit, so a chunk that comes out slightly larger isn't split again into a few seconds - format-transcriptions.py: log and record the coverage per chunk, and save rejected output with --rejected-dir (the workflow keeps it in the artifact) - videolib.py: also quiet the httpx2 logger
In a local run the model skipped small talk, logistics ("can you see my
slides?"), and parts of the Q&A, leaving chunks at 78-82% of the spoken
words. The prompt said to keep the text verbatim but never said not to
leave things out. With an explicit instruction to include every cue, the
same video came back at 97-100% per chunk.
Chunks that still lose more than 10% of their words are requested once
more, and the better answer is kept; both requests count toward the usage
and cost in the summary.
A short anchor such as "For us," could match a later occurrence hundreds of words ahead, after which every following anchor was behind the search position: in one real transcript 565 of 1,298 anchors went unmatched. Anchors shorter than the 4-word key now only match within the next 40 words. That video now has 2 unmatched anchors, and exact timestamps across the published transcripts go from 97.1% to 97.4%.
✅ Deploy Preview for posit-open-source canceled.
|
Contributor
The first dry run on the runners failed both downloads with "Sign in to confirm you're not a bot". Run the bgutil provider as a service container and add its yt-dlp plugin, with debug logging for transcribe.py, to see whether PO tokens get the downloads through.
yt-dlp's verbose output shows the JavaScript runtime, PO token providers, and player clients it uses, which is what we need to see when downloads are blocked.
Two dry runs on the runners, the second with the bgutil PO token provider (loaded and working, with Deno for the JS challenges), both got "Sign in to confirm you're not a bot": the block is based on the runner's IP. So: - The nightly run no longer transcribes (a manual run can still try, with the new transcribe input); the PO token provider is removed again - It still formats transcripts with Claude, which works fine on the runners - format-transcriptions.py records the videos without a transcription.vtt, and the PR description lists them with the commands to transcribe them on the PR branch; the next run formats them and updates the PR
The model occasionally writes '<a a data-t="623">' or '<a data-t="="1876">' (4 of the published transcripts, and again during the re-format). Repair those instead of rejecting an otherwise complete transcript.
It ran dry runs on the PR that adds the workflow, which settled that YouTube blocks downloads from GitHub's runners. From now on the workflow runs on its schedule and by hand from main.
A pull quote whose text matches no prose anchor kept the model's value unchecked; one re-formatted transcript had data-t="46039" in a 1-hour video.
A ten-second network interruption failed 15 videos in a row: the backoff (1, 2, 4, 8 s) gave up before the connection was back. Now 6 attempts starting at 2 s, each wait capped at 60 s.
- inject_links: in plain text (such as an <h3>), a shorter name could match inside the URL of a link inserted just before, e.g. "Shiny" in /software/shiny-python/, producing broken HTML (once on the live site). Insert placeholders first and the links at the end. - coverage: count a cue whose text repeats an earlier cue once. Whisper sometimes loops on a line, and the model rightly leaves the copies out; three short videos were rejected for that.
3 of 4 tasks
jeroenjanssens
marked this pull request as ready for review
November 12, 2026 18:09
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Merge #487 first (catch-up of the videos since September plus the fixed transcripts). Then the first nightly run starts from an up-to-date site and should produce a small PR, the best test that the churn fix works. After merging this PR, run the workflow once by hand with
refresh_stats(view counts were last updated in July, so that PR touches nearly every video page).Prerequisite check
Related Issue
None.
Description of changes
Moves the manual video update (four scripts run by hand, in the right order) to a nightly GitHub Actions workflow that opens a pull request, like the course sync.
Workflow (
.github/workflows/sync-videos.yml), 05:00 UTC:update-youtube-videos.py: YouTube metadata →data/videos.tomlsync-videos.py: video pages and thumbnailstranscribe.py: audio download (yt-dlp) + Whisper →transcription.vttformat-transcriptions.py: Claude →transcription.txtIt then opens or updates an
automation/sync-videosPR with theposit-open-source-website-syncapp. If last night's PR is still open, the run builds on it instead of redoing its work. Logs and machine-readable run summaries are attached to every run.Transcripts stay a (small) local step. YouTube blocks audio downloads from GitHub's runners: both dry runs on this PR got "Sign in to confirm you're not a bot", the second one with the bgutil PO token provider loaded and working. The block is based on the runner's IP. So the nightly run skips step 3 (a manual run can still try), and the PR description lists the videos waiting for a transcript with the commands to make them on the PR branch (
just transcribe, then push). The next run formats them with Claude and updates the PR.Script changes
last_updatedonly change when a field changes; view counts are refreshed on Mondays only. Before, every run rewrote all ~1,700 entries and pages. Against the live API, a second run in a row now changes nothing.transcribe: falserespected,--limit, temporary download dirs, yt-dlp errors classified (bot check, PO token, 403, …) and logged in full, a circuit breaker after three blocked downloads, and detection and retry of Whisper output with long stretches of empty cues.--check-existing,--repair-timestamps, and--reformat-invalidmodes for the published transcripts (the fixes themselves are in Sync videos from YouTube and fix the published transcripts #487).--summary-jsonand exit codes 0 (ok), 1 (fatal), 2 (partial).Tests: 336 tests in 8 files (all services faked, plus a real ffmpeg test and an end-to-end test that checks a second run changes nothing), run by
just testand the newtest-scripts.ymlworkflow.Repository secrets
YOUTUBE_API_KEY,OPENAI_API_KEY, andANTHROPIC_API_KEYhave been added.Before merging
pull_requesttrigger fromsync-videos.yml.Screenshots or previews (if applicable)
Not applicable: no layout changes.
Checklist
CONTRIBUTING.mdguidelines.