Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8046bd6
Add shared video pipeline helpers, test runner, and justfile recipes
jeroenjanssens Nov 12, 2026
6e80698
Only record real metadata changes in update-youtube-videos.py
jeroenjanssens Nov 12, 2026
e1ef6a6
Only write changed video pages in sync-videos.py and report changes
jeroenjanssens Nov 12, 2026
116d834
Make transcribe.py robust for unattended runs
jeroenjanssens Nov 12, 2026
1175f0e
Add video-sync-report.py to turn run summaries into a PR description
jeroenjanssens Nov 12, 2026
7eb48fe
Add nightly Sync Videos workflow and a workflow for the script tests
jeroenjanssens Nov 12, 2026
7045c5f
Validate formatted transcripts and chunk videos longer than 20 minutes
jeroenjanssens Nov 12, 2026
08c0a91
Add an end-to-end test of the video pipeline and document the video sync
jeroenjanssens Nov 12, 2026
78e70f7
Repair transcript timestamps from the VTT instead of trusting the model
jeroenjanssens Nov 12, 2026
9ba2781
Catch degenerate Whisper output and keep rejected transcripts for deb…
jeroenjanssens Nov 12, 2026
fd04cc1
Tell the model not to skip anything, and retry chunks that lost words
jeroenjanssens Nov 12, 2026
680cdb2
Keep short anchors from derailing the timestamp repair
jeroenjanssens Nov 12, 2026
197621e
Try the bgutil PO token provider for yt-dlp on GitHub's runners
jeroenjanssens Nov 12, 2026
0adff3b
Make yt-dlp verbose when transcribe.py runs with --verbose
jeroenjanssens Nov 12, 2026
0fa12cf
Transcribe locally: YouTube blocks downloads from GitHub's runners
jeroenjanssens Nov 12, 2026
9aa7476
Fix the model's known <a> tag typos before validating
jeroenjanssens Nov 12, 2026
49ef184
Remove the temporary pull_request trigger from Sync Videos
jeroenjanssens Nov 12, 2026
816edb6
Cap unmatched pull-quote timestamps at the last cue too
jeroenjanssens Nov 12, 2026
7cdffc6
Retry API calls for about a minute instead of 15 seconds
jeroenjanssens Nov 12, 2026
b40a0ed
Fix a link inserted inside another link, and Whisper repeats in coverage
jeroenjanssens Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/scripts/restore-pending-videos.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env bash
# Carry over the work of an unmerged sync PR into this run.
#
# The nightly sync starts from main. If last night's PR hasn't been merged
# yet, its new video directories and transcripts would otherwise be created
# (and paid for) again, and the nondeterministic formatting would rewrite
# the PR every night. So: take every file the PR branch *added* under
# content/resources/videos/ (relative to its merge base with main) and put
# it in the working tree, unless main has a file at that path by now.
# Files the PR modified are not restored; the scripts regenerate them.
#
# Usage: restore-pending-videos.sh [BRANCH] [BASE]
# BRANCH defaults to automation/sync-videos, BASE to main.
# Needs a checkout with full history (actions/checkout fetch-depth: 0).

set -euo pipefail

branch="${1:-automation/sync-videos}"
base="${2:-main}"
prefix="content/resources/videos/"

if ! git ls-remote --exit-code --heads origin "$branch" >/dev/null 2>&1; then
echo "No pending branch $branch; nothing to restore."
exit 0
fi

git fetch --quiet --no-tags origin "$branch" "$base"
remote_branch="origin/$branch"
remote_base="origin/$base"

added=()
while IFS= read -r -d '' path; do
added+=("$path")
done < <(git diff -z --name-only --diff-filter=A "$remote_base...$remote_branch" -- "$prefix")

restored=()
skipped=0
for path in ${added[@]+"${added[@]}"}; do
if [[ -e "$path" ]]; then
echo " keep main's version: $path"
skipped=$((skipped + 1))
continue
fi
restored+=("$path")
done

if ((${#restored[@]})); then
# --worktree only: leave the index alone, create-pull-request picks the
# files up as changes.
git restore --source="$remote_branch" --worktree -- "${restored[@]}"
printf ' restored: %s\n' "${restored[@]}"
fi

echo "Restored ${#restored[@]} file(s) from $branch (${#added[@]} added there, $skipped already on $base)."
229 changes: 229 additions & 0 deletions .github/workflows/sync-videos.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
name: Sync Videos

# Syncs Posit's YouTube videos into content/resources/videos/ and opens (or
# updates) a pull request with the changes, like sync-courses.yml:
#
# 1. update-youtube-videos.py YouTube metadata -> data/videos.toml
# (view counts are only refreshed on Mondays)
# 2. sync-videos.py data/videos.toml -> video pages, thumbnails
# 3. transcribe.py new videos -> transcription.vtt (yt-dlp, Whisper)
# 4. format-transcriptions.py transcription.vtt -> transcription.txt (Claude)
#
# Step 3 only runs when asked for (the `transcribe` input): YouTube blocks
# audio downloads from GitHub's runners ("Sign in to confirm you're not a
# bot"), also with a PO token provider (tested in November 2026). So
# transcripts are made locally: the PR description lists the videos that need
# one, you run `just transcribe` on the PR branch and push, and the next run
# formats them (step 4) and updates the PR. Steps 3 and 4 never fail the job.
# Logs and machine-readable run summaries of every step are attached to the
# workflow run as an artifact.

on:
schedule:
- cron: "0 5 * * *" # 5 AM UTC: before sync-courses (6 AM) and the nightly production deploy (8 AM)
workflow_dispatch:
inputs:
refresh_stats:
description: Refresh the view, like, and comment counts of all videos
type: boolean
default: false
transcribe:
description: Try to download audio and transcribe new videos (YouTube usually blocks GitHub's runners)
type: boolean
default: false
limit:
description: Maximum number of videos to transcribe and format
type: number
default: 15
dry_run:
description: Don't open a pull request; attach the diff to the run instead
type: boolean
default: false

concurrency:
group: sync-videos
cancel-in-progress: false

# Pushing the branch and opening the PR use the app token below, not
# GITHUB_TOKEN, so that deploy-preview.yml runs on the sync PRs.
permissions:
contents: read

jobs:
sync-videos:
runs-on: ubuntu-latest
timeout-minutes: 120
env:
# Only scheduled runs and manual runs on main open a pull request.
DRY_RUN: ${{ inputs.dry_run == true || github.ref != 'refs/heads/main' }}
LIMIT: ${{ inputs.limit || 15 }}
TRANSCRIBE: ${{ inputs.transcribe == true }}
REFRESH_STATS_INPUT: ${{ inputs.refresh_stats == true }}
TRANSCRIPT_MODEL: claude-sonnet-4-6
# Keep log lines unwrapped in the job log
COLUMNS: "200"
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
fetch-depth: 0 # restore-pending-videos.sh diffs against the merge base

- name: Prepare
id: prepare
shell: bash
run: |
sync_dir="$RUNNER_TEMP/video-sync"
mkdir -p "$sync_dir/logs" "$sync_dir/summaries"
echo "SYNC_DIR=$sync_dir" >> "$GITHUB_ENV"

if [ "$REFRESH_STATS_INPUT" = "true" ] || [ "$(date -u +%u)" = "1" ]; then
refresh_stats=true
else
refresh_stats=false
fi
echo "refresh_stats=$refresh_stats" >> "$GITHUB_OUTPUT"

if [ "$TRANSCRIBE" = "true" ]; then
echo "expect=update-youtube-videos,sync-videos,transcribe,format-transcriptions" >> "$GITHUB_OUTPUT"
else
echo "expect=update-youtube-videos,sync-videos,format-transcriptions" >> "$GITHUB_OUTPUT"
fi
echo "event=$GITHUB_EVENT_NAME ref=$GITHUB_REF dry_run=$DRY_RUN limit=$LIMIT"
echo "transcribe=$TRANSCRIBE refresh_stats=$refresh_stats model=$TRANSCRIPT_MODEL"

- name: Restore files from the open sync pull request
shell: bash
run: ./.github/scripts/restore-pending-videos.sh automation/sync-videos main 2>&1 | tee "$SYNC_DIR/logs/0-restore-pending-videos.log"

- name: Setup uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2

- name: Setup Deno (JavaScript runtime for yt-dlp)
if: env.TRANSCRIBE == 'true'
uses: denoland/setup-deno@22d081ff2d3a40755e97629de92e3bcbfa7cf2ed # v2.0.5

- name: Install ffmpeg
if: env.TRANSCRIBE == 'true'
shell: bash
run: |
if ! command -v ffmpeg >/dev/null || ! command -v ffprobe >/dev/null; then
sudo apt-get update -q
sudo apt-get install -y -q --no-install-recommends ffmpeg
fi
ffmpeg -version | head -1
deno --version | head -1

- name: 1. Update YouTube metadata
id: update
shell: bash
env:
YOUTUBE_API_KEY: ${{ secrets.YOUTUBE_API_KEY }}
REFRESH_STATS: ${{ steps.prepare.outputs.refresh_stats }}
run: |
set +e
args=()
if [ "$REFRESH_STATS" = "true" ]; then args+=(--refresh-stats); fi
./scripts/update-youtube-videos.py "${args[@]}" \
--summary-json "$SYNC_DIR/summaries/1-update-youtube-videos.json" \
2>&1 | tee "$SYNC_DIR/logs/1-update-youtube-videos.log"
status=${PIPESTATUS[0]}
echo "exit_code=$status" >> "$GITHUB_OUTPUT"
# Exit code 2 (a source failed) is partial success: keep going.
[ "$status" -eq 0 ] || [ "$status" -eq 2 ]

- name: 2. Sync video pages
id: sync
shell: bash
run: |
set +e
./scripts/sync-videos.py \
--summary-json "$SYNC_DIR/summaries/2-sync-videos.json" \
2>&1 | tee "$SYNC_DIR/logs/2-sync-videos.log"
status=${PIPESTATUS[0]}
echo "exit_code=$status" >> "$GITHUB_OUTPUT"
[ "$status" -eq 0 ] || [ "$status" -eq 2 ]

- name: 3. Transcribe new videos
id: transcribe
if: env.TRANSCRIBE == 'true'
shell: bash
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
set +e
./scripts/transcribe.py --limit "$LIMIT" --jobs 3 \
--summary-json "$SYNC_DIR/summaries/3-transcribe.json" \
2>&1 | tee "$SYNC_DIR/logs/3-transcribe.log"
status=${PIPESTATUS[0]}
echo "exit_code=$status" >> "$GITHUB_OUTPUT"
# Never fail the job here: failures are listed in the PR description.
if [ "$status" -ne 0 ]; then echo "::warning title=Transcription::transcribe.py exited with $status"; fi

- name: 4. Format transcriptions
id: format
shell: bash
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
set +e
./scripts/format-transcriptions.py --limit "$LIMIT" \
--rejected-dir "$SYNC_DIR/rejected" \
--summary-json "$SYNC_DIR/summaries/4-format-transcriptions.json" \
2>&1 | tee "$SYNC_DIR/logs/4-format-transcriptions.log"
status=${PIPESTATUS[0]}
echo "exit_code=$status" >> "$GITHUB_OUTPUT"
if [ "$status" -ne 0 ]; then echo "::warning title=Formatting::format-transcriptions.py exited with $status"; fi

- name: Write report
if: always()
shell: bash
env:
EXPECT: ${{ steps.prepare.outputs.expect }}
run: |
./scripts/video-sync-report.py "$SYNC_DIR/summaries" \
--expect "$EXPECT" --output "$SYNC_DIR/pr-body.md" --annotations
cat "$SYNC_DIR/pr-body.md" >> "$GITHUB_STEP_SUMMARY"

- name: Save the diff (dry run)
if: always() && env.DRY_RUN == 'true'
shell: bash
run: |
git add --intent-to-add -- data/videos.toml content
git diff --stat -- data/videos.toml content > "$SYNC_DIR/diff-stat.txt"
git diff -- data/videos.toml content > "$SYNC_DIR/diff.patch"
echo "Dry run: no pull request. $(tail -1 "$SYNC_DIR/diff-stat.txt")"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "**Dry run:** no pull request was opened. Changes: $(tail -1 "$SYNC_DIR/diff-stat.txt")" >> "$GITHUB_STEP_SUMMARY"

- name: Upload logs and run summaries
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: video-sync-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/video-sync
retention-days: 30
if-no-files-found: warn

- name: Generate app token
if: env.DRY_RUN != 'true'
id: app-token
uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0
with:
client-id: ${{ secrets.POSIT_OPEN_SOURCE_WEBSITE_SYNC_CLIENT_ID }}
private-key: ${{ secrets.POSIT_OPEN_SOURCE_WEBSITE_SYNC_PEM }}

- name: Create or update pull request
if: env.DRY_RUN != 'true'
uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1
with:
token: ${{ steps.app-token.outputs.token }}
sign-commits: true
branch: automation/sync-videos
delete-branch: true
# All of content/: renamed videos also update links in other pages.
add-paths: |
data/videos.toml
content
commit-message: Sync videos from YouTube
title: Sync videos from YouTube
body-path: ${{ runner.temp }}/video-sync/pr-body.md
41 changes: 41 additions & 0 deletions .github/workflows/test-scripts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Test Scripts

# Runs the tests of the Python scripts (scripts/test-*.py) when they change.

on:
pull_request:
paths:
- "scripts/**"
- ".github/workflows/test-scripts.yml"
- "justfile"
push:
branches: ["main"]
paths:
- "scripts/**"
- ".github/workflows/test-scripts.yml"
- "justfile"
workflow_dispatch:

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Setup uv
uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2

- name: Install ffmpeg
run: |
if ! command -v ffmpeg >/dev/null || ! command -v ffprobe >/dev/null; then
sudo apt-get update -q
sudo apt-get install -y -q --no-install-recommends ffmpeg
fi

- name: Run tests
run: ./scripts/run-tests.sh -ra
Loading
Loading