Backfill pre-roll videos for existing product campaigns - #277
Merged
Merged
Conversation
Adds the sweep the rest of this package kept promising, a classifier for what counts as a product ad, and a backfill script that queues a render for each one. The sweep first, because without it the backfill would not work and several comments in this package were writing cheques nothing cashed. ad_video_jobs is the durable record and the BullMQ job is the transient work item, which is the right way round, but it leaves a gap: if Redis is down at save time, or the process dies between the insert and the enqueue, the row sits `queued` with nothing scheduled to look at it. processDueVideoRenders reschedules those after a minute's grace, and fails a row outright if it carries no design snapshot so a broken row cannot become a hot loop against Redis. It also means a backfill can simply insert rows from wherever it runs and let the worker schedule them. The classifier exists because "generate videos for the ads" is not the same set as "every campaign row". Of 450 campaigns, 269 point at blog posts and 4 at social links; only 177 advertise a product. Getting that wrong is expensive in one direction specifically — rendering videos for the posts that were excluded — so the shapes are taken from the live table rather than imagined. Most blog campaigns have no /blog/ path prefix at all: dev.profullstack.com/~anthony/blog/126-post.html 152 campaigns, ~user dir dev.to/chovy/<slug> 117 campaigns, /user/slug A naive /^\/blog/ match catches neither, which is why the publishing platforms are matched on the domain. Affiliate and referral links are kept as products: they advertise something a person can buy, which is the distinction that matters. An unparseable destination is deliberately not classified as a product — something that cannot be shown to be one should not get a render. The script calls the same queueCampaignVideo the dashboard calls, so backfilled videos cannot differ from freshly saved ones with nothing to explain why. It is re-runnable: jobs dedupe on a hash of the design, so a campaign that already has a job for its current copy is skipped rather than re-encoded. It passes bumpRevision false, because a backfill is not an edit and bumping would invalidate media an earlier pass already rendered for the same unchanged design. It never rewrites campaign copy and never changes campaign status. Dry run against production: 450 campaigns, 177 product, 269 blog, 4 social, 177 queueable with no campaign lacking a usable creative. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ThreatCrush Security Scan48 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 31 | LOW: 15
Snippets are redacted; ThreatCrush never prints matched credential material. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| */ | ||
| export function classifyCampaign(destinationUrl: string): CampaignKind { | ||
| let host = ""; | ||
| let path = "/"; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Queues a five-second pre-roll render for every existing product campaign. Three pieces: the sweep, a classifier, and the backfill script.
The sweep, because without it none of this works
ad_video_jobsis the durable record and the BullMQ job is the transient work item — the right way round, but it leaves a gap. If Redis is down when a campaign is saved, or the process dies between the insert and the enqueue, the row sitsqueuedwith nothing scheduled to look at it.Several comments in this package already promised "a sweep can pick it up." Nothing did.
processDueVideoRendersis that sweep: it reschedules stranded rows after a minute's grace, and fails a row outright if it carries no design snapshot, so a broken row can't become a hot loop against Redis. It also means a backfill can just insert rows from wherever it runs and let the worker schedule them.The classifier, because "the ads" isn't every campaign row
Of 450 campaigns, 269 point at blog posts and 4 at social links — only 177 advertise a product.
Getting this wrong is expensive in one direction specifically: rendering videos for the posts that were explicitly excluded. So the shapes come from the live table rather than from imagination. Most blog campaigns have no
/blog/path prefix at all:dev.profullstack.com/~anthony/blog/126-post.html~userdirectory, so/^\/blog/never matchesdev.to/chovy/<slug>/user/slugwith nothing blog-shaped in itMy own first pass at this classified only 12 campaigns as blog posts and 348 as "product deep links" — sampling the data is what caught it.
Affiliate and referral links are kept as products (
m.do.co/c/…, Amazon/dp/…,aff.php?aff=…): they advertise something a person can buy, which is the distinction that matters. An unparseable destination is deliberately not classified as a product — something we can't show to be one shouldn't get a render.9 tests pin this against real production URL shapes.
The script
Calls the same
queueCampaignVideothe dashboard calls, so backfilled videos can't differ from freshly-saved ones with nothing to explain why.bumpRevision: false— a backfill is not an edit, and bumping would invalidate media an earlier pass already rendered for the same unchanged design.rejectedcampaigns and creatives — a rejected campaign must not gain servable media.--dry-run,--limit N,--delay ms.Verification
Dry run against production:
That matches an independent SQL classification of the same table. Repo-wide: 2547 passed / 1 failed — the pre-existing
tracker-geofailure (geolite2-city-mmdbnot installed locally), unrelated. Root and worker typechecks clean.Rollout
Not run yet beyond the dry run. After this merges and deploys I'll run
--limit 1as a canary, confirm it produces a real validated MP4 end to end, then release the remaining 176. Rough cost: ~177 renders at roughly a minute or two each, concurrency 2, and about 8 MB of storage per campaign (~1.4 GB total) in the existingad-assetsbucket.