fix(apify): include live streams in YouTube profile scrapes#754
Conversation
The streamers/youtube-scraper actor takes three per-tab depth knobs; posts was forwarded to maxResults (Videos) and maxResultsShorts (Shorts) but maxResultStreams stayed hardcoded to 0, so the Streams tab never appeared in scrape datasets. Forward posts to it like the other two. Legacy snapshot (posts omitted) still excludes them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 11 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
No issues found across 2 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant Client as HTTP Client
participant API as POST /api/socials/{id}/scrape
participant Scraper as startYoutubeProfileScraping()
participant Apify as Apify Actor SDK
participant Webhook as Webhook Handler
Note over Client,Webhook: YouTube Profile Scrape – Current State (after fix)
Client->>API: POST /api/socials/{id}/scrape?posts=N
API->>Scraper: startYoutubeProfileScraping(handle, posts)
alt posts provided (e.g., N=20)
Scraper->>Scraper: Build input = {<br/> startUrls,<br/> maxResults: N,<br/> maxResultsShorts: N,<br/> maxResultStreams: N,<br/> ...DEFAULT_INPUT<br/>}
Note over Scraper: NEW: streams enabled via posts value
else posts omitted (legacy snapshot)
Scraper->>Scraper: Build input = {<br/> startUrls,<br/> maxResults: 1,<br/> maxResultsShorts: 0,<br/> maxResultStreams: 0,<br/> ...DEFAULT_INPUT<br/>}
Note over Scraper: Streams still excluded (unchanged)
end
Scraper->>Apify: call("streamers/youtube-scraper", input, webhooks)
Apify-->>Scraper: { runId, datasetId }
Scraper-->>API: { runId, datasetId }
API-->>Client: 200 { runId, datasetId }
Note over Webhook: Later, when actor completes...
Apify-->>Webhook: POST webhook with dataset items
Webhook->>Webhook: Persist channel profile from items[0]
Webhook-->>Apify: ACK
Auto-approved: Forwards posts to maxResultStreams so live streams are included with videos and shorts; tests updated.
Re-trigger cubic
| ...DEFAULT_INPUT, | ||
| startUrls: [{ url: targetUrl }], | ||
| // The legacy snapshot excludes Shorts; a requested posts depth includes them. | ||
| // The legacy snapshot excludes Shorts and streams; a requested posts depth includes both. |
There was a problem hiding this comment.
YAGNI - remove legacy notes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Auto-approved: Trivial forward of posts to maxResultStreams in YouTube scrape function, mirroring existing logic for videos and Shorts; test updated. Low impact, well-reasoned, no risk to core logic or infrastructure.
Re-trigger cubic
✅ Preview deployment test — streams come backDeployment: Test: Result — dataset is now 3-way, 15 items:
Stream items returned (newest-first from the Streams tab, each with views/likes/duration like regular videos):
This also resolves the open question from the PR description: the 🤖 Generated with Claude Code |
1 participant
Why
POST /api/socials/{id}/scrape?posts=Non a YouTube profile returns up to N videos + N Shorts but zero live streams (verified live on @mycowtf: 50-item dataset = 25 videos + 25 shorts, 0 streams, while the channel's Streams tab has 27). Thestreamers/youtube-scraperactor takes three per-tab depth knobs;postswas forwarded tomaxResults(Videos) andmaxResultsShorts(Shorts), butmaxResultStreamsstayed hardcoded to0inDEFAULT_INPUT— so the Streams tab was never scraped.What
Forward
poststomaxResultStreamsexactly like the other two knobs (one line instartYoutubeProfileScraping.ts). The legacy profile snapshot (postsomitted) still excludes streams, matching current behavior.TDD: test updated first (red —
maxResultStreams: 20unmet, stayed 0), then the fix (green). Full suite: 705 files / 3893 tests pass.Downstream is already stream-safe: the webhook handler only persists the channel profile from
items[0], andGET /api/apify/runs/{runId}returns raw dataset items untouched.Pricing note (no change needed)
getSocialScrapeCreditCostwas margin-checked against a 2×posts YouTube worst case (chat#1836); streams make it 3×posts, dropping worst-case margin atposts=100from ~1.75× to ~1.17× — still profitable, keeping the flat5 + postsrule (confirmed with @sweetmantech).Verify
POST /api/socials/{id}/scrape?posts=25on a channel with streams → dataset includes Streams-tab items (also confirms theisLive: falsedefault input is a search-mode filter that doesn't suppress channel-scrape stream items)🤖 Generated with Claude Code
Summary by cubic
Include live streams in YouTube profile scrapes by forwarding the
postsparameter tomaxResultStreams, so requests now return videos, Shorts, and streams whenpostsis set. Whenpostsis omitted, the legacy snapshot remains (Shorts and streams excluded).poststomaxResultStreamsin startYoutubeProfileScraping.ts (aligned withmaxResultsandmaxResultsShorts); removed an outdated inline legacy note.postsand excluded without it forstreamers/youtube-scraper.Written for commit 3e6e886. Summary will update on new commits.