feat(apify): persist TikTok/X post rows like Instagram (chat#1840)#753
Conversation
TikTok and X scrapes persisted follower counts (api#740) but no post
rows — only the Instagram handler wrote posts/social_posts, so
GET /api/artists/{id}/posts stayed empty for those platforms and
succeeded runs left no reusable content data (issue evidence: runs
9AYX8xyaHWyHtnGtC / bx3asRqfbNnkKgogG SUCCEEDED with real items, zero
rows landed).
- lib/apify/persistPostsForSocial.ts — the posts half of the Instagram
flow (upsertPosts -> getPosts -> link via upsertSocialPosts), shared
by both handlers; no-ops on empty rows, skips linking when the social
row is missing.
- TikTok: post_url from webVideoUrl, updated_at from createTimeISO
(fields verified on real run 9AYX8xyaHWyHtnGtC).
- X: post_url from the tweet url, createdAt converted from Twitter's
legacy date format to ISO (verified on real run bx3asRqfbNnkKgogG);
invalid dates fall back to the column default.
TDD RED->GREEN: helper suite (persist+link, empty no-op, missing-social
skip) + per-platform handler assertions with real-run fixtures. Full
lib/apify + lib/supabase suites green (238 tests); tsc no new errors vs
main; eslint clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 13 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (3)
📝 WalkthroughWalkthroughAdds a new ChangesPost persistence feature
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ScraperHandler as TikTok/Twitter Handler
participant persistPostsForSocial
participant PostsTable
participant SocialsTable
participant SocialPostsTable
ScraperHandler->>ScraperHandler: extract post URLs/timestamps
ScraperHandler->>persistPostsForSocial: postRows, profileUrl
persistPostsForSocial->>PostsTable: upsert postRows
persistPostsForSocial->>PostsTable: fetch posts by post_url
persistPostsForSocial->>SocialsTable: select social by profileUrl
alt social and posts found
persistPostsForSocial->>SocialPostsTable: upsert social_posts links
end
persistPostsForSocial-->>ScraperHandler: { posts, social }
ScraperHandler-->>ScraperHandler: return { social, posts }
Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Preview verification — round 1 (2026-07-03)Preview The end-to-end Done-when ( What was verified against reality today:
Will post round 2 (live scrape → webhook → posts diff) with a fresh admin token, either against the |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 6 files
Confidence score: 3/5
- In
lib/apify/tiktok/handleTiktokProfileScraperResults.ts, forwarding rawcreateTimeISOintoposts.updated_atcan cause otherwise valid TikTok rows to fail persistence when that field is malformed, which risks dropped posts during ingestion — normalize/validate the timestamp (or fall back safely) before writing. - In
lib/apify/twitter/handleTwitterProfileScraperResults.ts, relying onnew Date(value)for Twitter’s legacy date format can parse inconsistently across runtimes, causing tweet times to silently fall back to the DB default and skew post chronology — switch to a deterministic parser for that actor format before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/twitter/handleTwitterProfileScraperResults.ts">
<violation number="1" location="lib/apify/twitter/handleTwitterProfileScraperResults.ts:29">
P2: Tweet timestamps can silently fall back to the posts-table default on runtimes that do not parse Twitter’s legacy date string, because `new Date(value)` relies on implementation-defined parsing for this actor format. A small explicit parser for `EEE MMM dd HH:mm:ss Z yyyy` would keep `updated_at` tied to the tweet time across runtimes.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Webhook as Apify Webhook
participant Route as API Route
participant Handler as TikTok/X Handler
participant Persist as persistPostsForSocial
participant DB as Supabase (DB)
Webhook->>Route: POST /api/socials/{id}/scrape (webhook)
Route->>Handler: handleTiktok/TwitterProfileScraperResults(payload)
alt dataset has items with valid data
Handler->>Handler: Extract post rows from items
Note over Handler: TikTok: webVideoUrl → post_url, createTimeISO → updated_at
Note over Handler: X: url → post_url, createdAt → ISO (invalid/absent → column default)
Handler->>DB: upsertSocials([normalized profile])
DB-->>Handler: social row
Handler->>Persist: persistPostsForSocial({postRows, profileUrl})
Persist->>DB: upsertPosts(postRows)
DB-->>Persist:
Persist->>DB: getPosts(postUrls)
DB-->>Persist: posts with IDs
Persist->>DB: selectSocials(profileUrl)
DB-->>Persist: social row or null
alt social found AND posts exist
Persist->>DB: upsertSocialPosts(links)
else social not found
Note over Persist: Skip linking – posts stored unlinked
end
Persist-->>Handler: {posts, social}
Handler-->>Route: {social, posts}
else dataset empty (or no post data)
Handler->>Handler: No‑op – skip upsertSocials and persistPostsForSocial
Handler-->>Route: {social: null}
end
Route-->>Webhook: 200 OK
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| * (the posts upsert applies its column default). */ | ||
| const toIsoDate = (value?: string): string | undefined => { | ||
| if (!value) return undefined; | ||
| const parsed = new Date(value); |
There was a problem hiding this comment.
P2: Tweet timestamps can silently fall back to the posts-table default on runtimes that do not parse Twitter’s legacy date string, because new Date(value) relies on implementation-defined parsing for this actor format. A small explicit parser for EEE MMM dd HH:mm:ss Z yyyy would keep updated_at tied to the tweet time across runtimes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/twitter/handleTwitterProfileScraperResults.ts, line 29:
<comment>Tweet timestamps can silently fall back to the posts-table default on runtimes that do not parse Twitter’s legacy date string, because `new Date(value)` relies on implementation-defined parsing for this actor format. A small explicit parser for `EEE MMM dd HH:mm:ss Z yyyy` would keep `updated_at` tied to the tweet time across runtimes.</comment>
<file context>
@@ -16,9 +21,19 @@ type TweetItem = {
+ * (the posts upsert applies its column default). */
+const toIsoDate = (value?: string): string | undefined => {
+ if (!value) return undefined;
+ const parsed = new Date(value);
+ return isNaN(parsed.getTime()) ? undefined : parsed.toISOString();
+};
</file context>
There was a problem hiding this comment.
Triaged in 54a21a5: kept new Date() rather than a hand-rolled EEE MMM dd HH:mm:ss Z yyyy parser — this service deploys only to Node/V8 (Vercel functions), where parsing of this format is deterministic, and the extracted toIsoDate now has a unit test pinning the exact Twitter string to its ISO result, so any runtime change fails loudly in CI instead of silently. A custom parser adds surface without a reachable failure mode here (KISS/YAGNI); the graceful fallback (unparseable → column default) already bounds the worst case.
There was a problem hiding this comment.
The parent comment is too broad for this PR: this service only runs on Node/V8, the legacy Twitter format is pinned in unit tests, and unparseable dates already fall back to the column default. In that setup, a hand-rolled parser adds surface without a reachable failure mode here.
Thanks for the feedback! I've saved this as a new learning to improve future reviews.
Preview verification — round 2 (2026-07-03) ✅ end-to-endPreview Baseline (before): Live run:
One data finding along the way (not a code issue in this PR): the artist was linked to the legacy Cost note: 2 × (5+10) = 30 credits on the testing account, one Apify actor run per platform. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
lib/apify/twitter/handleTwitterProfileScraperResults.ts (1)
24-31: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
toIsoDaterelies on implementation-defined Date parsing.
new Date(value)on Twitter's legacy format ("Thu Jul 02 17:21:21 +0000 2026") is not part of the format the ECMAScript spec guarantees — "the key rule: only string formats defined in the ECMAScript specification are guaranteed to work across all environments", everything else is implementation-specific. V8/Node happens to parse this format today (and the author verified against real data), but this isn't a spec guarantee across runtimes/versions.The
isNaNguard is good defensive coding regardless. If you want to remove the implementation-specific dependency, an explicit parser (e.g.date-fns/parsewith the known Twitter format string) would be more robust — though given this is already verified against production data, this is more a documented risk than a required fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/apify/twitter/handleTwitterProfileScraperResults.ts` around lines 24 - 31, The `toIsoDate` helper currently depends on `new Date(value)` parsing Twitter’s legacy timestamp format, which is implementation-specific. Update `toIsoDate` to use an explicit parser for the known format (for example in the `handleTwitterProfileScraperResults` flow) instead of relying on built-in `Date` parsing, while keeping the existing invalid/absent date handling and `isNaN` guard. Use the `toIsoDate` symbol to locate the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@lib/apify/twitter/handleTwitterProfileScraperResults.ts`:
- Around line 24-31: The `toIsoDate` helper currently depends on `new
Date(value)` parsing Twitter’s legacy timestamp format, which is
implementation-specific. Update `toIsoDate` to use an explicit parser for the
known format (for example in the `handleTwitterProfileScraperResults` flow)
instead of relying on built-in `Date` parsing, while keeping the existing
invalid/absent date handling and `isNaN` guard. Use the `toIsoDate` symbol to
locate the change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 51462068-aa90-459a-b8cb-9c879e0577c9
⛔ Files ignored due to path filters (3)
lib/apify/__tests__/persistPostsForSocial.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**
📒 Files selected for processing (3)
lib/apify/persistPostsForSocial.tslib/apify/tiktok/handleTiktokProfileScraperResults.tslib/apify/twitter/handleTwitterProfileScraperResults.ts
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
| /** The actor emits Twitter's legacy format ("Thu Jul 02 17:21:21 +0000 2026"); | ||
| * Postgres can't parse it, so convert to ISO. Invalid/absent dates → undefined | ||
| * (the posts upsert applies its column default). */ | ||
| const toIsoDate = (value?: string): string | undefined => { |
There was a problem hiding this comment.
SRP - new lib file for toIsoDate
There was a problem hiding this comment.
Done in 54a21a5 — toIsoDate extracted to lib/apify/toIsoDate.ts with its own unit suite (Twitter legacy format, ISO passthrough, unparseable → undefined, empty/absent → undefined). Both handlers now import it.
Review feedback (api#753): - SRP: toIsoDate moves from an inline helper in the Twitter handler to lib/apify/toIsoDate.ts (one exported function per file), with its own unit suite pinning the Twitter legacy format on this Node/V8 runtime. - cubic P2 (valid): the TikTok mapper forwarded raw createTimeISO into posts.updated_at, so one malformed date could fail the whole upsert batch. Both handlers now normalize through toIsoDate; unparseable dates fall back to the column default. lib/apify suite: 86 passed; prettier + eslint + tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 5 files (changes from recent commits).
Requires human review: Adds new business logic for persisting posts from TikTok/X scrapers with database writes and date parsing. Not a trivial change; requires human review for data integrity and edge cases.
Re-trigger cubic
1 participant
Item 2 of recoupable/chat#1840. Independent of api#752 (item 1) — no ordering constraint between the two. Contract note: docs#262 (merge docs first). No database dependency.
Why
api#740 made TikTok/X scrapes persist follower counts, but post rows still only persisted for Instagram —
handleInstagramProfileScraperResultswas the sole caller ofupsertPosts/upsertSocialPosts. Succeeded TikTok/X runs leftGET /api/artists/{id}/postsempty (issue evidence: runs9AYX8xyaHWyHtnGtC/bx3asRqfbNnkKgogGSUCCEEDED with real items, zero post rows landed). This is the half of item 2 that #740 didn't cover.What
lib/apify/persistPostsForSocial.ts— the posts half of the Instagram flow (upsertPosts→getPosts→ link viaupsertSocialPosts), extracted as a shared helper (SRP: one export). No-ops on empty rows; skips linking when the social row isn't found. Called afterupsertSocialsso first-seen profiles link correctly.post_url←webVideoUrl,updated_at←createTimeISO— fields verified on real run9AYX8xyaHWyHtnGtC.post_url← tweeturl(display casing kept — posts key on exact URL),updated_at←createdAtconverted from Twitter's legacy format (Thu Jul 02 17:21:21 +0000 2026) to ISO; invalid dates fall back to the column default. Verified on real runbx3asRqfbNnkKgogG.Tests (TDD, RED→GREEN)
upsertSocialPostsrows), empty-rows no-op, missing-social skip.lib/apify+lib/supabasesuites: 238 passed;tsc --noEmitno new errors vs main; eslint clean.Done-when (issue) — preview verification to follow as a comment
POST /api/socials/{id}/scrapeon a TikTok social and an X social → within one webhook cycle,GET /api/artists/{id}/postsreturns the scraped posts — verified live 2026-07-03 (49 → 69 posts: +10 tiktok, +10 x), results table in comments🤖 Generated with Claude Code
Summary by cubic
Persist TikTok and X post rows and link them to socials, matching the Instagram flow, so artist posts now populate for these platforms. Timestamps are normalized to ISO and bad dates no longer break upserts. Implements item 2 of
chat#1840.New Features
persistPostsForSocialto upsert posts and link to the matching social; no-op on empty, skip link if social missing.webVideoUrl->post_url;createTimeISO->updated_atviatoIsoDate(unparseable dates are omitted so the column default applies).url->post_url; legacycreatedAt-> ISOupdated_atviatoIsoDate.GET /api/artists/{id}/postsnow returns TikTok/X posts after a scrape.Dependencies
docs#262(merge first). Independent ofapi#752.Written for commit 54a21a5. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes