feat(apify): profile-level fallback for tweetless X accounts#756
feat(apify): profile-level fallback for tweetless X accounts#756sweetmantech wants to merge 1 commit into
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 17 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (4)
📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 9 files
Confidence score: 5/5
- In
lib/apify/twitter/__tests__/startTwitterUserScraping.test.ts, the riskiest gap is missing coverage for handles with surrounding whitespace, so a subtle parsing regression could slip through and send an untrimmed value in the API call — add the ` - KETTAMA_
style test case before merging to confirm it resolves totwitterHandles: ["KETTAMA_"]`.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/twitter/__tests__/startTwitterUserScraping.test.ts">
<violation number="1" location="lib/apify/twitter/__tests__/startTwitterUserScraping.test.ts:15">
P3: Add a test case for a handle with surrounding whitespace (e.g., `" KETTAMA_ "`) to verify `trim()` handles the edge case correctly — that it resolves to `twitterHandles: ["KETTAMA_"]` in the API call rather than keeping the whitespace or rejecting. The `" "` case only validates the empty-after-trim guard, not the trim-and-pass path.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Webhook as Apify Webhook
participant Validator as validateApifyWebhookRequest
participant Router as getApifyResultHandler
participant TweetHandler as handleTwitterProfileScraperResults
participant KVStore as Apify Key-Value Store
participant FallbackStarter as startTwitterUserScraping
participant UserHandler as handleTwitterUserScraperResults
participant Dataset as Apify Dataset
participant DB as Supabase socials
Note over Webhook,DB: Tweet-based scrape webhook fires (empty dataset)
Webhook->>Validator: POST webhook payload
Validator->>Validator: Parse & validate (zod now passes defaultKeyValueStoreId)
Validator-->>Router: Parsed payload (actorId: nfp1fpt5gUlBwPcor)
Router->>Router: Lookup handler → handleTwitterProfileScraperResults
Router-->>TweetHandler: Route to handler
TweetHandler->>Dataset: listItems() on defaultDatasetId
Dataset-->>TweetHandler: [] (empty - tweetless account)
alt Empty dataset fallback
TweetHandler->>TweetHandler: Check defaultKeyValueStoreId exists
alt KV store ID present
TweetHandler->>KVStore: getRecord("INPUT")
KVStore-->>TweetHandler: { twitterHandles: ["KETTAMA_"] }
TweetHandler->>TweetHandler: Extract handle from INPUT
TweetHandler->>FallbackStarter: start("KETTAMA_")
FallbackStarter->>FallbackStarter: Start apidojo/twitter-user-scraper with {twitterHandles, maxItems:1}
FallbackStarter-->>TweetHandler: { runId, datasetId }
TweetHandler-->>Webhook: { social: null, fallbackRunId }
else No KV store ID or no handle in INPUT
TweetHandler-->>Webhook: { social: null } (no felback)
end
end
Note over Webhook,DB: Fallback user-scraper webhook fires
Webhook->>Validator: POST webhook payload (actorId: V38PZzpEgOfeeWvZY)
Validator-->>Router: Parsed payload
Router->>Router: Lookup handler → handleTwitterUserScraperResults
Router-->>UserHandler: Route to handler
UserHandler->>KVStore: getRecord("INPUT") on defaultKeyValueStoreId
KVStore-->>UserHandler: { twitterHandles: ["KETTAMA_"] }
UserHandler->>Dataset: listItems() on defaultDatasetId
Dataset-->>UserHandler: [padded related users..., requested user]
UserHandler->>UserHandler: Filter dataset to requested handle (case-insensitive match)
alt Requested user found
UserHandler->>UserHandler: Build social object (lowercase profile_url)
UserHandler->>DB: upsertSocials([social])
DB-->>UserHandler: Updated row
UserHandler-->>Webhook: { social }
else Requested user not in dataset
UserHandler-->>Webhook: { social: null }
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
|
||
| describe("startTwitterUserScraping", () => { | ||
| it("starts the profile-level user scraper for one handle", async () => { | ||
| const r = await startTwitterUserScraping("KETTAMA_"); |
There was a problem hiding this comment.
P3: Add a test case for a handle with surrounding whitespace (e.g., " KETTAMA_ ") to verify trim() handles the edge case correctly — that it resolves to twitterHandles: ["KETTAMA_"] in the API call rather than keeping the whitespace or rejecting. The " " case only validates the empty-after-trim guard, not the trim-and-pass path.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/twitter/__tests__/startTwitterUserScraping.test.ts, line 15:
<comment>Add a test case for a handle with surrounding whitespace (e.g., `" KETTAMA_ "`) to verify `trim()` handles the edge case correctly — that it resolves to `twitterHandles: ["KETTAMA_"]` in the API call rather than keeping the whitespace or rejecting. The `" "` case only validates the empty-after-trim guard, not the trim-and-pass path.</comment>
<file context>
@@ -0,0 +1,28 @@
+
+describe("startTwitterUserScraping", () => {
+ it("starts the profile-level user scraper for one handle", async () => {
+ const r = await startTwitterUserScraping("KETTAMA_");
+ expect(apifyClient.actor).toHaveBeenCalledWith("apidojo/twitter-user-scraper");
+ expect(start).toHaveBeenCalledWith(
</file context>
sweetmantech
force-pushed
the
feat/x-tweetless-profile-fallback
branch
from
1d8e76a to
27563d2
Compare
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
apidojo/twitter-scraper-lite carries profile stats on tweet items, so an X account with zero recent tweets never persists followers/bio — its run succeeds with an empty dataset and the connected socials row stays stale (recoupable/chat#1851, observed for a real artist handle). On an empty dataset the tweet handler now recovers the requested handle from the run INPUT and starts apidojo/twitter-user-scraper (V38PZzpEgOfeeWvZY); a new handler persists the profile, filtering out the related-user padding the actor appends (shape verified on a live run 2026-07-06). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sweetmantech
force-pushed
the
feat/x-tweetless-profile-fallback
branch
from
27563d2 to
3a63874
Compare
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Closing unmerged — deferred on a YAGNI call (sweetman, 2026-07-06): no current artist on the platform is in the live-but-tweetless population this covers (the motivating fixture turned out to be a deleted/renamed account, which no scraper can help). The core chat#1851 bug is fully fixed by api#755 + database#40. Revisit if a specific artist hits this: the branch ( |
1 participant
Summary
X profile stats ride on tweet items (
items[0].author), so an account with zero recent tweets succeeds on Apify with an empty dataset and persists nothing — the connectedsocialsrow stays stale forever. This adds a fallback: on an empty tweet dataset, recover the requested handle from the run INPUT and start a profile-level scrape (apidojo/twitter-user-scraper), whose results persist through the same webhook receiver.Tracking issue: recoupable/chat#1851 (optional PR #3 — independent of api#755 and database#40).
What changed
lib/apify/twitter/startTwitterUserScraping.ts(new) — startsapidojo/twitter-user-scraper(resolved actor idV38PZzpEgOfeeWvZY) with{twitterHandles: [handle], maxItems: 1}+ the shared webhooks.lib/apify/twitter/handleTwitterUserScraperResults.ts(new) — persists the profile; reads the run INPUT to recover the requested handle and filters out the related-user padding the actor appends (verified live: a 1-handle request returned the user plus 9 similar accounts).lib/apify/twitter/handleTwitterProfileScraperResults.ts— empty-dataset branch: read INPUT handle → start the fallback → return{ social: null, fallbackRunId }. No-ops (unchanged behavior) when the payload has no key-value store id or the INPUT has no handle.lib/apify/getApifyResultHandler.ts— registry entry forV38PZzpEgOfeeWvZY.lib/apify/validateApifyWebhookRequest.ts— schema now passes through the optionalresource.defaultKeyValueStoreId(present on Apify's default payload; previously stripped by zod).Verified live (2026-07-06, real Apify runs — not simulated)
GET /v2/acts/apidojo~twitter-user-scraper→V38PZzpEgOfeeWvZYtwitterHandles: ["ashnikko"]→ requested user (followers: 239289,url: https://x.com/ashnikko) plus 9 related users → hence the handle filtertwitterHandlesinput acceptedincludeUnavailableUsers: true— that account appears deleted/renamed, so for it specifically no scraper can help; the fallback covers accounts that exist but don't tweetTests (RED→GREEN)
New/updated suites failed before implementation (2 assertion failures + 2 module-not-found), pass after.
vitest run lib/apify lib/socials→ 128 tests passed.tsc --noEmit: 200 pre-existing errors on cleanorigin/test, 200 after — none new. eslint clean.Pending preview verification
End-to-end webhook chain (empty tweet run → fallback run → profile persisted to the connected row) needs a preview/test-deployment pass with a real tweetless-but-live handle; will post the results table here.
🤖 Generated with Claude Code
Summary by cubic
Adds a profile-level fallback for tweetless X accounts so follower and bio stats still update. If the tweet scraper returns no items, we start
apidojo/twitter-user-scraperfor the handle and persist the profile via a new handler.resource.defaultKeyValueStoreId) and startapidojo/twitter-user-scraperwithmaxItems: 1using shared webhooks.x.comprofile URL, and upserts only the requested handle’s profile tosocials.V38PZzpEgOfeeWvZYin the result-handler map.resource.defaultKeyValueStoreIdto read the run INPUT; no-op when missing or when the INPUT has no handle.Written for commit 3a63874. Summary will update on new commits.