⚡ perf: optimize webhook deduplication by eliminating N+1 query#67
⚡ perf: optimize webhook deduplication by eliminating N+1 query#67Sparkier wants to merge 1 commit into
Conversation
Pre-fetches existing posts in a single query based on the incoming payload instead of firing individual 'maybeSingle' queries for each iteration. Creates a Set to handle O(1) lookups during the deduplication phase. Also updates the Set progressively to catch duplicate entries in the same payload batch. Added performance benchmarks verifying the query load drop. Co-authored-by: Sparkier <5690524+Sparkier@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
1 participant
💡 What
Modified the Google Sheets webhook
POSThandler to perform a single batched read (.in('title', titles).in('contact', contacts)) instead ofNindividual.maybeSingle()queries. Uses an in-memorySetforO(1)deduplication logic across the entire payload batch, and tracks newly inserted posts live within the batch loop to mirror the original functionality perfectly.🎯 Why
The original code contained an N+1 query loop when reading duplicate-check data, where
Nis the number of rows inserted into the webhook payload. For larger payloads, this generated linear delays due to excessive round-trips to Supabase, slowing down processing dramatically.📊 Measured Improvement
I wrote a
perf.test.tsbenchmark targeting the/api/webhooks/google-sheetsendpoint to measure the difference with 2,000 synthetic rows of data.maybeSingle. After optimization, it drops to 1 large.in()query (an execution count reduction from 2,000 to 1). The test runtime executes almost identically in mock (~160ms) because it relies on mock overhead, but in a real-world scenario over network I/O, eliminating 1,999 database queries directly resolves severe performance degradation on large payload synchronization.PR created automatically by Jules for task 5224960296347097962 started by @Sparkier