Skip to content

perf(backend): batch digest and push writes, read in parallel, share parsers, delete dead code - #2444

Merged
Makisuo merged 18 commits into
fix/audit-agent-runtimefrom
perf/audit-backend-services
Nov 10, 2026
Merged

Makisuo merged 18 commits into
fix/audit-agent-runtimefrom
perf/audit-backend-services

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #2443. Part of the audit stack that starts at #2434.

Efficiency

  • The daily membership sweep wrote one upsert per member, one org at a time. It now reads four orgs' member lists at once and writes multi-row upserts of 500. Rows are deduplicated by (org, user) first, the last entry winning as before: one statement cannot touch the same row twice, and offset paging can repeat a member.
  • Marking devices as pushed was one UPDATE per device id with identical values. It is one UPDATE now.
  • Independent reads ran one after another in the on-call schedule list, the anomaly alert source and the escalation step recipients. They now run concurrently. Two reported sites were left alone on purpose: one runs inside a transaction under the org advisory lock, and one is sequential by design to cap connections.

Behaviour change

  • Deleting an org now purges its fix-verification rows. A waiting row left behind was selected and skipped by the verification tick on every run.

Cleanup

  • getBearerToken had four identical copies in the auth layers and parseEncryptionKey three near-copies. Each lives in one module now.
  • Removed with no production user: the alert-templates/ directory (imported only by its own test), two unused GitHub client types and captureReusableContext. Its test now covers withoutInvocationContext, which had no other coverage.
  • Two comments still said each database call dials its own client. The invocation's pool is reused.

Verification

  • 17 test files in packages/backend (238 tests) and the mobile devices route test in apps/api pass.
  • The org-deletion test fails without the fix. The tests for the batched writes assert the resulting rows; they do not count statements and were not run against the old code.
  • packages/backend typecheck is clean. Lint and format are clean on the changed files.

Left for a decision

  • A failed warehouse query during digest generation skips that org's digest for the week, because the claim is stamped first. At-most-once looks intended, so it is unchanged.
  • ErrorIssuePullRequests and the other issue child tables are still not purged with an org.
  • docs/product-events-dashboard-parity.md still names alert-templates/ as the place for a new template.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

…fix verifications with the org

- DigestService: the daily membership sweep wrote one upsert per member.
  It now writes multi-row upserts of 500, deduped by (org, user) with the
  last entry winning as before, because Postgres rejects an upsert that
  touches a row twice in one statement. Clerk member lists are read four
  orgs at a time instead of one.
- MobileDevicesService.markPushed: one UPDATE for the whole id list
  instead of one per device.
- OnCallSchedulesService.list, AnomalyAlertSource.observe and
  loadStepRecipients: independent reads outside a transaction now run
  together on the invocation's pool instead of one after another.
- OrganizationService: error_issue_verifications is purged on org
  deletion, like the issue tables it points at. A leftover waiting row was
  selected by the verification tick on every run and skipped, taking a
  slot of the per-tick limit for good.
… dead code

- getBearerToken had four identical copies, one per auth layer. It lives
  in services/auth/bearer-token.ts now.
- parseEncryptionKey had three copies differing only in the error they
  build. platform/Crypto.ts exports parseIngestKeyEncryptionKey, which
  takes the error constructor.
- Deleted services/alerts/alert-templates (and its test): no production
  importer in any workspace, the dashboard and the MCP tool keep their own.
- Deleted the unreferenced GithubApiPullRequestFile and GithubReviewThread
  types, and captureReusableContext, which only its test called. That test
  now exercises withoutInvocationContext, the function production uses.
- Two comments said every execute dials its own Postgres client. The
  invocation's pool is reused, so they now say what batching saves.
@coderabbitai

coderabbitai Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0e131527-bf2f-4667-a2da-e801e100ed86
Commits

Reviewing files that changed from the base of the PR and between da8e7ed and 0d1d10a.

Files selected for processing: 27

  • packages/backend/src/platform/Crypto.ts
  • packages/backend/src/platform/invocation-context.test.ts
  • packages/backend/src/platform/invocation-context.ts
  • packages/backend/src/services/alerts/AnomalyDetectionService.ts
  • packages/backend/src/services/alerts/OnCallSchedulesService.ts
  • packages/backend/src/services/alerts/alert-escalation-runs.ts
  • packages/backend/src/services/alerts/alert-templates/index.ts
  • packages/backend/src/services/alerts/alert-templates/planetscale.test.ts
  • packages/backend/src/services/alerts/alert-templates/planetscale.ts
  • packages/backend/src/services/alerts/alert-templates/types.ts
  • packages/backend/src/services/alerts/anomaly/AnomalyAlertSource.ts
  • packages/backend/src/services/auth/ApiAuthorizationLive.ts
  • packages/backend/src/services/auth/ApiAuthorizationV2Live.ts
  • packages/backend/src/services/auth/SessionAuthorizationLive.ts
  • packages/backend/src/services/auth/UserSessionAuthorizationLive.ts
  • packages/backend/src/services/auth/bearer-token.ts
  • packages/backend/src/services/digest/DigestService.test.ts
  • packages/backend/src/services/digest/DigestService.ts
  • packages/backend/src/services/integrations/ScrapeTargetsService.ts
  • packages/backend/src/services/integrations/scrape-check-retention.ts
  • packages/backend/src/services/integrations/vcs/vendor/github/GithubAppClient.ts
  • packages/backend/src/services/org/OrgClickHouseSettingsService.ts
  • packages/backend/src/services/org/OrgIngestKeysService.ts
  • packages/backend/src/services/org/OrganizationService.delete.test.ts
  • packages/backend/src/services/org/OrganizationService.ts
  • packages/backend/src/services/push/MobileDevicesService.test.ts
  • packages/backend/src/services/push/MobileDevicesService.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@Makisuo
Makisuo added this pull request to stack #2450 November 10, 2026 03:23
@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@Makisuo

Makisuo commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

On the review's manual step (orgs deleted before this deploy keep their waiting verification rows, so a one-off purge is needed): no purge is needed once this stack is in. #2439, lower in the stack, makes the verification tick settle those rows itself. A deleted org's issues are already purged (ErrorIssues is in ORG_SCOPED_TABLES), so the tick's subject lookup for such a row returns nothing and the row is marked abandoned on the first tick that reaches it. The rows stay in the table as settled history; they just stop being selected. This PR's purge covers orgs deleted from now on.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

maple-review-bot Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. Batched writes and the org purge are pinned by new PGlite tests; the three concurrency edits only reorder independent reads. (The kickoff's "changed since then" list named files this diff does not con…
no findings · tests covered · risk high

Batches the daily membership sweep and the device push stamp into multi-row writes, runs three groups of independent reads concurrently, and purges fix-verification rows with the org. The new writes and the purge are exercised by tests; no correctness, scoping or data-loss defect found.

  • reconcileSubscriptions upserts 500 members at a time, deduped by (org, user)
  • markPushed stamps every device id in one IN (...) UPDATE
  • ErrorIssueVerifications moves into the org-delete purge
  • getBearerToken and parseIngestKeyEncryptionKey collapse to one copy each

0d1d10a · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit db71d63 into main Nov 10, 2026
38 checks passed
@Makisuo
Makisuo deleted the perf/audit-backend-services branch November 10, 2026 11:30
@maple-review-bot

Copy link
Copy Markdown

Maple: after this shipped

⚠️ Something changed after this shipped. Details below.

Deployed to maple-api (pr-2460) at 2026-11-10 11:32 UTC as 5b7e30f, the first version after the merge (the merge commit itself never reported). Compared the 60 minutes before and after.

New errors since the deploy:

  • @maple/http/errors/AlertDeliveryTargetMissingError (alerting, 4/h)
Operation Calls/h Errors p95
Clerk.organizations.getOrganizationMembershipList 0 → 0 0.0% → 0.0% 0 → 0 ms
Clerk.organizations.getOrganizationMembershipList 46 → 11 0.0% → 0.0% 130 → 276 ms
ScrapeCheckRetention.pruneForTargets 4 → 5 0.0% → 0.0% 1445 → 1476 ms

Maple looks once, an hour after the first deploy that carries the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant