fix: fence org lifecycle status and lease writes by Makisuo · Pull Request #2317 · MapleTechLabs/maple

Makisuo · 2026-11-06T18:04:14Z

Stacked on #2310. Closes the remaining status and lease races outside alerts.

ClickHouse schema apply: the claim writes a run id to workflow_instance_id and passes it to the workflow. Every progress write, the schema-version stamp and the failure path are fenced on that id. A reclaimed instance stops before its next DDL and ends as superseded without touching the org's settings. Long statements heartbeat every 5 minutes, so a live backfill is never reclaimed as stale.
PR review post-merge: settle is fenced on waiting plus the examination's lease, like reported already was, so a stale tick cannot overwrite a report.
Onboarding reward: reserve returns the reservation it took. finalize and release only clear that one. A holder whose lease lapsed still records the claim but leaves the next holder's lease alone.
VCS repository sync status: each write names the states it expects and requires the repo to be active. A continuation or a stale error can no longer regress a ready repo, and removed repos are not written.
Recommendation issues: status is branded with the literal schema (no migration; db:generate reports no change). Reconcile's status moves are guarded on the status it read. Concurrent inserts no longer collide. dismiss now stamps resolvedAt, like reconcile's resolve does.
Dependency scans: each scan of a repo first settles that repo's scans that have been running past the rescan window as failed.

Rollout note: a schema-apply instance already running when this deploys has no run id. It fails on its payload, and the org can re-apply once the 30-minute stale window passes.

Tests (PGlite):

apps/api  ClickHouseSchemaApplyWorkflow.run.test.ts            Tests  8 passed (8)
backend   PrReviewPostMergeService.test.ts                     Tests  5 passed (5)
backend   OnboardingService.test.ts + OnboardingChecklistService  Tests  23 passed (23)
backend   vcs __tests__/                                       Tests  150 passed (150)
backend   RecommendationIssueService.test.ts                   Tests  2 passed (2)
backend   dependency-scan/                                     Tests  52 passed (52)
db        parity.test.ts                                       Tests  5 passed (5)

tsrs --noEmit is clean for packages/db, packages/domain, packages/backend and apps/api. oxlint -c .oxlintrc.effect.json is clean on the touched files.

🤖 Generated with Claude Code

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

transitionRow runs a caller-built UPDATE guarded on the states the writer read and fails with TransitionConflictError when the row already moved.

Writers updated status columns by id alone, so one that read state earlier silently overwrote a concurrent transition. Every writer now guards its UPDATE on the state it decided from, through transitionRow. Error issues: applyTransition returns a 409 when it loses the race; the tick only regresses rows still done; alert and integration reopens land in regressed with the tick's bookkeeping. Investigations: late diagnoses on failed or resolved runs are dropped; manual status changes may only resolve a finished run or reopen it to its own verdict (MANUAL_INVESTIGATION_TRANSITIONS, 409 otherwise); restart refuses a running pass; one heartbeat-aware stale sweep; anomaly triage settles only once.

…iters The enqueue path's stale sweep now uses the shared heartbeat-aware sweep, markFailed is guarded on investigating, restart rescues a run whose start never claimed it, and the errors tick skips a wake or lease expiry that lost to a concurrent move (a renewed snooze included) instead of failing the org's tick. The investigation detail page disables resolve and restart on a run in flight, matching the server's rules.

…beat The lease clear is guarded on still being expired, and only an issue still in_progress drops to todo. The detail page lets an unclaimed run be run again, and a renewed snooze is covered by a test.

…tbeat long steps

… caller's reservation

…tes and an active repo

…ismiss stamps resolvedAt

…ew findings

coderabbitai · 2026-11-06T18:04:24Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 24 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: 8687ec7a-d7ec-4507-9782-9076fa76a33c

Commits

Reviewing files that changed from the base of the PR and between a86d988 and c4edac2.

Files selected for processing: 18

apps/api/src/workflows/ClickHouseSchemaApplyWorkflow.run.test.ts
apps/api/src/workflows/ClickHouseSchemaApplyWorkflow.run.ts
packages/backend/src/platform/bindings.ts
packages/backend/src/services/dependency-scan/DependencyScanStore.ts
packages/backend/src/services/dependency-scan/DependencyScanner.test.ts
packages/backend/src/services/dependency-scan/DependencyScanner.ts
packages/backend/src/services/errors/RecommendationIssueService.test.ts
packages/backend/src/services/errors/RecommendationIssueService.ts
packages/backend/src/services/integrations/vcs/VcsRepository.ts
packages/backend/src/services/integrations/vcs/VcsSyncService.ts
packages/backend/src/services/integrations/vcs/__tests__/vcs.test.ts
packages/backend/src/services/org/OnboardingChecklistService.ts
packages/backend/src/services/org/OnboardingService.test.ts
packages/backend/src/services/org/OnboardingService.ts
packages/backend/src/services/org/OrgClickHouseSettingsService.ts
packages/backend/src/services/pr-review/PrReviewPostMergeService.test.ts
packages/backend/src/services/pr-review/PrReviewPostMergeService.ts
packages/db/src/tables/org-recommendation-issues.ts

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

maple-review-bot · 2026-11-06T19:44:19Z

Maple review

🟢 Confidence 10/10 · safe to merge. Every fence and lease is pinned by a PGlite test at the state boundary it guards, and nothing outside those writes changed.
_{no findings · tests covered · risk high · 4/4 new units observable}

Each lifecycle write now states the state it expects: schema apply fences on its run id, post-merge settle on its lease, reward finalize on its reservation, repo sync status on an active repo in the prior state. All paths are pinned by PGlite tests. The open VcsQueueError/GithubAppError reports are untouched.

Schema apply progress, stamp and failure writes fence on the claimed workflow_instance_id, with a 5-minute heartbeat
updateRepoSyncStatus requires an active repo in one of the caller's expected sync states
finalizeRewardClaim/releaseRewardClaim clear only the reservation they took
Post-merge settle requires waiting plus the examination's lease

Before merge

Manual · Expect a schema apply in flight at deploy time to fail on its payload; the org re-applies after the 30-minute stale window

Production impact

Open errors in the changed files (3)

Issue	Service	Occurrences	File
@maple/backend/vcs/GithubAppError (4 issues)	maple-chat	75	`packages/backend/src/services/integrations/vcs/VcsRepository.ts`
@maple/http/errors/VcsProviderError (2 issues)	maple-chat	9	`packages/backend/src/services/integrations/vcs/VcsRepository.ts`
@maple/http/errors/VcsQueueError (2 issues)	maple-api	6	`packages/backend/src/services/integrations/vcs/VcsSyncService.ts`

After this merges, Maple checks whether they stop.

_{c4edac2 · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.}

maple-review-bot · 2026-11-06T19:56:40Z

Merged with 1 step from "Before merge" still open. If it is done, tick it on the review comment; if not, now is the time.

Manual · Expect a schema apply in flight at deploy time to fail on its payload; the org re-applies after the 30-minute stale window

maple-review-bot · 2026-11-06T22:30:20Z

Maple: after this shipped

⚠️ Something changed after this shipped. Details below.

Deployed to maple-api (pr-2320) at 2026-11-06 19:57 UTC as 2b42ce3, the first version after the merge (the merge commit itself never reported). Compared the 60 minutes before and after.

New errors since the deploy:

@maple/http/errors/AlertDeliveryRejectedError (maple-api, 6/h)
@maple/http/errors/AlertDeliveryRejectedError (maple-api, 2/h)

Errors in the changed files:

➖ @maple/backend/vcs/GithubAppError: 0/h → 0/h
➖ @maple/backend/vcs/GithubAppError: 0/h → 0/h
➖ @maple/http/errors/VcsProviderError: 0/h → 0/h
➖ @maple/backend/vcs/GithubAppError: 0/h → 0/h
➖ @maple/http/errors/VcsProviderError: 0/h → 0/h
➖ @maple/http/errors/VcsQueueError: 0/h → 0/h
➖ @maple/http/errors/VcsQueueError: 0/h → 0/h
➖ @maple/backend/vcs/GithubAppError: 0/h → 0/h

Operation	Calls/h	Errors	p95
`VcsRepository.updateRepoSyncStatus`	0 → 233	0.0% → 0.0%	0 → 76 ms
`PrReviewPostMergeService.examine`	85 → 89	0.0% → 0.0%	1336 → 659 ms

_{Maple looks once, an hour after the first deploy that carries the merge.}

Makisuo added 11 commits November 6, 2026 18:11

feat(backend): shared guarded status transition helper

079fc02

transitionRow runs a caller-built UPDATE guarded on the states the writer read and fails with TransitionConflictError when the row already moved.

fix(backend): lease expiry keeps a holder's concurrent move and heart…

cd7872c

…beat The lease clear is guarded on still being expired, and only an issue still in_progress drops to todo. The detail page lets an unclaimed run be run again, and a renewed snooze is covered by a test.

fix(api): fence schema-apply progress on the claiming run id and hear…

be0a9aa

…tbeat long steps

fix(backend): fence the post-merge settle on the examination's lease

b5ae7a4

fix(backend): fence the onboarding reward finalize and release on the…

df31aad

… caller's reservation

fix(backend): guard repository sync status writes on the expected sta…

16f441c

…tes and an active repo

fix(backend): guard recommendation reconcile on the status it read; d…

bb7b3c8

…ismiss stamps resolvedAt

fix(backend): settle dependency scans abandoned in running as failed

0b0242f

fix: fence the schema-version stamp, test the heartbeat, tighten revi…

f352be6

…ew findings

This comment has been minimized.

Sign in to view

Base automatically changed from fix/guarded-status-transitions to main November 6, 2026 19:42

Merge remote-tracking branch 'origin/main' into HEAD

c4edac2

Makisuo merged commit a132495 into main Nov 6, 2026
35 checks passed

Makisuo deleted the fix/org-lifecycle-races branch November 6, 2026 19:56

Navigation Menu

Repository navigation

fix: fence org lifecycle status and lease writes - #2317

fix: fence org lifecycle status and lease writes#2317
Makisuo merged 12 commits into
mainfrom
fix/org-lifecycle-races

Makisuo commented Nov 6, 2026 •

edited by blacksmith-sh Bot

Loading

Uh oh!

coderabbitai Bot commented Nov 6, 2026 •

edited

Loading

Review limit reached

Uh oh!

This comment has been minimized.

maple-review-bot Bot commented Nov 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

maple-review-bot Bot commented Nov 6, 2026

Uh oh!

maple-review-bot Bot commented Nov 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Makisuo commented Nov 6, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

This comment has been minimized.

maple-review-bot Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Maple review

Before merge

Production impact

Uh oh!

Uh oh!

maple-review-bot Bot commented Nov 6, 2026

Uh oh!

maple-review-bot Bot commented Nov 6, 2026

Maple: after this shipped

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Makisuo commented Nov 6, 2026 •

edited by blacksmith-sh Bot

Loading

coderabbitai Bot commented Nov 6, 2026 •

edited

Loading

maple-review-bot Bot commented Nov 6, 2026 •

edited

Loading