Repository navigation
fix(productions): make overlapping deactivates idempotent instead of 500 - #405
Conversation
birme
left a comment
There was a problem hiding this comment.
Reviewed by the daily-backlog-pr housekeeping pass (Phase 3.5). Verified against main: deactivate is made idempotent via a per-productionId in-flight Promise map (cleared in finally) plus a writeDeactivatedDoc conflict-retry that re-reads and only re-applies when the stored doc still names no flow/intercom other than those this run tore down — correctly refusing to clobber a reactivation or a guest's new intercom grouping. Teardown, the #366 isStillRecording rev-guard, and RecordingDoc persistence are preserved verbatim in the extracted deactivate(). MAX_DB_WRITE_RETRIES=3 matches the "gives up" test; concurrency/conflict coverage is strong. No fabrication, no security impact.
Note: this rewrites the deactivate handler in src/routes/productions.ts, which PR #407 (the startedAt line) also edits — no code dependency, but whichever merges second will need to resolve a conflict. Branch is behind main after #406 merged; it will be refreshed server-side so a required check runs against current main, then merged on a subsequent pass.
|
This PR now conflicts with |
A deactivate reads the production doc, runs the teardown (the VOD upload alone can take tens of seconds when object storage is unreachable), then writes the doc with the revision it read at the start. A second deactivate sent during that window repeats the teardown against the same revision and its final write fails with a CouchDB 409, which the route returns as 500. - A deactivate that arrives while another one for the same production is running now waits for that run and returns its result. - A 409 on the final write re-reads the doc. If only unrelated fields changed (tally, name), the deactivation is applied on top and retried. If another stop path already left the production stopped with no flow, its state is returned as the result. Anything else, such as a new activation, still fails rather than being overwritten. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 409 retry in the deactivate write matched only status and Strom flow, so it could clear an intercom grouping it never tore down. Two cases: - A guest joins during the teardown and provisions a new grouping. The write now fails as a conflict, and the stored doc keeps the grouping for the next deactivate to tear down. - The idle watchdog stops the production first. The watchdog does not clear the grouping, which this run has already deleted. The write now keeps the watchdog's status and clears the stale refs, instead of returning the watchdog's doc unchanged. A retry happens only when the stored doc names no flow or grouping other than the ones this run tore down. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
50716e4 to
db760d6
Compare
birme
left a comment
There was a problem hiding this comment.
VERDICT: APPROVE
Reviewed at db760d6 (head of wagenet/fix-deactivate-conflict), diffed against main (95c00ee). Correctness-focused review; I did not author this.
Correctness — verified
- Root cause & fix match. The old handler read the doc, ran a teardown that can take tens of seconds (VOD upload with object storage down), then wrote with the stale
_rev, so overlapping deactivates 409'd and surfaced as 500. The fix adds (a) a per-processinFlightDeactivationsmap so a concurrent request joins the running teardown and shares its result, and (b) a 409-aware final write (writeDeactivatedDoc) that re-reads and re-applies. Both are sound. - In-flight dedup is leak-free and race-safe. The run promise is created eagerly, stored, and removed in a
finally. Every consumer (creator and joiners) awaitsruninside atry/catch, so a rejected run cannot produce an unhandled rejection, and a failed shared run correctly fails all waiters while the next request starts fresh (deactivatecalled twice in that test). Mirrors the existingactivationAbortControllersper-process pattern, as claimed. writeDeactivatedDocretry predicate is correct for each case I traced:- unrelated field changed (same status + same flow) → re-apply deactivation on the fresh rev;
- already
inactive/endedwith no flow (idle watchdog / other instance) → keep that writer's status/endedReasonand clear the stale refs this run already tore down; - a resource this run did not tear down (new activation with a different flow; a guest's new intercom grouping) →
tornDownis false, so it rethrows the 409 → 500, leaving the resource for the next deactivate. ThetornDownguard (stromFlowId/intercomProductionIdeither undefined or equal to what this run saw) correctly distinguishes these. - give-up bound honored:
attempt >= MAX_DB_WRITE_RETRIES - 1→ at most 3 inserts.
- #403 (
ended/endedReason: "deactivated") contract preserved.deactivatedDoc()reproducesstatus = stoppedStatus(doc.status)andendedReason = status === 'ended' ? 'deactivated' : undefinedexactly asmain, and when keeping another stop path's status it preserves that writer'sendedReason(e.g.idle). The teardown body (abort, state clears, relay force-stops, VOD upload + RecordingDoc persist, intercom teardown, guest sweep) is moved verbatim intodeactivate()with no behavioral change; response shape{id,name,status,_rev}is unchanged.
Merge-conflict question
No real conflict remains. src/routes/productions.ts is unchanged on main since the merge-base (bfcdefa), which already carries the #403 contract; git merge-tree produces no conflict markers. mergeStateStatus: BEHIND reflects only unrelated new commits on main. The overlap a prior comment flagged is resolved: the branch sits on top of the #403 change and preserves it. A rebase would be clean.
Fabrication / guesswork check
None found. All referenced symbols (stoppedStatus, MAX_DB_WRITE_RETRIES, EndedReason, teardownIntercomProduction, sweepGuestsOnProductionEnd, uploadRecordings, etc.) exist and type-check; EndedReason includes 'deactivated'/'idle'.
Security
No new surface. No auth/route changes, no secrets, best-effort teardown contracts and the guest-invite sweep (#325 security boundary) are retained unchanged.
Quality gates (local, Node 22 / pnpm 10)
pnpm install --frozen-lockfile: OKpnpm typecheck: PASSpnpm build: PASSnpx vitest run: PASS — 823/823 tests in 80 files (incl. the 8 newdeactivate-concurrencycases and the existingproduction-lifecycledeactivate contract)gh pr checks 405:build-and-testPASS
Clean fix, well-tested, idempotency and the #403 contract are consistent. Approving.
Opened by Claude (AI assistant) on behalf of @wagenet.
Problem
POST /productions/:id/deactivatereads the production doc, runs the teardown, then writes with the_revit read at the start. On a local rig with recording on and object storage unreachable, each request spent a long time in the VOD upload step, so three deactivates sent within about 25 s overlapped. The first succeeded. The other two hit a CouchDB 409 on the final write and returned500 Deactivation failed, although the production was already deactivated. Any other write to the doc during the teardown, such as a tally update, has the same effect.Change
Deactivate is now idempotent. A repeated request returns 200 with the production's resulting state and does not run the teardown again.
activationAbortControllers.MAX_DB_WRITE_RETRIES);inactive/endedwith no flow (idle watchdog, another instance): that status is kept, and the refs the other writer left behind (such as the intercom grouping this run deleted) are cleared;Most of the
productions.tsdiff is the handler body moving intodeactivate()with new indentation, so review with whitespace hidden (?w=1). If #403 lands, its status change goes indeactivatedDoc().Not in this PR: the upload still blocks the response
With storage down, deactivate still takes tens of seconds to respond. Moving the upload after the response needs two checks this PR does not make: that Strom keeps recordings after the flow is deleted, and a new signal for the #366
isStillRecordingguard, which compares_revand would always skip deletion. Left as a follow-up.Tests
src/__tests__/deactivate-concurrency.test.tscovers:Each case fails when its behaviour is reverted on its own. Against main's
productions.ts, 5 of the 8 tests fail. The three that pass ("runs again", the new activation, and the guest's grouping) are cases main already handles.pnpm typecheckpasses.pnpm vitest run: 773 tests pass in 74 files.🤖 Generated with Claude Code