feat(cyclotron): add V1 to V2 job relocation script#73016
Draft
dmarchuk wants to merge 2 commits into
Draft
Conversation
One-shot operator script to empty the legacy cyclotron V1 (postgres)
backend of a queue's parked jobs so V1 can be decommissioned without
waiting ~90 days for natural drain.
For the target queue (default hogflow) it relocates every `available`
job scheduled within the next year into V2 — preserving the original id
and scheduled time via the existing `cyclotronJobToInvocation` +
`queueInvocations({ overwriteExisting: true })` conversions — then
deletes those rows from V1 only after verifying each id is present in
V2. The corrupt far-future rows (a delay-overflow bug parked them so
they never come due) are deleted, never relocated.
Dry-run by default; `--apply` to execute. Exports
`cyclotronJobToInvocation` so the script reuses the exact
payload -> invocation conversion the drain worker uses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FL1Pj5nCDJvJwKiU59Biu
Refactors the relocation script so the orchestration is dependency- injected (`relocate(deps, args)`) and importable, guarded by `require.main === module`. Adds an end-to-end test that stands up both real cyclotron test databases (V1 legacy postgres + V2 cyclotron-node) and drives the full flow: - legit rows relocated to V2 with id, schedule, priority, parentRunId and decoded payload preserved (including a gzip-compressed vm_state), lookup columns (distinct_id/person_id/action_id) derived, and corrupt rows deleted; - dry-run writes and deletes nothing; - idempotent re-run after a full drain is a no-op; - a straggler already active in V2 (overwrite conflict) still drains from V1 via the verification gate; - the V2 write failing leaves V1 completely untouched (safety gate). Removes the runbook markdown. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014FL1Pj5nCDJvJwKiU59Biu
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cyclotron V1 (legacy PostgreSQL backend) needs to be decommissioned. To do so safely without waiting ~90 days for natural job drain, we need a one-shot operator script that relocates parked jobs from V1 to V2 while preserving job identity and scheduling, and cleans up corrupt rows that will never execute.
Changes
Added a new operator script
relocate-cyclotron-v1-jobs.tsthat:availablejob scheduled within the next year from V1 to V2, preserving the original job ID and scheduled timeoverwriteExisting: truewith original IDs; re-runs after partial failures complete the job; once drained, re-runs are no-ops--applyto actually write and delete; otherwise reports what would happenThe script includes:
cyclotronJobToInvocationconversion logic to ensure payload handling matches the drain workerExported
cyclotronJobToInvocationfromjob-queue-postgres.tsto allow the relocation script to use the same conversion logic as production workers.How did you test this code?
Added comprehensive end-to-end tests in
relocate-cyclotron-v1-jobs.test.tsthat exercise the script against real V1 and V2 test databases:All tests pass with the real job-queue producer and database pools.
https://claude.ai/code/session_014FL1Pj5nCDJvJwKiU59Biu