Skip to content

workflow: recover stranded local runs on restart - #131

Draft
msullivan wants to merge 1 commit into
retryloopfrom
recover
Draft

workflow: recover stranded local runs on restart#131
msullivan wants to merge 1 commit into
retryloopfrom
recover

Conversation

@msullivan

Copy link
Copy Markdown
Contributor

A sleep persists a wait and schedules a delayed queue message to wake it. Under vercel dev that message lives in an in-memory queue, so restarting the dev server mid-run loses the wake-up and strands the run in running.

On dev worker startup, LocalWorld now re-enqueues every non-terminal run, retrying via RetryLoop until the dev queue is reachable; workflow_handler turns any elapsed wait into a wait_completed and continues. Replay is idempotent, so this should be fine?

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
vercel-py Ready Ready Preview Jun 16, 2026 12:39am

Request Review

A sleep persists a wait and schedules a delayed queue message to wake it.
Under `vercel dev` that message lives in an in-memory queue, so restarting
the dev server mid-run loses the wake-up and strands the run in `running`.

On dev worker startup, LocalWorld now re-enqueues every non-terminal
run, retrying via RetryLoop until the dev queue is reachable;
workflow_handler turns any elapsed wait into a wait_completed and
continues. Replay is idempotent, so this should be fine?
@fantix

fantix commented Jun 16, 2026

Copy link
Copy Markdown
Member

Yeah I think this is good; I'm working on implementing the local queue out of vercel dev but it would also be an in-memory queue, so this will be still needed.

fantix added a commit that referenced this pull request Jul 28, 2026
The port to vercel-queue routed workflow topics through
`vqs.sanitize_name()`, which doubles underscores, so `__wkf_workflow_`
became `____wkf__workflow__`. That is neither what the TypeScript SDK
publishes nor what the builder's queue trigger subscribes to. Subscribers
also registered under a derived consumer group, while VQS pushes arrive
with the `default` group the trigger declares, so nothing matched them.
Both worlds now share `get_physical_topic()` and `QUEUE_CONSUMER_GROUP`
in `world.py`, mirroring `@workflow/world-vercel` and
`createWorkflowQueueTrigger`.

The Vercel push callback hands the delivery to the world's own queue
client, so acks and lease renewals inherit the proxy URL and token
instead of falling back to an unconfigured default.

The local world now re-enqueues non-terminal runs the first time it
touches the queue. Its queue is in memory, so a sleeping run's wake-up
was lost whenever the dev server restarted. This supersedes #131, which
did the same thing before the queue moved in-process: back then recovery
had to run on a throwaway thread and retry until `vercel dev`'s queue
came up, and it predated queue namespaces.
fantix added a commit that referenced this pull request Jul 28, 2026
The port to vercel-queue routed workflow topics through
`vqs.sanitize_name()`, whose reversible encoding doubles underscores, so
`__wkf_workflow_` went out as `____wkf__workflow__` — outside the
`__wkf_*` prefix that the builder's trigger pattern and platform service
resolution key on, and out of step with what `@workflow/world-vercel`
publishes. It also left `consumer_group` unset on `VercelWorld`, so
subscribers registered under a name derived from the handler's qualname,
which moves whenever that function is renamed. Dispatch is keyed on
`(consumer_group, topic)`, so both matter. Both worlds now share
`get_physical_topic()` and `QUEUE_CONSUMER_GROUP` in `world.py`.

The Vercel push callback hands the delivery to the world's own queue
client, so acks and lease renewals inherit the proxy URL and token
instead of falling back to an unconfigured default.

The local world now re-enqueues non-terminal runs the first time it
touches the queue, and logs how many. Its queue is in memory, so a
sleeping run's wake-up was lost whenever the dev server restarted. This
supersedes #131, which did the same thing before the queue moved
in-process: back then recovery had to run on a throwaway thread and
retry until `vercel dev`'s queue came up, and it predated queue
namespaces.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants