Skip to content

[bridge-dbal] SingleResumeLockMiddleware deadlocks a worker against its own lock: every workflow step costs the 300s TTL #254

Description

@gabiudrescu

What happens

On the DBAL backend with Symfony Messenger, a workflow advances one step every 300 seconds - exactly the lock TTL - instead of advancing immediately. The worker process appears alive but idle, and outlives its own --time-limit.

Why

SingleResumeLockMiddleware::handle() acquires durable-resume-{executionId} on every pass of the bus, including the send pass:

$lock = $this->lockFactory->createLock('durable-resume-' . $executionId, $this->ttlSeconds);
$lock->acquire(true);   // blocking

There is no ReceivedStamp check. So:

  1. A worker receives ResumeWorkflowMessage → the middleware acquires the lock.
  2. The handler runs the execution and, still inside that stack, dispatches the next ResumeWorkflowMessage for the same execution.
  3. That dispatch goes through the same bus, hits the same middleware, and calls acquire(true) for the same key.

symfony/lock is not reentrant across Lock objects - createLock() returns a new Key - so the blocking acquire waits for the first lock to expire. It expires by TTL, never by release, because the release is in the finally of the outer pass that is still blocked. 300 seconds later the send proceeds and the cycle repeats on the next step.

Evidence

Journal of one execution, single worker consuming both queues:

10  ActivityCompleted        2026-09-02 08:28:54
13  WorkflowSignalReceived   2026-09-02 08:33:54    <- +300.0s, no work in between

lock_keys held durable-resume-order-000000020 with key_expiration exactly 300s after acquisition, and the worker (pid 78, --time-limit=45) was still alive six minutes later. Killing it released the queue instantly.

Reproduce

  • Sylius 2.2.1 / Symfony 7.4.2 / PHP 8.3, gplanchat/durable-bridge-dbal v0.1.0-alpha10.
  • event_store.type: dbal, activity_transport: messenger, Doctrine transports for ResumeWorkflowMessage and ActivityMessage, framework.lock on a DBAL store.
  • Start any workflow that calls one activity and then continues.
  • Run one worker: messenger:consume durable_workflows durable_activities.
  • Each step takes 300s.

Suggested fix

Only the receive pass needs the lock - it is what serialises two workers replaying the same execution. A send pass is not a replay.

if (null === $envelope->last(ReceivedStamp::class)) {
    return $stack->next()->handle($envelope, $stack);
}

Workaround in use

A decorator on durable.dbal.single_resume_lock that delegates to the bundle's middleware only when a ReceivedStamp is present. With it, the same run completes in 3 seconds instead of 15 minutes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions