Skip to content

fix: eliminate AB-BA deadlock hazard in WebSocket connection registration #287

Open

Description

@ZhuchkaTriplesix

Problem Description

In issue #205, OxyRoute established a mandatory global lock hierarchy:

// Acquire the GIL *before* the write lock: `rebuild_snapshot` needs the GIL to clone
// `Py<PyAny>` handles, and taking a Rust lock first would let another thread hold the
// GIL while blocked on this same lock — classic AB-BA deadlock (issue #205).

Methods modifying app state (ensure_compiled_snapshot, connect_database, close_database) adhere to: GIL acquired first -> state.write() acquired second.

However, in src/dispatch.rs#L2086-L2097 inside run_rsgi_websocket:

    let ws_id = {
        let st = state.read(); // <-- 1. Rust read lock acquired FIRST
        let id = st
            .next_ws_id
            .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
        Python::with_gil(|py| { // <-- 2. GIL acquired SECOND (INVERTED!)
            st.active_websockets
                .lock()
                .insert(id, protocol.clone_ref(py));
        });
        id
    };

Because run_rsgi_websocket runs on a background Tokio worker thread without the GIL, acquiring state.read() before Python::with_gil causes a classic AB-BA deadlock:

  1. Thread 1 (Tokio Worker): Acquires state.read() during an incoming WebSocket connection.
  2. Thread 2 (Worker / Dispatch): Acquires the GIL and calls ensure_compiled_snapshot (or connect_database), attempting to acquire state.write(). Thread 2 blocks waiting for Thread 1's read lock to be released.
  3. Thread 1: Attempts to acquire the GIL via Python::with_gil. Thread 1 blocks because Thread 2 holds the GIL.
  4. Deadlock: Both threads are frozen permanently.

Proposed Solution

Wrap the ws_id block in Python::with_gil so the GIL is acquired before reading state:

    let ws_id = Python::with_gil(|py| {
        let st = state.read();
        let id = st
            .next_ws_id
            .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
        st.active_websockets
            .lock()
            .insert(id, protocol.clone_ref(py));
        id
    });

Alternatively, move next_ws_id and active_websockets outside of AppState (e.g. into top-level atomics/mutexes on App) so that state.read() is not required at all for WebSocket tracking.

Acceptance Criteria

  • WebSocket registration always acquires the GIL prior to taking any AppState lock.
  • Concurrency stress test under simultaneous route compilation / DB connect and WebSocket connections completes without deadlocking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions