diff --git a/README.md b/README.md
index 94286cc..c769c37 100644
--- a/README.md
+++ b/README.md
@@ -17,8 +17,8 @@ TeleFuser is a high-performance runtime for world model inference and multimodal
## News 📰
-- ✨ **2026-07-27**: Unified streaming on LiveKit with room sessions, worker admission, reconnect-friendly browser
- transport, and support for both server-push and bidirectional pipeline contracts.
+- ✨ **2026-07-27**: Unified streaming on LiveKit with room sessions, retained multi-session admission, LingBot
+ chunk-boundary time slicing, reconnect-friendly browser transport, and server-push/bidirectional contracts.
- ✨ **2026-07-22**: **NEW** Added [**LingBot-Video**](examples/lingbot_video/README.md) support for Dense and MoE T2I/T2V/TI2V generation, native four-GPU CFG/SP execution, and in-memory MoE refinement.
- ✨ **2026-07-15**: Added [**LingBot-World v2**](https://github.com/Robbyant/lingbot-world-v2) support for offline generation, interactive WebRTC streaming, and multi-GPU inference.
@@ -140,9 +140,14 @@ telefuser stream-serve examples/lingbot/lingbot_world_v2_image_to_video_h100.py
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey --livekit-api-secret secret \
--num-workers 1 --worker-gpu-map 0,1,2,3 \
+ --max-sessions-per-worker 2 --control-idle-timeout 10 \
--port 8088 --skip-validation
```
+This is one four-GPU model worker and one loaded LingBot service instance, not four replicas. It can retain two
+independent user sessions; the shared LingBot execution lease runs at most one session chunk at a time and yields at
+a chunk boundary after the active controller becomes idle while another session waits.
+
Terminal 4 — serve the browser controller and proxy its session API:
```bash
diff --git a/docs/en/index.md b/docs/en/index.md
index 451adfd..affd8a2 100644
--- a/docs/en/index.md
+++ b/docs/en/index.md
@@ -99,7 +99,7 @@ telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.p
Service GuideBatch serving, task APIs, and SDK.
-
Stream ServerLiveKit sessions, media, data topics, and bidirectional control.
+
Stream ServerLiveKit sessions, retained capacity, LingBot time slicing, and bidirectional control.
Stream SchedulerActor ownership, bounded dataflow, lifecycle, metrics, and GPU placement.
AIPerf BenchmarkBatch video and LingBot LiveKit workflows.
ConfigurationRuntime, attention, quantization, and offload settings.
diff --git a/docs/en/service.md b/docs/en/service.md
index a0acc97..a847478 100644
--- a/docs/en/service.md
+++ b/docs/en/service.md
@@ -45,15 +45,11 @@ telefuser serve \
--port 8000 \
--parallelism 1
-# LiveKit-backed world-model streaming
-# Set TF_MODEL_ZOO_PATH and PPL_CONFIG["parallelism"] in
-# examples/lingbot/lingbot_world_fast_image_to_video_h100.py
-telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
+# LiveKit-backed server-push streaming; start LiveKit Server separately
+telefuser stream-serve examples/stream_server/stream_video_replay.py \
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey \
--livekit-api-secret secret \
- --num-workers 1 \
- --worker-gpu-map 0,1,2,3 \
-p 8088 \
--skip-validation
```
@@ -93,9 +89,10 @@ Use for real-time world models, interactive generation, speech-driven animation,
- LiveKit server-push tracks for progressive video/audio output
- LiveKit bidirectional sessions for interactive control loops
- Stateful sessions with continuous chunk generation
-- Worker admission, controller/viewer roles, and reconnect handling
+- Worker admission, controller/viewer roles, and LiveKit transport reconnects
-See the [Stream Server Guide](stream_server.md) for full streaming documentation.
+See the [Stream Server Guide](stream_server.md) for the runtime topology, room roles, retained capacity, execution
+lease, GPU placement, lifecycle, and complete local stack.
---
@@ -307,6 +304,8 @@ The request-response service is intentionally local and single-process by defaul
- `telefuser stream-serve` exposes LiveKit session routes under `/v1/stream/*` and service routes under
`/v1/service/*`. Media and reliable control messages travel through the configured LiveKit deployment. It does
not expose task, file-download, OpenAI-compatible request-response, or direct SDP routes.
+- Stream runtime ownership, room topology, capacity, and replica boundaries are defined in the
+ [Stream Server Guide](stream_server.md); they are intentionally not shared with the request-response runtime.
### Artifact Storage and Cleanup
diff --git a/docs/en/stream_scheduler.md b/docs/en/stream_scheduler.md
index 054e310..28e732e 100644
--- a/docs/en/stream_scheduler.md
+++ b/docs/en/stream_scheduler.md
@@ -13,14 +13,13 @@ stage groups; the streaming scheduler owns bounded, per-session dataflow and per
The scheduler executes a directed acyclic graph of typed artifacts:
-```text
-external input
- |
- v
- encode -- condition --> denoise -- latent --> decode -- frames --> output
- ^
- |
- control
+```mermaid
+flowchart LR
+ I[External input] --> E[Encode actor]
+ E -->|condition| D[Denoise actor]
+ C[Control] --> D
+ D -->|latent| V[Decode actor]
+ V -->|frames| O[Output]
```
Each logical stage is represented by one long-lived actor. Independent actors may run concurrently even when their
@@ -45,6 +44,37 @@ Edges and outputs have explicit capacities. When a downstream stage cannot accep
backpressure rather than retaining unbounded tensors. Pipeline implementations must therefore treat submission as
admission-controlled, not as an unbounded queue.
+## Relationship to stream-service scheduling
+
+The [Stream Server Guide](stream_server.md) owns room, admission, and user-facing lifecycle semantics. This guide
+starts after a pipeline session has been admitted. Three schedulers operate at different boundaries and must not be
+treated as one queue:
+
+```mermaid
+flowchart TB
+ H[HTTP session request] --> A[Retained-session admission]
+ A -->|admitted pipeline session| L[LingBot execution lease]
+ L -->|one whole chunk| O[StreamingPipelineOrchestrator]
+ O --> E[Encode actor]
+ O --> D[Denoise actor]
+ O --> V[Decode actor]
+
+ Q1[HTTP admission FIFO] -. waits before .-> A
+ Q2[Execution-lease FIFO] -. waits before .-> L
+ Q3[Bounded artifact edges] -. backpressure inside .-> O
+```
+
+| Boundary | Owner | Purpose |
+| --- | --- | --- |
+| Retained-session admission | LiveKit runtime | Assign an HTTP session to capacity on a model worker, or place it in the bounded HTTP admission queue. |
+| Cross-session model execution | LingBot service instance | Grant one execution lease so only one retained LingBot session submits a whole chunk at a time. |
+| Intra-pipeline dataflow | `StreamingPipelineOrchestrator` | Schedule encode, denoise, and decode stage work with bounded artifacts and per-session ordering. |
+
+`max_sessions_per_worker` changes only the first boundary. It does not change service-instance count, execution
+leases, or graph-edge capacities. The second boundary is a LingBot service policy, not a generic orchestrator
+feature: its lease surrounds one session chunk, while the orchestrator may still overlap independent stages within
+that chunk. Other `BidirectionalService` implementations define their own cross-session policy.
+
## LingBot Condition Prefetch
LingBot condition encoding is independent of the corresponding control input. The session therefore keeps a fixed
@@ -65,7 +95,8 @@ session cleanup still runs through the owning actors.
## Actor Ownership and Session Lifecycle
-A state-owning worker has exactly one actor owner for its entire lifetime. In particular, one `ParallelWorker` must
+A state-owning stage worker has exactly one actor owner for its entire lifetime. This pipeline-level stage worker is
+not the stream-server model worker that owns retained-session capacity. In particular, one `ParallelWorker` must
not be invoked directly by a session facade or shared by multiple stage actors. This preserves result ordering and
ensures that cache mutation and release occur in one well-defined execution context.
diff --git a/docs/en/stream_server.md b/docs/en/stream_server.md
index bac6a8a..30886d9 100644
--- a/docs/en/stream_server.md
+++ b/docs/en/stream_server.md
@@ -1,155 +1,275 @@
# Stream Server
-TeleFuser uses LiveKit as its only streaming transport. The `telefuser stream-serve` command accepts pipeline files
-whose `get_service()` returns either `ServerPushService` or `BidirectionalService`; there is no separate backend
-selector or direct SDP endpoint.
+`telefuser stream-serve` exposes TeleFuser's LiveKit-backed streaming API. It accepts a pipeline file whose
+`get_service()` returns either `ServerPushService` or `BidirectionalService`.
+
+LiveKit owns signaling, WebRTC connections, SFU media delivery, and transport reconnects. TeleFuser owns HTTP
+admission, tokens, model workers, pipeline sessions, execution policy, and model-state cleanup. A LiveKit Cloud
+project or self-hosted LiveKit Server is required; TeleFuser does not expose a direct SDP endpoint.
+
+Use the three service guides at different boundaries:
+
+- [Service](service.md) compares `serve` and `stream-serve`.
+- This guide defines the LiveKit API, room roles, capacity, lifecycle, and deployment behavior.
+- [Streaming Pipeline Scheduler](stream_scheduler.md) defines actor ownership and bounded intra-pipeline dataflow.
+
+## Runtime topology
+
+```mermaid
+flowchart LR
+ C[Controller] -->|create / delete session| API[TeleFuser HTTP API]
+ V[Viewers] -->|request viewer tokens| API
+ C <-->|WebRTC| LK[LiveKit signaling + SFU]
+ V <-->|WebRTC| LK
+ API --> A[Registry + admission]
+ A --> W[One in-process model worker]
+ W <-->|one room runner per session| LK
+ W --> S[One shared service instance]
+ S --> P1[Pipeline session A]
+ S --> P2[Pipeline session B]
+```
+
+| Term | Meaning and ownership |
+|---|---|
+| Service process | One `telefuser stream-serve` process containing the HTTP API, registry, admission scheduler, and current in-process worker. |
+| Model worker | Loads the pipeline file once, owns one service instance, and accounts for retained-session capacity. |
+| Service instance | The single object returned by `get_service()`; model weights and its pipeline actor graph are loaded once. |
+| HTTP session | TeleFuser's public admission and lifecycle record. It maps one-to-one to a room name and, after admission, a room runner. |
+| Room runner | One task and one TeleFuser worker participant connected to a LiveKit room. Multiple runners share the service instance. |
+| Pipeline session | Per-user state returned by `BidirectionalService.create_session()`, such as control, noise, VAE, and model-cache state. |
+| Stage actor | An internal pipeline execution owner. It is not the model worker that owns retained-session capacity. |
-LiveKit terminates browser WebRTC connections and provides rooms, reconnect handling, media delivery, and reliable
-data messages. TeleFuser owns model workers, admission, session state, pipeline execution, and token issuance. A
-LiveKit Cloud project or a self-hosted LiveKit Server is therefore required.
+The current runtime supports exactly one `in-process` model worker and calls `get_service()` once. Multiple users do
+not load multiple model replicas. Additional replicas require separate `stream-serve` processes and external
+request routing; their registries, queues, health, and session state are independent.
-## Why LiveKit
+## Service contracts and capacity
-TeleFuser targets high-performance multimodal generation inference. Its streaming service must support continuous
-media output, bidirectional control, and long-running stateful model sessions. LiveKit's realtime transport
-capabilities align with these goals:
+| Contract | Input and output | Retained-session capacity |
+|---|---|---|
+| `ServerPushService` | Starts from request configuration and publishes progressive video/audio without room controls. | Exactly one; startup rejects `max_sessions_per_worker > 1`. |
+| `BidirectionalService` | Creates per-user state, accepts normalized controls, and yields output chunks. | May exceed one only when the implementation isolates state and defines safe cross-session execution. |
-- A room provides a stable transport boundary for a model session, keeping browser connections independent from
- session-owned model state.
-- Media tracks carry progressive video and audio, while data topics carry controls, status, and bounded telemetry.
-- Scoped tokens distinguish controller, viewer, and worker roles, matching TeleFuser's session ownership and permission
- model.
-- `ServerPushService` and `BidirectionalService` share one streaming entrypoint and client connection model.
-- LiveKit owns connections, reconnects, and media delivery so TeleFuser can focus on model workers, admission, pipeline
- execution, and resource release.
+`max_sessions_per_worker` is an admission limit, not a replica count, batch size, or graph-edge capacity. The
+checked-in LingBot-World-Fast and LingBot-World v2 services support multiple retained sessions and serialize their
+model chunks with a shared execution lease. Other bidirectional services must provide their own concurrency policy.
-## Install and start locally
+## Local development stack
-The LiveKit Python SDKs are included in the base TeleFuser installation:
+The LiveKit Python SDK is included in TeleFuser. Install the LiveKit Server and your platform's `coturn` package
+separately:
```bash
pip install -e .
-```
-Install the development LiveKit Server and your platform's `coturn` package separately:
-
-```bash
-# Debian/Ubuntu; use the equivalent coturn package on other platforms.
+# Debian/Ubuntu; use the equivalent package on other platforms.
sudo apt-get update
sudo apt-get install -y coturn
curl -sSL https://get.livekit.io | bash
-livekit-server --dev
```
-The development server listens on `ws://127.0.0.1:7880` and uses `devkey` / `secret`. Do not use development
-credentials in production.
-
-Start TeleFuser:
+The checked-in browser demo forces TCP TURN relay. Run the following development-only stack in four terminals:
```bash
-telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
- --livekit-url ws://127.0.0.1:7880 \
- --livekit-api-key devkey \
- --livekit-api-secret secret \
- --port 8088 \
- --skip-validation
-```
+# Terminal 1: TURN relay matching the browser configuration
+turnserver -n -m 1 \
+ --listening-ip=127.0.0.1 --relay-ip=127.0.0.1 \
+ --listening-port=3478 --min-port=49160 --max-port=49200 \
+ --user=livekit-demo:livekit-demo-password --realm=livekit.local \
+ --fingerprint --lt-cred-mech --no-tls --no-dtls --no-cli \
+ --allow-loopback-peers
-The same command serves server-push pipelines:
+# Terminal 2: signaling and SFU
+livekit-server --dev
-```bash
-telefuser stream-serve examples/stream_server/stream_video_replay.py \
+# Terminal 3: model, admission, and session API
+TF_MODEL_ZOO_PATH=/path/to/model_zoo \
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey \
--livekit-api-secret secret \
+ --worker-gpu-map 0,1,2,3 \
+ --max-sessions-per-worker 2 \
+ --control-idle-timeout 10 \
--port 8088 \
--skip-validation
-```
-
-Configuration may instead use `TELEFUSER_LIVEKIT_*` environment variables. Explicit CLI values take precedence.
-
-## Browser demo
-
-The checked-in page sets `iceTransportPolicy: relay`, so its matching TCP TURN service is required even though a
-production LiveKit deployment may provide TURN differently. Start this development-only coturn process first:
-
-```bash
-turnserver -n -m 1 \
- --listening-ip=127.0.0.1 \
- --relay-ip=127.0.0.1 \
- --listening-port=3478 \
- --min-port=49160 --max-port=49200 \
- --user=livekit-demo:livekit-demo-password \
- --realm=livekit.local \
- --fingerprint --lt-cred-mech \
- --no-tls --no-dtls --no-cli \
- --allow-loopback-peers
-```
-
-Start the LingBot control page in a fourth terminal:
-```bash
+# Terminal 4: browser page and HTTP API proxy
python examples/stream_server/livekit_bidirectional_demo.py \
--server-url http://127.0.0.1:8088 \
--port 8092 \
--no-open
```
-Open `http://127.0.0.1:8092`, select an initial image, and click **Start**. The demo proxies `/v1/stream/*` requests
-to TeleFuser, obtains a controller token, joins the assigned LiveKit room, renders the published video track, and
-sends the on-page or keyboard camera controls on `tf.control`.
+Open `http://127.0.0.1:8092`, select an image, and click **Start**. With VS Code Remote SSH, forward TCP ports
+`8092`, `7880`, and `3478` to the same local ports. The page proxies the session API, so `8088` does not need
+browser-side forwarding.
+
+The loopback TURN listener, static password, disabled TLS, `--allow-loopback-peers`, LiveKit development
+credentials, and `--skip-validation` are for a trusted development host only. Stop the browser session before
+stopping terminals 4 through 1.
+
+## Session creation and room joining
+
+TeleFuser assigns a unique room name and mints scoped tokens. It does not call the LiveKit room-management API to
+create the room; LiveKit materializes it when the first participant joins.
+
+```mermaid
+sequenceDiagram
+ participant C as Controller
+ participant API as TeleFuser API
+ participant A as Admission scheduler
+ participant W as Room runner
+ participant S as Shared service instance
+ participant LK as LiveKit
+
+ C->>API: POST /v1/stream/sessions
+ API->>A: reserve retained capacity
+ alt slot available
+ A-->>API: assigned
+ API-->>C: 200 session_id, room, controller token
+ W->>LK: join as worker
+ W->>S: create pipeline session or stream task
+ C->>LK: join with controller token
+ else HTTP queue has space
+ A-->>API: queued
+ API-->>C: 202 queue_position and token
+ A->>W: start after a slot is released
+ else no queue capacity
+ API-->>C: 429
+ end
+```
+
+A queued response already contains a room name and controller token, but no room runner publishes output until
+admission. Token lifetime limits when a token may be used to join; it is separate from TeleFuser session cleanup.
-For VS Code Remote SSH, forward the demo HTTP port, LiveKit signaling port, and the TURN listener configured for
-LiveKit. The checked-in demo uses TCP relay at `turn:127.0.0.1:3478?transport=tcp` with development credentials
-`livekit-demo` / `livekit-demo-password`. Change this browser configuration and the LiveKit deployment together for
-production.
+## One controller and multiple viewers
-Forward remote TCP ports `8092`, `7880`, and `3478` to the same local ports, then open
-`http://127.0.0.1:8092`. The loopback listener, static password, disabled TLS, and `--allow-loopback-peers` are only
-for a trusted development host reached through the tunnel. Do not copy them to a public deployment.
+```mermaid
+flowchart LR
+ C[Controller] -->|reliable tf.control| R[One LiveKit room]
+ W[TeleFuser worker] -->|one media publication + tf.status| R
+ R -->|tracks + room data| C
+ R -->|same tracks + room data| V1[Viewer 1]
+ R -->|same tracks + room data| VN[Viewer N]
+```
-The complete browser stack is now coturn (`3478`), LiveKit (`7880`), TeleFuser (`8088`), and the page (`8092`).
-`curl http://127.0.0.1:8088/v1/service/health` should report a ready idle worker before a session starts. During a
-successful run, the page shows the video track and status messages including `control_state`, generation stages,
-and `chunk_sent`. Stop or close the browser session before stopping the four processes in reverse order; this avoids
-the browser reconnecting while LiveKit and the model worker drain.
+| Role | LiveKit grants | TeleFuser semantics |
+|---|---|---|
+| Controller | Subscribe; publish data; no media-track publication | The session's configured controller identity. Only its `tf.control` messages are accepted. |
+| Viewer | Subscribe; publish neither data nor media tracks | Watches the same output and status without pipeline-control permission. |
+| Worker | Publish media and data; no subscription | Runs the session and publishes one output for LiveKit to fan out. |
+
+Create the HTTP session once, then call `POST /v1/stream/sessions/{session_id}/tokens` with a distinct identity for
+each viewer. A viewer joins the existing room and does not create another HTTP session, runner, or pipeline session.
+Viewers do not consume `max_sessions_per_worker`, enter a TeleFuser queue, acquire an execution lease, duplicate
+model state, or trigger inference. LiveKit/SFU delivery bandwidth and subscriber work still grow with viewer count.
+
+Viewer joins and departures do not change TeleFuser admission or session state. Controller departure is also not
+currently observed; clients must explicitly close the session when control ends.
+
+## Admission, queues, and LingBot execution
+
+```mermaid
+flowchart TD
+ N[New HTTP session] --> C{Retained slot available?}
+ C -->|yes| R[Start room runner]
+ C -->|no| Q{HTTP queue has space?}
+ Q -->|yes| H[HTTP 202, FIFO wait]
+ H -->|slot released| R
+ Q -->|no| X[HTTP 429]
+ R --> P[Retained pipeline session]
+ P --> L{LingBot valid control?}
+ L -->|yes| E[Execution-lease FIFO]
+ E --> G[One active session submits a chunk]
+ G --> B[Chunk boundary]
+ B -->|idle timeout + waiter| K[Park holder, grant next]
+ B -->|otherwise| G
+```
-## Architecture and lifecycle
+There are three independent scheduling boundaries:
-```text
-Browser ── HTTP /v1/stream/* ──> TeleFuser session API
- │ │
- └── LiveKit media/data ──> LiveKit room <── TeleFuser worker
- │
- └── stream pipeline actor graph
+| Boundary | Capacity owner | What waiting means |
+|---|---|---|
+| HTTP admission queue | LiveKit runtime | All retained slots are occupied. `queue_size` bounds this FIFO; zero disables it. |
+| LingBot execution-lease queue | Shared LingBot service instance | An admitted session wants model execution while another session holds the lease. |
+| Pipeline artifact queues | `StreamingPipelineOrchestrator` | A stage or downstream bounded edge cannot yet admit another sequence item. |
+
+The execution lease is LingBot-specific. A valid `control_state`, `control`, `prompt`, or `reset` records
+activity and queues a waiting or parked session. If another session is waiting and the holder has been control-idle
+for `control_idle_timeout`, the holder completes its in-flight chunk, parks, and hands off the lease. Handoff never
+interrupts a chunk.
+
+```mermaid
+stateDiagram-v2
+ [*] --> waiting
+ waiting --> queued: valid control
+ queued --> active: lease granted
+ active --> parked: waiter + idle timeout + chunk boundary
+ parked --> queued: new valid control
+ active --> closing: session cleanup
+ queued --> closing: session cleanup
+ parked --> closing: session cleanup
+ closing --> [*]
```
-1. The controller creates a session through `POST /v1/stream/sessions`.
-2. The scheduler admits, queues, or rejects it and assigns one worker.
-3. TeleFuser creates the LiveKit room and returns a scoped controller token.
-4. The worker joins the room and starts the pipeline.
-5. Video and PCM16 audio are published as LiveKit tracks. Status and metrics use reliable data topics.
-6. For `BidirectionalService`, only the controller can send normalized control messages to the pipeline.
-7. Deletion, timeout, controller departure, or pipeline completion drains actor-owned state and closes the room.
+Parking does not close the session, release its retained slot, or free its cache. Set
+`max_sessions_per_worker` from measured per-session memory headroom. Controllers representing held input must resend
+`control_state`; the checked-in browser sends it once per second while a key remains held. Releasing the execution
+lease never moves a session back to the HTTP queue.
+
+## Session lifecycle and current limits
+
+```mermaid
+stateDiagram-v2
+ [*] --> pending: POST session
+ pending --> assigned: slot available
+ pending --> queued: wait for slot
+ pending --> [*]: rejected
+ queued --> assigned: slot released
+ assigned --> joining_room
+ joining_room --> starting_pipeline
+ starting_pipeline --> running
+ queued --> draining: DELETE
+ assigned --> draining: DELETE
+ joining_room --> draining: DELETE
+ starting_pipeline --> draining: DELETE
+ running --> draining: DELETE
+ draining --> closed: cleanup complete
+ running --> closed: stop or normal completion
+ joining_room --> failed: runner error
+ starting_pipeline --> failed: pipeline error
+ running --> failed: runner or pipeline error
+```
+
+Cleanup stops new work, closes the pipeline session through its owner, disconnects the worker participant, releases
+the retained slot, and admits the next HTTP-queued session. TeleFuser does not explicitly delete the LiveKit room;
+remaining browser participants and the LiveKit deployment determine the transport room's later lifetime.
-Each streaming stage worker remains owned by one pipeline actor. Reconnects do not move actor-owned cache state
-between workers. A server-push pipeline starts from its request config and produces chunks without incoming controls;
-a bidirectional pipeline additionally exposes create, pull, control, and close operations.
+The current runtime has these deliberate documentation-visible limitations:
+
+- `session_timeout` records `expires_at`, but no background task currently changes the session to `expired`.
+- `controller_timeout` and `room_empty_timeout` are accepted configuration values but are not enforced.
+- Participant events are not monitored, so `participant_count` remains `0` and departure does not trigger cleanup.
+- Terminal records remain in the in-memory registry for the lifetime of the process. They are not shared or restored.
+- A controller should send `stop` or call DELETE. Closing a browser tab alone does not release capacity.
## HTTP API
| Endpoint | Method | Purpose |
|---|---|---|
| `/v1/stream/sessions` | POST | Create and admit a controller session |
-| `/v1/stream/sessions/{session_id}` | GET | Read session status |
-| `/v1/stream/sessions/{session_id}` | DELETE | Drain and close a session |
-| `/v1/stream/sessions/{session_id}/tokens` | POST | Create a viewer token |
-| `/v1/stream/health` | GET | LiveKit scheduler and worker health |
-| `/v1/service/health` | GET | Generic service health |
+| `/v1/stream/sessions/{session_id}` | GET | Read the in-memory session record |
+| `/v1/stream/sessions/{session_id}` | DELETE | Drain, close, and release the session |
+| `/v1/stream/sessions/{session_id}/tokens` | POST | Mint a subscribe-only viewer token |
+| `/v1/stream/health` | GET | Read scheduler and aggregate worker health |
+| `/v1/service/health` | GET | Read generic service health |
| `/v1/service/ready` | GET | Readiness probe |
-| `/v1/service/metadata` | GET | Pipeline and transport metadata |
-| `/v1/service/metrics` | GET | Prometheus metrics |
+| `/v1/service/metadata` | GET | Runtime topology and service metadata |
+| `/v1/service/metrics` | GET | Prometheus text metrics |
+| `/v1/service/metrics/json` | GET | JSON service and LiveKit health metrics |
Create a controller session:
@@ -164,27 +284,7 @@ curl -X POST http://127.0.0.1:8088/v1/stream/sessions \
}'
```
-For a one-minute LingBot-World v2 replay, start
-`examples/lingbot/lingbot_world_v2_image_to_video_h100.py` and use:
-
-```json
-{
- "fps": 16,
- "chunk_size": 4,
- "frame_num": 957,
- "max_duration_seconds": 60.0
-}
-```
-
-The complete-chunk policy maps this request to 60 chunks and 59.75 seconds of output media. The v2 example uses
-`local_attn_size=18` and `sink_size=6`; its KV capacity therefore remains fixed while the session-owned noise and
-VAE state advance incrementally. The reproducible LiveKit workload and dated four-H100 result are documented in
-[TeleFuser and AIPerf](benchmark_aiperf.md).
-
-A successful response includes `session_id`, `room`, `livekit_url`, `token`, `worker_id`, and `status`. A queued
-session returns HTTP 202 with `queue_position`; a full zero-length queue returns HTTP 429.
-
-Create a viewer token without granting control permission:
+Create a viewer token for the same room:
```bash
curl -X POST http://127.0.0.1:8088/v1/stream/sessions/
/tokens \
@@ -192,22 +292,26 @@ curl -X POST http://127.0.0.1:8088/v1/stream/sessions//tokens \
-d '{"identity":"viewer-1"}'
```
-Close the session explicitly:
+Close and release the session:
```bash
curl -X DELETE http://127.0.0.1:8088/v1/stream/sessions/
```
+A direct admission returns HTTP 200. A bounded wait returns HTTP 202 with `queue_position`; a disabled or full queue
+returns HTTP 429. The one-minute LingBot-World v2 workload and observed four-H100 results are documented in
+[TeleFuser and AIPerf](benchmark_aiperf.md).
+
## LiveKit data protocol
-| Topic | Direction | Content |
-|---|---|---|
-| `tf.control` | controller to worker | Reliable JSON control messages |
-| `tf.status` | worker to room | Lifecycle and chunk status |
-| `tf.metrics` | worker to room | Bounded runtime metrics |
-| `tf.asset` | reserved | Future bounded asset messages |
+| Topic | Direction | Delivery | Current use |
+|---|---|---|---|
+| `tf.control` | Controller to worker | Reliable in the checked-in clients | `control_state`, `control`, `prompt`, `reset`, and `stop` |
+| `tf.status` | Worker to room | Reliable | Runner lifecycle, errors, chunk metadata, and completion |
+| `tf.metrics` | Worker to room | Lossy | Supported by the room client, but not emitted by the generic runner today |
+| `tf.asset` | Reserved | Not defined | Future bounded asset messages |
-Accepted control types are `control_state`, `control`, `prompt`, `reset`, and `stop`. For example:
+Example control:
```json
{"type":"control_state","controls":["w","j"]}
@@ -219,56 +323,84 @@ An optional versioned envelope is also accepted:
{"version":1,"session_id":"","type":"control_state","payload":{"controls":["w"]}}
```
-Messages are bounded by `TELEFUSER_LIVEKIT_MAX_DATA_MESSAGE_BYTES` (12 KiB by default). Unknown controls, duplicate
-entries, invalid JSON, wrong topics, session mismatches, and control messages from viewers are rejected.
+Inbound messages are bounded by `max_data_message_bytes` (12 KiB by default). Wrong topics, non-controller senders,
+invalid JSON, unknown controls, duplicates, and session mismatches are rejected.
-## CLI and environment configuration
+## CLI, environment, and GPU placement
```text
telefuser stream-serve PIPE_PATH [OPTIONS]
```
-Important options are `--host`, `--port`, `--livekit-url`, `--livekit-api-key`, `--livekit-api-secret`,
-`--num-workers`, `--worker-gpu-map`, `--queue-size`, `--session-timeout`, `--token-ttl`,
-`--controller-timeout`, `--room-empty-timeout`, and `--worker-mode`.
+Use `telefuser stream-serve --help` for the complete option list. The options with important runtime semantics are:
-| Environment variable | Default | Meaning |
+| Option | Default | Semantics |
|---|---:|---|
-| `TELEFUSER_LIVEKIT_URL` | required | LiveKit WebSocket URL |
-| `TELEFUSER_LIVEKIT_API_KEY` | required | API key used to mint room tokens |
-| `TELEFUSER_LIVEKIT_API_SECRET` | required | API secret used to mint room tokens |
-| `TELEFUSER_LIVEKIT_HOST` | `0.0.0.0` | HTTP API bind host |
-| `TELEFUSER_LIVEKIT_PORT` | `8088` | HTTP API port |
-| `TELEFUSER_LIVEKIT_NUM_WORKERS` | `1` | Model workers |
-| `TELEFUSER_LIVEKIT_WORKER_GPU_MAP` | unset | Semicolon-separated GPU groups, such as `0,1;2,3` |
-| `TELEFUSER_LIVEKIT_QUEUE_SIZE` | `0` | Queued sessions; zero rejects when busy |
-| `TELEFUSER_LIVEKIT_SESSION_TIMEOUT` | `1800` | Maximum session lifetime in seconds |
-| `TELEFUSER_LIVEKIT_TOKEN_TTL` | `3600` | Join-token lifetime in seconds |
-| `TELEFUSER_LIVEKIT_CONTROLLER_TIMEOUT` | `60` | Grace period after controller departure |
-| `TELEFUSER_LIVEKIT_ROOM_EMPTY_TIMEOUT` | `30` | Grace period after the room becomes empty |
-
-The current runtime supports one in-process worker. Use separate service processes for additional workers until
-process-worker isolation is implemented. `--skip-validation` is intended for trusted local files, not production.
-
-## Production deployment
+| `--host`, `--port` | `0.0.0.0`, `8088` | HTTP bind address |
+| `--num-workers` | `1` | Must remain `1` in the current runtime |
+| `--worker-gpu-map` | unset | One logical GPU group for the current worker, for example `0,1,2,3` |
+| `--max-sessions-per-worker` | `1` | Retained bidirectional sessions; not replicas |
+| `--queue-size` | `0` | HTTP admission FIFO length; zero rejects at capacity |
+| `--control-idle-timeout` | `10` | LingBot lease idle threshold when another session waits |
+| `--session-timeout` | `1800` | Records `expires_at`; not currently enforced |
+| `--token-ttl` | `3600` | Join-token lifetime |
+| `--controller-timeout` | `60` | Reserved; not currently enforced |
+| `--room-empty-timeout` | `30` | Reserved; not currently enforced |
+| `--worker-mode` | `in-process` | `process` is accepted by the CLI but not implemented by the runtime |
+
+The CLI can fall back to `TELEFUSER_LIVEKIT_URL`, `TELEFUSER_LIVEKIT_API_KEY`,
+`TELEFUSER_LIVEKIT_API_SECRET`, `TELEFUSER_LIVEKIT_WORKER_GPU_MAP`,
+`TELEFUSER_LIVEKIT_MAX_SESSIONS_PER_WORKER`, and `TELEFUSER_LIVEKIT_CONTROL_IDLE_TIMEOUT` when their matching CLI
+value is unset. Environment-only settings include `TELEFUSER_LIVEKIT_DEFAULT_FPS` (default `16`),
+`TELEFUSER_LIVEKIT_MAX_DATA_MESSAGE_BYTES` (default `12288`), and
+`TELEFUSER_LIVEKIT_CORS_ALLOW_ORIGINS` (default `["*"]`).
+
+Other Click options currently pass their displayed defaults explicitly, so use the CLI option rather than a
+same-named environment variable for those fields.
+
+In the current in-process runtime, `worker_gpu_map` records scheduler topology and its group size becomes the
+`gpu_num` passed to `get_service()`. It does not set `CUDA_VISIBLE_DEVICES`, isolate devices, or rewrite
+`ModelRuntimeConfig`. Select physical GPUs with `CUDA_VISIBLE_DEVICES` and ensure that the pipeline uses the
+corresponding process-local device indices. For example:
+
+```bash
+CUDA_VISIBLE_DEVICES=4,5,6,7 \
+telefuser stream-serve PIPE_PATH --worker-gpu-map 0,1,2,3
+```
+
+This exposes physical GPUs 4-7 as local devices 0-3 and passes `gpu_num=4`; it still loads one service instance.
+
+## Observability
+
+| Signal | Exact interpretation |
+|---|---|
+| `workers_busy` | Model workers retaining at least one session; with the current runtime this is `0` or `1`. |
+| `workers_idle` | Non-failed model workers retaining no sessions. |
+| `workers_failed` | Workers whose aggregate state is failed. |
+| `queued_sessions` | HTTP admission queue depth only; it excludes LingBot lease and pipeline artifact waits. |
+| `livekit_connected` | Derived from aggregate worker status being `starting_pipeline`, `running`, or `draining`; it is not a direct LiveKit server probe. |
+| `participant_count` | Currently always `0` because participant events are not wired into the registry. |
+| `lease_queued`, `lease_granted`, `lease_parked` | LingBot execution-lease transitions published through `tf.status`. |
+
+`livekit_connected=false` is expected before any room runner reaches pipeline startup and does not mean model loading
+failed. For pipeline performance, keep target compute metrics distinct from client delivery metrics; see
+[Metrics](metrics.md) and [TeleFuser and AIPerf](benchmark_aiperf.md).
+
+## Production and troubleshooting
- Use LiveKit Cloud or the official self-hosted deployment guidance; do not expose `livekit-server --dev`.
-- Use unique API credentials and keep the API secret only on the TeleFuser server.
-- Configure TLS, advertised node addresses, UDP/TCP media ports, and TURN in LiveKit itself.
-- Restrict TeleFuser's HTTP API with the deployment's authentication and network policy.
-- Monitor `/v1/service/ready`, worker failures, queue depth, and session expiration.
-- Interpret chunk period as adjacent output cadence: p95 cadence must remain below one chunk's media duration with
- margin for transport and encoding. Pipeline residence and client delivery FPS are separate measurements.
-
-## Troubleshooting
-
-- **HTTP health is ready but no media arrives:** verify that both browser and worker can reach the LiveKit URL and
- inspect LiveKit participant/track logs.
-- **Browser reconnects repeatedly:** verify signaling, TURN, firewall, and advertised LiveKit node addresses.
-- **Controls are ignored:** ensure the sender used the controller token, topic `tf.control`, and a supported control.
-- **HTTP 429:** all workers are busy and `queue_size` is zero, or the queue is full.
-- **Session remains active:** call the session DELETE endpoint and check controller/room timeout settings.
-- **Local LiveKit connection returns proxy HTTP 503:** some native SDK paths honor `HTTP_PROXY` but not
- `NO_PROXY`. Start TeleFuser with `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and their lowercase variants unset when
- connecting to `ws://127.0.0.1:7880`.
-- **Stale GPU workers after a forced exit:** terminate remaining `spawn_main` processes before restarting.
+- Keep the LiveKit API secret on the TeleFuser server. Add deployment-layer authentication around the HTTP API.
+- Configure TLS, advertised node addresses, UDP/TCP media ports, and TURN in LiveKit.
+- Size retained-session capacity from GPU memory and LiveKit viewer capacity from SFU bandwidth.
+- Monitor readiness, worker failure, HTTP queue depth, pipeline cadence, and explicit session cleanup.
+
+Common failures:
+
+- **Ready but no media:** verify that the worker and browser can reach LiveKit and inspect participant/track logs.
+- **Repeated browser reconnects:** check signaling, TURN credentials, firewall, and advertised LiveKit addresses.
+- **Controls ignored:** use the controller token, `tf.control`, a supported type, and the configured identity.
+- **HTTP 429:** retained slots and the configured HTTP queue are full, or the queue is disabled.
+- **Session remains after clients leave:** departure cleanup is not implemented; send `stop` or call DELETE.
+- **Local LiveKit returns proxy HTTP 503:** unset upper- and lowercase `HTTP_PROXY`, `HTTPS_PROXY`, and `ALL_PROXY`
+ for a local `ws://127.0.0.1:7880` deployment; some native SDK paths do not apply `NO_PROXY`.
+- **Workers remain after forced exit:** terminate stale `spawn_main` children before restarting.
diff --git a/docs/zh/index.md b/docs/zh/index.md
index 8a6c6db..a224e06 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -98,7 +98,7 @@ telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.p
服务指南批量服务、任务 API 和 SDK。
-
流式服务LiveKit session、媒体、data topic 和双向控制。
+
流式服务LiveKit session、常驻容量、LingBot 时分复用和双向控制。
流式调度器Actor 所有权、有界数据流、生命周期、指标和 GPU 卡位。
AIPerf 基准测试Batch 视频与 LingBot LiveKit 测试流程。
配置运行时、注意力、量化和卸载配置。
diff --git a/docs/zh/service.md b/docs/zh/service.md
index 3246677..55b1b05 100644
--- a/docs/zh/service.md
+++ b/docs/zh/service.md
@@ -45,15 +45,11 @@ telefuser serve \
--port 8000 \
--parallelism 1
-# LiveKit-backed 世界模型流服务
-# 在 examples/lingbot/lingbot_world_fast_image_to_video_h100.py 中设置
-# TF_MODEL_ZOO_PATH 和 PPL_CONFIG["parallelism"]
-telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
+# LiveKit-backed server-push 流服务;LiveKit Server 需要单独启动
+telefuser stream-serve examples/stream_server/stream_video_replay.py \
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey \
--livekit-api-secret secret \
- --num-workers 1 \
- --worker-gpu-map 0,1,2,3 \
-p 8088 \
--skip-validation
```
@@ -93,9 +89,10 @@ TeleFuser 提供两种服务命令,针对不同工作负载类型优化:
- LiveKit server-push track:渐进式视频/音频输出
- LiveKit bidirectional session:交互式控制循环
- 有状态会话,连续 chunk 生成
-- Worker 准入、controller/viewer 角色和重连处理
+- Worker 准入、controller/viewer 角色和 LiveKit 传输层重连
-完整流式文档请参阅[流式服务指南](stream_server.md)。
+Runtime 拓扑、room 角色、常驻容量、execution lease、GPU 放置、生命周期和完整本地链路见
+[流式服务指南](stream_server.md)。
---
@@ -306,6 +303,8 @@ TELEFUSER_RATE_LIMIT_REQUESTS_PER_MINUTE=100
- `telefuser serve` 只暴露请求-响应路由:`/v1/tasks/*`、`/v1/files/*`、`/v1/images/*`、`/v1/videos/*` 和 `/v1/service/*`。
- `telefuser stream-serve` 暴露 `/v1/stream/*` 下的 LiveKit session 路由和 `/v1/service/*`。媒体与可靠
control 消息经配置的 LiveKit 部署传输;它不暴露 task、file-download、OpenAI 兼容请求-响应或直接 SDP 路由。
+- 流式 runtime 的所有权、room 拓扑、容量和副本边界由[流式服务指南](stream_server.md)定义,不与请求-响应
+ runtime 共享。
### Artifact 存储与清理
diff --git a/docs/zh/stream_scheduler.md b/docs/zh/stream_scheduler.md
index c88362e..e52d59b 100644
--- a/docs/zh/stream_scheduler.md
+++ b/docs/zh/stream_scheduler.md
@@ -12,14 +12,13 @@ per-session 数据流和长期存活的 stage actor。
调度器执行由带类型 artifact 构成的有向无环图:
-```text
-外部输入
- |
- v
-encode -- condition --> denoise -- latent --> decode -- frames --> 输出
- ^
- |
- control
+```mermaid
+flowchart LR
+ I[外部输入] --> E[Encode actor]
+ E -->|condition| D[Denoise actor]
+ C[Control] --> D
+ D -->|latent| V[Decode actor]
+ V -->|frames| O[输出]
```
每个逻辑 stage 对应一个长期存活的 actor。相互独立的 actor 可以并发执行,即使其 worker 使用同一张物理 GPU。
@@ -42,6 +41,35 @@ CUDA device placement 本身不代表串行执行或资源所有权。
edge 和输出均有显式容量。下游 stage 无法继续接收任务时,调度器施加 backpressure,而不是无限保留 tensor。
因此管线实现必须把提交视为受准入控制的操作,而不是无界队列。
+## 与流服务调度的关系
+
+[流式服务指南](stream_server.md)负责 room、准入和面向用户的生命周期语义;本文从 pipeline session 已准入
+之后开始。系统中有三个边界不同的 scheduler,不能把它们视为同一条队列:
+
+```mermaid
+flowchart TB
+ H[HTTP session 请求] --> A[常驻 session 准入]
+ A -->|已准入 pipeline session| L[LingBot execution lease]
+ L -->|一个完整 chunk| O[StreamingPipelineOrchestrator]
+ O --> E[Encode actor]
+ O --> D[Denoise actor]
+ O --> V[Decode actor]
+
+ Q1[HTTP 准入 FIFO] -. 在此之前等待 .-> A
+ Q2[Execution-lease FIFO] -. 在此之前等待 .-> L
+ Q3[有界 artifact edge] -. Pipeline 内 backpressure .-> O
+```
+
+| 边界 | 所有者 | 用途 |
+| --- | --- | --- |
+| 常驻 session 准入 | LiveKit runtime | 把 HTTP session 分配到模型 worker 的容量,或放入有界 HTTP 准入队列。 |
+| 跨 session 模型执行 | LingBot 服务实例 | 授予唯一 execution lease,使常驻 LingBot session 每次只提交一个完整 chunk。 |
+| Pipeline 内数据流 | `StreamingPipelineOrchestrator` | 以有界 artifact 和 per-session 顺序调度 encode、denoise、decode stage。 |
+
+`max_sessions_per_worker` 只改变第一层边界,不改变服务实例数、execution lease 或 graph edge capacity。
+第二层是 LingBot 服务策略,不是通用 orchestrator 能力:lease 包围一个 session chunk,而 orchestrator 仍可让
+该 chunk 内的独立 stage 相互重叠。其他 `BidirectionalService` 实现需要自行定义跨 session 策略。
+
## LingBot Condition 预取
LingBot 的 condition encode 不依赖对应 control,因此 session 使用固定深度为 2 的 lookahead,让 VAE encode
@@ -61,9 +89,9 @@ actor 执行。
## Actor 所有权与 Session 生命周期
-一个有状态 worker 在整个生命周期内只能有一个 actor owner。特别是,一个 `ParallelWorker` 不得由 session
-facade 直接调用,也不得被多个 stage actor 共享。该约束保证 result ordering,并让 cache 更新与释放发生在
-唯一、明确的执行上下文中。
+一个有状态 stage worker 在整个生命周期内只能有一个 actor owner。这里的 pipeline stage worker 不是拥有常驻
+session 容量的 stream-server 模型 worker。特别是,一个 `ParallelWorker` 不得由 session facade 直接调用,也
+不得被多个 stage actor 共享。该约束保证 result ordering,并让 cache 更新与释放发生在唯一、明确的执行上下文中。
session 关闭按以下顺序执行:
diff --git a/docs/zh/stream_server.md b/docs/zh/stream_server.md
index 16ed9ca..92a374d 100644
--- a/docs/zh/stream_server.md
+++ b/docs/zh/stream_server.md
@@ -1,144 +1,271 @@
# 流式服务
-TeleFuser 仅使用 LiveKit 作为流式传输后端。`telefuser stream-serve` 接受 `get_service()` 返回
-`ServerPushService` 或 `BidirectionalService` 的 pipeline 文件;不再提供 backend 选择器或直接 SDP 接口。
+`telefuser stream-serve` 提供基于 LiveKit 的 TeleFuser 流式 API。它接收一个 pipeline 文件,其中
+`get_service()` 必须返回 `ServerPushService` 或 `BidirectionalService`。
+
+LiveKit 负责 signaling、WebRTC 连接、SFU 媒体分发和传输层重连;TeleFuser 负责 HTTP 准入、token、模型
+worker、pipeline session、执行策略和模型状态清理。因此必须使用 LiveKit Cloud 或自托管 LiveKit Server;
+TeleFuser 不提供直接 SDP 接口。
+
+三份服务文档分别描述不同边界:
+
+- [服务指南](service.md)比较 `serve` 与 `stream-serve`;
+- 本文定义 LiveKit API、room 角色、容量、生命周期和部署行为;
+- [流式 Pipeline 调度器](stream_scheduler.md)定义 actor 所有权与 pipeline 内有界数据流。
+
+## Runtime 拓扑
+
+```mermaid
+flowchart LR
+ C[Controller] -->|创建 / 删除 session| API[TeleFuser HTTP API]
+ V[Viewers] -->|申请 viewer token| API
+ C <-->|WebRTC| LK[LiveKit signaling + SFU]
+ V <-->|WebRTC| LK
+ API --> A[Registry + 准入]
+ A --> W[一个进程内模型 worker]
+ W <-->|每个 session 一个 room runner| LK
+ W --> S[一个共享服务实例]
+ S --> P1[Pipeline session A]
+ S --> P2[Pipeline session B]
+```
+
+| 术语 | 含义与所有权 |
+|---|---|
+| 服务进程 | 一个 `telefuser stream-serve` 进程,包含 HTTP API、registry、准入 scheduler 和当前进程内 worker。 |
+| 模型 worker | 只加载一次 pipeline 文件,拥有一个服务实例,并统计常驻 session 容量。 |
+| 服务实例 | `get_service()` 返回的单个对象;模型权重及其 pipeline actor graph 只加载一次。 |
+| HTTP session | TeleFuser 对外提供的准入和生命周期记录;它与 room name 一一对应,准入后也与 room runner 一一对应。 |
+| Room runner | 一个 task,以及一个连接到 LiveKit room 的 TeleFuser worker participant;多个 runner 共享服务实例。 |
+| Pipeline session | `BidirectionalService.create_session()` 返回的用户独立状态,例如 control、noise、VAE 和模型 cache。 |
+| Stage actor | Pipeline 内部的执行所有者,不是拥有常驻 session 容量的模型 worker。 |
-LiveKit 负责浏览器 WebRTC 连接、room、重连、媒体传输和可靠数据消息;TeleFuser 负责模型 worker、准入、
-session 状态、pipeline 执行和 token 签发。因此必须使用 LiveKit Cloud 或自托管 LiveKit Server。
+当前 runtime 只支持一个 `in-process` 模型 worker,并且只调用一次 `get_service()`。多个用户不会加载多个
+模型副本。额外副本需要启动独立的 `stream-serve` 进程并在外部路由请求;各进程的 registry、队列、健康状态
+和 session 状态彼此独立。
-## 为什么选择 LiveKit
+## 服务契约与容量
-TeleFuser 面向高性能多模态生成模型推理,流式服务需要同时支持持续媒体输出、双向控制和长时间运行的
-有状态模型 session。LiveKit 提供的实时传输能力与这些目标相契合:
+| 契约 | 输入与输出 | 常驻 session 容量 |
+|---|---|---|
+| `ServerPushService` | 根据请求配置启动,无需 room control,渐进发布视频/音频。 | 只能为 1;`max_sessions_per_worker > 1` 时启动失败。 |
+| `BidirectionalService` | 创建用户独立状态,接收规范化 control,并持续输出 chunk。 | 只有在实现隔离状态并定义安全的跨 session 执行策略时才可大于 1。 |
-- room 为模型 session 提供稳定的传输边界,使浏览器连接与 session 自有的模型状态相互独立;
-- 媒体轨道承载渐进式视频和音频,data topic 承载控制、状态与有界 telemetry;
-- 受限 token 区分 controller、viewer 和 worker,适配 TeleFuser 的 session 所有权与权限模型;
-- `ServerPushService` 和 `BidirectionalService` 共用一个流式入口和客户端连接模型;
-- LiveKit 负责连接、重连和媒体交付,TeleFuser 可以专注于模型 worker、准入、pipeline 执行和资源释放。
+`max_sessions_per_worker` 是准入上限,不是副本数、batch size 或 graph edge capacity。仓库内
+LingBot-World-Fast 和 LingBot-World v2 支持多个常驻 session,并通过共享 execution lease 串行执行模型
+chunk。其他双向服务必须自行提供跨 session 并发策略。
-## 本地安装与启动
+## 本地开发链路
-LiveKit Python SDK 已包含在 TeleFuser 基础依赖中:
+LiveKit Python SDK 已包含在 TeleFuser 中。LiveKit Server 与当前平台的 `coturn` 软件包需要单独安装:
```bash
pip install -e .
-```
-另外安装开发用 LiveKit Server,并通过当前操作系统的包管理器安装 `coturn`:
-
-```bash
-# Debian/Ubuntu;其他平台请安装对应的 coturn 软件包。
+# Debian/Ubuntu;其他平台请使用对应的软件包。
sudo apt-get update
sudo apt-get install -y coturn
curl -sSL https://get.livekit.io | bash
-livekit-server --dev
```
-开发服务器监听 `ws://127.0.0.1:7880`,默认凭据为 `devkey` / `secret`,生产环境不得使用这些凭据。
-
-启动 TeleFuser:
+仓库内浏览器 demo 强制使用 TCP TURN relay。请在四个终端运行以下仅供开发使用的链路:
```bash
-telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
- --livekit-url ws://127.0.0.1:7880 \
- --livekit-api-key devkey \
- --livekit-api-secret secret \
- --port 8088 \
- --skip-validation
-```
+# Terminal 1:与浏览器配置匹配的 TURN relay
+turnserver -n -m 1 \
+ --listening-ip=127.0.0.1 --relay-ip=127.0.0.1 \
+ --listening-port=3478 --min-port=49160 --max-port=49200 \
+ --user=livekit-demo:livekit-demo-password --realm=livekit.local \
+ --fingerprint --lt-cred-mech --no-tls --no-dtls --no-cli \
+ --allow-loopback-peers
-同一命令也支持 server-push pipeline:
+# Terminal 2:signaling 与 SFU
+livekit-server --dev
-```bash
-telefuser stream-serve examples/stream_server/stream_video_replay.py \
+# Terminal 3:模型、准入和 session API
+TF_MODEL_ZOO_PATH=/path/to/model_zoo \
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
+telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey \
--livekit-api-secret secret \
+ --worker-gpu-map 0,1,2,3 \
+ --max-sessions-per-worker 2 \
+ --control-idle-timeout 10 \
--port 8088 \
--skip-validation
-```
-也可使用 `TELEFUSER_LIVEKIT_*` 环境变量;显式 CLI 参数优先。
-
-## 浏览器 Demo
-
-仓库内页面设置了 `iceTransportPolicy: relay`,因此必须启动与其匹配的 TCP TURN 服务;生产 LiveKit 部署
-可以使用不同的 TURN 配置。先启动以下仅供开发使用的 coturn 进程:
-
-```bash
-turnserver -n -m 1 \
- --listening-ip=127.0.0.1 \
- --relay-ip=127.0.0.1 \
- --listening-port=3478 \
- --min-port=49160 --max-port=49200 \
- --user=livekit-demo:livekit-demo-password \
- --realm=livekit.local \
- --fingerprint --lt-cred-mech \
- --no-tls --no-dtls --no-cli \
- --allow-loopback-peers
-```
-
-在第四个终端启动 LingBot 控制页面:
-
-```bash
+# Terminal 4:浏览器页面与 HTTP API proxy
python examples/stream_server/livekit_bidirectional_demo.py \
--server-url http://127.0.0.1:8088 \
--port 8092 \
--no-open
```
-打开 `http://127.0.0.1:8092`,选择初始图片并点击 **Start**。Demo 会代理 `/v1/stream/*` 请求、获取
-controller token、加入 LiveKit room、播放视频轨道,并通过 `tf.control` 发送页面或键盘相机控制消息。
+打开 `http://127.0.0.1:8092`,选择图片并点击 **Start**。使用 VS Code Remote SSH 时,把远端 TCP
+`8092`、`7880` 和 `3478` 映射到相同本地端口。页面会代理 session API,因此浏览器侧无需映射
+`8088`。
+
+Loopback TURN listener、静态密码、禁用 TLS、`--allow-loopback-peers`、LiveKit 开发凭据和
+`--skip-validation` 仅适用于可信开发主机。关闭时应先停止浏览器 session,再按 terminal 4 到 1 的顺序停止。
+
+## Session 创建与 room 加入
+
+TeleFuser 分配唯一 room name 并签发受限 token,但不会调用 LiveKit room-management API 显式创建 room;
+第一个 participant 加入时由 LiveKit 建立 room。
+
+```mermaid
+sequenceDiagram
+ participant C as Controller
+ participant API as TeleFuser API
+ participant A as 准入 scheduler
+ participant W as Room runner
+ participant S as 共享服务实例
+ participant LK as LiveKit
+
+ C->>API: POST /v1/stream/sessions
+ API->>A: 申请常驻容量
+ alt 有空闲 slot
+ A-->>API: assigned
+ API-->>C: 200 session_id、room、controller token
+ W->>LK: 以 worker 身份加入
+ W->>S: 创建 pipeline session 或 stream task
+ C->>LK: 使用 controller token 加入
+ else HTTP 队列有空间
+ A-->>API: queued
+ API-->>C: 202 queue_position 与 token
+ A->>W: slot 释放后启动
+ else 无排队容量
+ API-->>C: 429
+ end
+```
-使用 VS Code Remote SSH 时,需要映射 demo HTTP 端口、LiveKit signaling 端口,以及 LiveKit 使用的 TURN
-listener。仓库 demo 固定使用 `turn:127.0.0.1:3478?transport=tcp` 和开发凭据 `livekit-demo` /
-`livekit-demo-password`;生产环境必须同时修改浏览器配置和 LiveKit 部署。
+排队响应已经包含 room name 和 controller token,但在准入并启动 room runner 前不会有输出。Token 生命周期
+限制 token 可用于加入的时间,与 TeleFuser session 清理是两个不同概念。
-把远端 TCP `8092`、`7880` 和 `3478` 映射到相同本地端口,然后打开 `http://127.0.0.1:8092`。Loopback
-listener、静态密码、禁用 TLS 和 `--allow-loopback-peers` 只适用于通过隧道访问的可信开发主机,不能复制到
-公网生产部署。
+## 一处控制与多处观看
-完整浏览器链路此时包含 coturn(`3478`)、LiveKit(`7880`)、TeleFuser(`8088`)和页面(`8092`)。
-启动 session 前,`curl http://127.0.0.1:8088/v1/service/health` 应显示 ready 且 worker idle。成功运行时,
-页面会显示视频轨道以及 `control_state`、生成 Stage 和 `chunk_sent` 等状态。关闭服务时先停止 session 或
-关闭浏览器页面,再按相反顺序停止四个进程,避免 LiveKit 和模型 worker drain 时浏览器持续重连。
+```mermaid
+flowchart LR
+ C[Controller] -->|reliable tf.control| R[一个 LiveKit room]
+ W[TeleFuser worker] -->|一份媒体发布 + tf.status| R
+ R -->|track + room data| C
+ R -->|相同 track + room data| V1[Viewer 1]
+ R -->|相同 track + room data| VN[Viewer N]
+```
-## 架构与生命周期
+| 角色 | LiveKit grant | TeleFuser 语义 |
+|---|---|---|
+| Controller | 可订阅、可发布 data、不能发布媒体 track | Session 配置的 controller identity;只有它发送的 `tf.control` 会被接受。 |
+| Viewer | 可订阅、不能发布 data 或媒体 track | 观看相同输出与状态,没有 pipeline 控制权限。 |
+| Worker | 可发布媒体与 data、不订阅 | 运行 session 并发布一份输出,由 LiveKit 分发给所有订阅者。 |
+
+HTTP session 只创建一次,然后为每个 viewer 使用不同 identity 调用
+`POST /v1/stream/sessions/{session_id}/tokens`。Viewer 加入已有 room,不创建新的 HTTP session、runner 或
+pipeline session,也不占用 `max_sessions_per_worker`、不进入 TeleFuser 队列、不申请 execution lease、不
+复制模型状态、不触发推理。不过 LiveKit/SFU 的分发带宽和订阅开销仍会随 viewer 数量增长。
+
+Viewer 加入或离开不会改变 TeleFuser 准入和 session 状态。当前也不会监听 controller 离开,控制结束时客户端
+必须显式关闭 session。
+
+## 准入、队列与 LingBot 执行
+
+```mermaid
+flowchart TD
+ N[新 HTTP session] --> C{有常驻 slot?}
+ C -->|有| R[启动 room runner]
+ C -->|无| Q{HTTP 队列有空间?}
+ Q -->|有| H[HTTP 202,FIFO 等待]
+ H -->|slot 释放| R
+ Q -->|无| X[HTTP 429]
+ R --> P[常驻 pipeline session]
+ P --> L{LingBot 收到合法 control?}
+ L -->|是| E[Execution-lease FIFO]
+ E --> G[一个 active session 提交 chunk]
+ G --> B[Chunk 边界]
+ B -->|超时且有等待者| K[挂起持有者并授权下一个]
+ B -->|否则| G
+```
-```text
-Browser ── HTTP /v1/stream/* ──> TeleFuser session API
- │ │
- └── LiveKit media/data ──> LiveKit room <── TeleFuser worker
- │
- └── stream pipeline actor graph
+系统有三个相互独立的调度边界:
+
+| 边界 | 容量所有者 | 等待的含义 |
+|---|---|---|
+| HTTP 准入队列 | LiveKit runtime | 所有常驻 slot 已占用;`queue_size` 限制该 FIFO,零表示禁用。 |
+| LingBot execution-lease 队列 | 共享 LingBot 服务实例 | 已准入 session 申请模型执行,但另一个 session 正持有 lease。 |
+| Pipeline artifact 队列 | `StreamingPipelineOrchestrator` | Stage 或下游有界 edge 暂时不能准入新的 sequence item。 |
+
+Execution lease 是 LingBot 专属策略。合法的 `control_state`、`control`、`prompt` 或 `reset` 会记录
+活跃时间,并让 waiting/parked session 排队。如果存在等待者,且持有者已超过
+`control_idle_timeout` 没有控制活动,持有者会完成在途 chunk,然后停放并交出 lease;切换不会中断 chunk。
+
+```mermaid
+stateDiagram-v2
+ [*] --> waiting
+ waiting --> queued: 合法 control
+ queued --> active: 获得 lease
+ active --> parked: 等待者 + 空闲超时 + chunk 边界
+ parked --> queued: 新的合法 control
+ active --> closing: session 清理
+ queued --> closing: session 清理
+ parked --> closing: session 清理
+ closing --> [*]
+```
+
+停放不会关闭 session、释放常驻 slot 或清除 cache。应根据实测单 session 显存余量设置
+`max_sessions_per_worker`。持续按住输入时 controller 必须重发 `control_state`;仓库内浏览器在按键保持时
+每秒发送一次。交出 execution lease 不会让 session 回到 HTTP 队列。
+
+## Session 生命周期与当前限制
+
+```mermaid
+stateDiagram-v2
+ [*] --> pending: POST session
+ pending --> assigned: slot 可用
+ pending --> queued: 等待 slot
+ pending --> [*]: 拒绝
+ queued --> assigned: slot 释放
+ assigned --> joining_room
+ joining_room --> starting_pipeline
+ starting_pipeline --> running
+ queued --> draining: DELETE
+ assigned --> draining: DELETE
+ joining_room --> draining: DELETE
+ starting_pipeline --> draining: DELETE
+ running --> draining: DELETE
+ draining --> closed: 清理完成
+ running --> closed: stop 或正常完成
+ joining_room --> failed: runner 错误
+ starting_pipeline --> failed: pipeline 错误
+ running --> failed: runner 或 pipeline 错误
```
-1. Controller 通过 `POST /v1/stream/sessions` 创建 session。
-2. Scheduler 对其准入、排队或拒绝,并绑定一个 worker。
-3. TeleFuser 创建 LiveKit room,返回权限受限的 controller token。
-4. Worker 加入 room 并启动 pipeline。
-5. 视频和 PCM16 音频作为 LiveKit track 发布;状态与指标使用可靠 data topic。
-6. 对 `BidirectionalService`,只有 controller 可以把规范化 control 消息送入 pipeline。
-7. 删除、超时、controller 离开或 pipeline 完成时,按 actor 所有权释放状态并关闭 room。
+清理过程停止接收新任务,通过状态所有者关闭 pipeline session,断开 worker participant,释放常驻 slot,并
+准入下一条 HTTP 排队 session。TeleFuser 不会显式删除 LiveKit room;剩余浏览器 participant 与 LiveKit 部署
+共同决定传输 room 后续的生命周期。
-每个流式 stage worker 只属于一个 pipeline actor;重连不会在 worker 之间搬移 actor cache。Server-push
-pipeline 根据请求 config 启动并持续输出 chunk;bidirectional pipeline 额外提供 create、pull、control、close。
+当前 runtime 有以下需要显式说明的限制:
+
+- `session_timeout` 会记录 `expires_at`,但当前没有后台任务把 session 改为 `expired`;
+- `controller_timeout` 和 `room_empty_timeout` 可配置,但尚未执行;
+- 没有监听 participant 事件,因此 `participant_count` 始终为 `0`,participant 离开不会触发清理;
+- 终态记录在进程生命周期内保留于内存 registry,不在进程间共享,也不会在重启后恢复;
+- Controller 应发送 `stop` 或调用 DELETE;仅关闭浏览器页面不会释放容量。
## HTTP API
| 接口 | 方法 | 用途 |
|---|---|---|
| `/v1/stream/sessions` | POST | 创建并准入 controller session |
-| `/v1/stream/sessions/{session_id}` | GET | 查询 session 状态 |
-| `/v1/stream/sessions/{session_id}` | DELETE | drain 并关闭 session |
-| `/v1/stream/sessions/{session_id}/tokens` | POST | 创建 viewer token |
-| `/v1/stream/health` | GET | LiveKit scheduler/worker 健康状态 |
+| `/v1/stream/sessions/{session_id}` | GET | 读取内存中的 session 记录 |
+| `/v1/stream/sessions/{session_id}` | DELETE | Drain、关闭并释放 session |
+| `/v1/stream/sessions/{session_id}/tokens` | POST | 签发仅订阅的 viewer token |
+| `/v1/stream/health` | GET | Scheduler 与聚合 worker 健康状态 |
| `/v1/service/health` | GET | 通用服务健康状态 |
-| `/v1/service/ready` | GET | readiness probe |
-| `/v1/service/metadata` | GET | Pipeline 与 transport metadata |
-| `/v1/service/metrics` | GET | Prometheus 指标 |
+| `/v1/service/ready` | GET | Readiness probe |
+| `/v1/service/metadata` | GET | Runtime 拓扑与服务 metadata |
+| `/v1/service/metrics` | GET | Prometheus 文本指标 |
+| `/v1/service/metrics/json` | GET | JSON 服务与 LiveKit 健康指标 |
创建 controller session:
@@ -153,26 +280,7 @@ curl -X POST http://127.0.0.1:8088/v1/stream/sessions \
}'
```
-如需执行一分钟 LingBot-World v2 回放,启动
-`examples/lingbot/lingbot_world_v2_image_to_video_h100.py` 并使用:
-
-```json
-{
- "fps": 16,
- "chunk_size": 4,
- "frame_num": 957,
- "max_duration_seconds": 60.0
-}
-```
-
-完整 chunk 策略把该请求映射为 60 个 chunk 和 59.75 秒输出媒体。v2 示例使用 `local_attn_size=18` 与
-`sink_size=6`,因此 KV 容量保持固定,session 自有的 noise 与 VAE 状态增量推进。可复现 LiveKit workload
-和日期化的四卡实测见 [TeleFuser 与 AIPerf](benchmark_aiperf.md)。
-
-成功响应包含 `session_id`、`room`、`livekit_url`、`token`、`worker_id` 和 `status`。排队时返回 HTTP 202
-和 `queue_position`;队列长度为零且 worker 全忙时返回 HTTP 429。
-
-创建没有控制权限的 viewer token:
+为同一 room 创建 viewer token:
```bash
curl -X POST http://127.0.0.1:8088/v1/stream/sessions/
/tokens \
@@ -180,82 +288,113 @@ curl -X POST http://127.0.0.1:8088/v1/stream/sessions//tokens \
-d '{"identity":"viewer-1"}'
```
-主动关闭:
+关闭并释放 session:
```bash
curl -X DELETE http://127.0.0.1:8088/v1/stream/sessions/
```
+直接准入返回 HTTP 200;有界等待返回 HTTP 202 和 `queue_position`;队列禁用或已满时返回 HTTP 429。
+一分钟 LingBot-World v2 workload 与四张 H100 的实测结果见
+[TeleFuser 与 AIPerf](benchmark_aiperf.md)。
+
## LiveKit 数据协议
-| Topic | 方向 | 内容 |
-|---|---|---|
-| `tf.control` | controller 到 worker | 可靠 JSON control 消息 |
-| `tf.status` | worker 到 room | 生命周期和 chunk 状态 |
-| `tf.metrics` | worker 到 room | 有界 runtime 指标 |
-| `tf.asset` | 保留 | 未来的有界 asset 消息 |
+| Topic | 方向 | 传输 | 当前用途 |
+|---|---|---|---|
+| `tf.control` | Controller 到 worker | 仓库内客户端使用 reliable | `control_state`、`control`、`prompt`、`reset` 和 `stop` |
+| `tf.status` | Worker 到 room | Reliable | Runner 生命周期、错误、chunk metadata 和完成状态 |
+| `tf.metrics` | Worker 到 room | Lossy | Room client 支持,但通用 runner 当前不发送 |
+| `tf.asset` | 保留 | 未定义 | 未来的有界 asset 消息 |
-支持 `control_state`、`control`、`prompt`、`reset` 和 `stop`,例如:
+Control 示例:
```json
{"type":"control_state","controls":["w","j"]}
```
-也支持带版本的 envelope:
+也可使用带版本的 envelope:
```json
{"version":1,"session_id":"","type":"control_state","payload":{"controls":["w"]}}
```
-消息默认受 `TELEFUSER_LIVEKIT_MAX_DATA_MESSAGE_BYTES`(12 KiB)限制。未知 control、重复项、非法 JSON、
-错误 topic、session 不匹配,以及 viewer 发出的 control 都会被拒绝。
+入站消息默认受 `max_data_message_bytes`(12 KiB)限制。错误 topic、非 controller sender、非法 JSON、未知
+control、重复项和 session 不匹配都会被拒绝。
-## CLI 与环境变量
+## CLI、环境变量与 GPU 放置
```text
telefuser stream-serve PIPE_PATH [OPTIONS]
```
-主要选项包括 `--host`、`--port`、`--livekit-url`、`--livekit-api-key`、`--livekit-api-secret`、
-`--num-workers`、`--worker-gpu-map`、`--queue-size`、`--session-timeout`、`--token-ttl`、
-`--controller-timeout`、`--room-empty-timeout` 和 `--worker-mode`。
+完整选项见 `telefuser stream-serve --help`。以下选项具有重要 runtime 语义:
-| 环境变量 | 默认值 | 含义 |
+| 选项 | 默认值 | 语义 |
|---|---:|---|
-| `TELEFUSER_LIVEKIT_URL` | 必填 | LiveKit WebSocket URL |
-| `TELEFUSER_LIVEKIT_API_KEY` | 必填 | 用于签发 token 的 API key |
-| `TELEFUSER_LIVEKIT_API_SECRET` | 必填 | 用于签发 token 的 API secret |
-| `TELEFUSER_LIVEKIT_HOST` | `0.0.0.0` | HTTP API 监听地址 |
-| `TELEFUSER_LIVEKIT_PORT` | `8088` | HTTP API 端口 |
-| `TELEFUSER_LIVEKIT_NUM_WORKERS` | `1` | 模型 worker 数 |
-| `TELEFUSER_LIVEKIT_WORKER_GPU_MAP` | 未设置 | 分号分隔的 GPU group,如 `0,1;2,3` |
-| `TELEFUSER_LIVEKIT_QUEUE_SIZE` | `0` | 排队数量;零表示 busy 时立即拒绝 |
-| `TELEFUSER_LIVEKIT_SESSION_TIMEOUT` | `1800` | session 最大生命周期(秒) |
-| `TELEFUSER_LIVEKIT_TOKEN_TTL` | `3600` | join token 生命周期(秒) |
-| `TELEFUSER_LIVEKIT_CONTROLLER_TIMEOUT` | `60` | controller 离开后的宽限期 |
-| `TELEFUSER_LIVEKIT_ROOM_EMPTY_TIMEOUT` | `30` | room 为空后的宽限期 |
-
-当前 runtime 仅支持一个 in-process worker。在 process-worker 隔离实现之前,如需更多 worker,应启动独立服务
-进程。`--skip-validation` 只应用于可信本地文件,不建议生产使用。
-
-## 生产部署
-
-- 使用 LiveKit Cloud 或官方自托管部署方式,不要暴露 `livekit-server --dev`。
-- 使用独立 API 凭据,API secret 只能保存在 TeleFuser 服务端。
-- 在 LiveKit 中配置 TLS、advertised node address、UDP/TCP media port 和 TURN。
-- 通过部署层的鉴权与网络策略限制 TeleFuser HTTP API。
-- 监控 `/v1/service/ready`、worker failure、queue depth 和 session expiration。
-- Chunk period 表示相邻输出 cadence;实时运行要求 p95 cadence 小于一个 chunk 的媒体时长,并为传输和编码
- 留出余量。Pipeline residence 和客户端 delivery FPS 是不同指标。
-
-## 故障排查
-
-- **HTTP ready 但没有媒体:**确认浏览器和 worker 都能访问 LiveKit URL,并检查 participant/track 日志。
-- **浏览器反复重连:**检查 signaling、TURN、防火墙和 LiveKit advertised node address。
-- **控制无效:**确认发送者使用 controller token、topic 为 `tf.control`,且 control 受支持。
-- **HTTP 429:**所有 worker 都在忙且 `queue_size=0`,或队列已满。
-- **Session 未释放:**调用 session DELETE 接口并检查 controller/room timeout。
-- **本地 LiveKit 连接返回代理 HTTP 503:**部分 native SDK 路径会读取 `HTTP_PROXY`,但不会应用
- `NO_PROXY`。连接 `ws://127.0.0.1:7880` 时,启动 TeleFuser 前应取消 `HTTP_PROXY`、`HTTPS_PROXY`、
- `ALL_PROXY` 及其小写变量。
-- **强制退出后残留 GPU worker:**重启前终止残留的 `spawn_main` 进程。
+| `--host`、`--port` | `0.0.0.0`、`8088` | HTTP 监听地址 |
+| `--num-workers` | `1` | 当前 runtime 必须保持为 `1` |
+| `--worker-gpu-map` | 未设置 | 当前 worker 的一个逻辑 GPU group,例如 `0,1,2,3` |
+| `--max-sessions-per-worker` | `1` | 常驻双向 session 数,不是副本数 |
+| `--queue-size` | `0` | HTTP 准入 FIFO 长度;零表示容量满时拒绝 |
+| `--control-idle-timeout` | `10` | 有其他 session 等待时,LingBot lease 的控制空闲阈值 |
+| `--session-timeout` | `1800` | 记录 `expires_at`,当前尚未执行 |
+| `--token-ttl` | `3600` | Join token 生命周期 |
+| `--controller-timeout` | `60` | 预留,当前尚未执行 |
+| `--room-empty-timeout` | `30` | 预留,当前尚未执行 |
+| `--worker-mode` | `in-process` | CLI 接受 `process`,但 runtime 尚未实现 |
+
+对应 CLI 值未设置时,命令可回退到 `TELEFUSER_LIVEKIT_URL`、
+`TELEFUSER_LIVEKIT_API_KEY`、`TELEFUSER_LIVEKIT_API_SECRET`、
+`TELEFUSER_LIVEKIT_WORKER_GPU_MAP`、`TELEFUSER_LIVEKIT_MAX_SESSIONS_PER_WORKER` 和
+`TELEFUSER_LIVEKIT_CONTROL_IDLE_TIMEOUT`。仅通过环境变量配置的字段包括
+`TELEFUSER_LIVEKIT_DEFAULT_FPS`(默认 `16`)、`TELEFUSER_LIVEKIT_MAX_DATA_MESSAGE_BYTES`(默认
+`12288`)和 `TELEFUSER_LIVEKIT_CORS_ALLOW_ORIGINS`(默认 `["*"]`)。
+
+其他 Click 选项当前会显式传递其界面默认值,因此这些字段应使用 CLI 选项,而不是同名环境变量。
+
+在当前进程内 runtime 中,`worker_gpu_map` 只记录 scheduler 拓扑,并把 group 大小作为 `gpu_num` 传给
+`get_service()`;它不会设置 `CUDA_VISIBLE_DEVICES`、隔离设备或重写 `ModelRuntimeConfig`。请先用
+`CUDA_VISIBLE_DEVICES` 选择物理 GPU,并确保 pipeline 使用对应的进程内 device index。例如:
+
+```bash
+CUDA_VISIBLE_DEVICES=4,5,6,7 \
+telefuser stream-serve PIPE_PATH --worker-gpu-map 0,1,2,3
+```
+
+此时物理 GPU 4-7 在进程内显示为 device 0-3,并向 `get_service()` 传递 `gpu_num=4`;仍只加载一个服务实例。
+
+## 可观测性
+
+| 信号 | 准确含义 |
+|---|---|
+| `workers_busy` | 至少保留一个 session 的模型 worker 数;当前 runtime 中只能为 `0` 或 `1`。 |
+| `workers_idle` | 没有常驻 session 且未失败的模型 worker 数。 |
+| `workers_failed` | 聚合状态为 failed 的 worker 数。 |
+| `queued_sessions` | 只统计 HTTP 准入队列,不包括 LingBot lease 或 pipeline artifact 等待。 |
+| `livekit_connected` | 根据聚合 worker 状态是否为 `starting_pipeline`、`running` 或 `draining` 推导,并非对 LiveKit Server 的直接探测。 |
+| `participant_count` | 当前始终为 `0`,因为 participant 事件尚未写入 registry。 |
+| `lease_queued`、`lease_granted`、`lease_parked` | 通过 `tf.status` 发布的 LingBot execution-lease 状态变化。 |
+
+Room runner 尚未进入 pipeline startup 前,`livekit_connected=false` 是正常状态,并不表示模型加载失败。
+Pipeline 性能指标应与客户端交付指标分开解释,参见[监控指标](metrics.md)和
+[TeleFuser 与 AIPerf](benchmark_aiperf.md)。
+
+## 生产部署与故障排查
+
+- 使用 LiveKit Cloud 或官方自托管部署方式,不要暴露 `livekit-server --dev`;
+- LiveKit API secret 只能保存在 TeleFuser 服务端,并应在部署层为 HTTP API 增加鉴权;
+- 在 LiveKit 中配置 TLS、advertised node address、UDP/TCP media port 和 TURN;
+- 根据 GPU 显存设置常驻 session 容量,根据 SFU 带宽设置 viewer 容量;
+- 监控 readiness、worker failure、HTTP queue depth、pipeline cadence 和显式 session 清理。
+
+常见问题:
+
+- **Ready 但没有媒体:**确认 worker 与浏览器都能访问 LiveKit,并检查 participant/track 日志;
+- **浏览器反复重连:**检查 signaling、TURN 凭据、防火墙和 LiveKit advertised address;
+- **控制无效:**使用 controller token、`tf.control`、支持的类型和配置的 identity;
+- **HTTP 429:**常驻 slot 和配置的 HTTP 队列已满,或队列被禁用;
+- **客户端离开后 session 仍存在:**尚未实现离开清理,请发送 `stop` 或调用 DELETE;
+- **本地 LiveKit 返回 proxy HTTP 503:**本地使用 `ws://127.0.0.1:7880` 时,取消大小写形式的
+ `HTTP_PROXY`、`HTTPS_PROXY` 和 `ALL_PROXY`;部分 native SDK 路径不会应用 `NO_PROXY`;
+- **强制退出后残留 worker:**重启前终止遗留的 `spawn_main` 子进程。
diff --git a/examples/lingbot/README.md b/examples/lingbot/README.md
index e26a846..24c3990 100644
--- a/examples/lingbot/README.md
+++ b/examples/lingbot/README.md
@@ -160,10 +160,32 @@ python examples/lingbot/lingbot_world_fast_image_to_video_h100.py --help
## Real-Time Streaming
-The same examples expose both offline generation and a stream-server `get_service()` entry point. Configure the
-service topology with `stream-serve --worker-gpu-map`; the size of the assigned GPU group is passed to
-`get_service(gpu_num=...)`. Use `CUDA_VISIBLE_DEVICES` to select the physical devices. Do not use `torchrun` because
-TeleFuser creates workers internally.
+The same examples expose offline generation and a stream-server `get_service()` entry point. Use
+`CUDA_VISIBLE_DEVICES` to select physical devices; the logical group size from `--worker-gpu-map` is passed to
+`get_service(gpu_num=...)`. The map does not change GPU visibility. Do not use `torchrun` because TeleFuser creates
+workers internally. The runnable coturn, LiveKit, TeleFuser, browser, and VS Code forwarding workflow is maintained in
+the [stream examples README](../stream_server/README.md).
+
+Each admitted LingBot room owns independent pipeline-session state, but every session shares one service instance and
+one model-execution lease:
+
+```mermaid
+flowchart LR
+ R1[LiveKit room A] --> S1[Pipeline session A]
+ R2[LiveKit room B] --> S2[Pipeline session B]
+ S1 --> L[One LingBot execution lease]
+ S2 --> L
+ L --> O[StreamingPipelineOrchestrator]
+ O --> E[VAE encode actor]
+ O --> D[DiT actor]
+ O --> V[VAE decode actor]
+```
+
+`max_sessions_per_worker=2` retains two isolated session states; it does not create a second model replica. Only one
+session submits a model chunk at a time. With a waiter present, a holder that has no valid control activity for
+`control_idle_timeout` yields after its current chunk, while its cache and retained slot remain allocated. The
+browser sends a one-second `control_state` heartbeat while a key remains held. Use separate service processes and
+external routing for additional replicas.
### Scheduler and Stage Placement
@@ -234,78 +256,15 @@ of media, while p95 cadence was 1.865 seconds. See the
[AIPerf benchmark guide](../../docs/en/benchmark_aiperf.md) for the workload,
metric boundary, artifact path, and the observed client cleanup issue.
-### LiveKit Transport
-
-LiveKit provides room lifecycle, signaling, reconnects, adaptive media transport, and reliable control messages.
-The Python clients are base TeleFuser dependencies; install the LiveKit Server and your platform's `coturn` package
-separately for local development:
-
-```bash
-curl -sSL https://get.livekit.io | bash
-```
-
-The checked-in browser page forces TCP TURN relay. Start its matching development coturn service in terminal 1:
-
-```bash
-turnserver -n -m 1 \
- --listening-ip=127.0.0.1 --relay-ip=127.0.0.1 \
- --listening-port=3478 --min-port=49160 --max-port=49200 \
- --user=livekit-demo:livekit-demo-password --realm=livekit.local \
- --fingerprint --lt-cred-mech --no-tls --no-dtls --no-cli \
- --allow-loopback-peers
-```
-
-Start LiveKit in terminal 2:
-
-```bash
-livekit-server --dev
-```
-
-Start the four-GPU LingBot worker and its session API in terminal 3:
-
-```bash
-TF_MODEL_ZOO_PATH=/path/to/model_zoo \
-CUDA_VISIBLE_DEVICES=0,1,2,3 \
-telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
- --livekit-url ws://127.0.0.1:7880 \
- --livekit-api-key devkey \
- --livekit-api-secret secret \
- --num-workers 1 \
- --worker-gpu-map 0,1,2,3 \
- --port 8088 \
- --skip-validation
-```
-
-For the fixed-window v2 model, replace the pipeline path in that command with
-`examples/lingbot/lingbot_world_v2_image_to_video_h100.py`; the LiveKit and worker options stay the same.
-
-Then start the bundled browser demo in terminal 4:
-
-```bash
-python examples/stream_server/livekit_bidirectional_demo.py \
- --server-url http://127.0.0.1:8088 \
- --port 8092 \
- --no-open
-```
-
-The demo proxies the TeleFuser session API, so VS Code Remote SSH needs TCP forwarding for demo port `8092`, LiveKit
-signaling port `7880`, and TURN listener `3478`; `8088` does not need forwarding. Open `http://127.0.0.1:8092`,
-select an initial image, and click **Start**. See the [Stream Server guide](../../docs/en/stream_server.md) for the API
-contract, control topics, shutdown order, troubleshooting, and production deployment notes.
-
-The current runtime supports exactly one `in-process` worker. One admitted LiveKit room owns one model pipeline
-instance until the session is closed or expires. Use separate service processes for additional workers until
-process-worker mode is implemented.
-
-Select the initial image in the browser before connecting; it is included in the session request. Real-time camera
-poses come from LiveKit control messages. The LingBot-World v2 service uses the bundled `intrinsics.npy` and its `832x480`
-calibration size by default, matching the offline example. A request can override that calibration. Other LingBot
-services with neither configured nor request-provided intrinsics center the principal point on the selected image and
-use its width as both focal lengths. Requests with calibrated intrinsics should also send `intrinsics_width` and
-`intrinsics_height` so the service can transform them from calibration pixels to output pixels.
-
### Camera Controls
+Select the initial image before connecting; it is included in the session request. Real-time camera poses arrive as
+LiveKit control messages. LingBot-World v2 uses the bundled `intrinsics.npy` and its `832x480` calibration size by
+default, matching the offline example. A request can override it. Services with no configured or request-provided
+intrinsics center the principal point on the selected image and use its width as both focal lengths. Calibrated
+requests should also send `intrinsics_width` and `intrinsics_height` so the service can transform calibration pixels
+to output pixels.
+
The page has separate translation and rotation pads:
| Input | Camera operation |
diff --git a/examples/stream_server/README.md b/examples/stream_server/README.md
index 4608d05..c45ac06 100644
--- a/examples/stream_server/README.md
+++ b/examples/stream_server/README.md
@@ -24,10 +24,15 @@ turnserver -n -m 1 \
livekit-server --dev
# Terminal 3: TeleFuser model and session API
+TF_MODEL_ZOO_PATH=/path/to/model_zoo \
+CUDA_VISIBLE_DEVICES=0,1,2,3 \
telefuser stream-serve examples/lingbot/lingbot_world_fast_image_to_video_h100.py \
--livekit-url ws://127.0.0.1:7880 \
--livekit-api-key devkey \
--livekit-api-secret secret \
+ --worker-gpu-map 0,1,2,3 \
+ --max-sessions-per-worker 2 \
+ --control-idle-timeout 10 \
--port 8088 \
--skip-validation
@@ -42,6 +47,11 @@ Open `http://127.0.0.1:8092`, choose an image, and click **Start**. For VS Code
`7880`, and `3478` to the same local ports; the API proxy means `8088` does not need forwarding. Stop the browser
session first, then stop terminals 4 through 1 in reverse order.
+This command starts one process, one in-process model worker, and one shared LingBot service instance. It exposes four
+physical GPUs as process-local devices 0-3, declares one four-device logical worker group, and retains up to two
+independent sessions. The LingBot execution lease serializes their model chunks; it is not a generic replication
+option.
+
The LiveKit Python SDK is part of TeleFuser's base dependencies; the LiveKit Server is installed and operated
-separately. See the [Stream Server guide](../../docs/en/stream_server.md) for the session API, data topics, worker
-lifecycle, remote development, and production deployment.
+separately. See the [Stream Server guide](../../docs/en/stream_server.md) for room roles and viewer fan-out, the exact
+GPU-map boundary, session API, queues, lifecycle, observability, remote development, and production deployment.
diff --git a/examples/stream_server/_control_demo_ui.py b/examples/stream_server/_control_demo_ui.py
index fe6e553..cb56460 100644
--- a/examples/stream_server/_control_demo_ui.py
+++ b/examples/stream_server/_control_demo_ui.py
@@ -366,6 +366,7 @@
const DEFAULT_PROMPT = __PROMPT__;
const ICE_GATHER_TIMEOUT_MS = __ICE_GATHER_TIMEOUT_MS__;
const MAX_IMAGE_BYTES = 10 * 1024 * 1024;
+const CONTROL_HEARTBEAT_MS = 1000;
let pc = null;
let dc = null;
@@ -513,11 +514,11 @@
if (btn) btn.classList.toggle("active", active);
}
-function sendControlState() {
+function sendControlState(logMessage = true) {
if (!dc || dc.readyState !== "open") return;
const msg = JSON.stringify({ type: "control_state", controls: Array.from(pressedControls).sort() });
dc.send(msg);
- log("out", msg);
+ if (logMessage) log("out", msg);
}
function setControlPressed(control, active) {
@@ -576,6 +577,9 @@
document.addEventListener("visibilitychange", () => {
if (document.hidden) releaseAllControls(true);
});
+setInterval(() => {
+ if (pressedControls.size > 0) sendControlState(false);
+}, CONTROL_HEARTBEAT_MS);
window.addEventListener("pagehide", () => {
releaseAllControls(true);
if (dc && dc.readyState === "open") dc.send(JSON.stringify({ type: "stop" }));
diff --git a/examples/stream_server/livekit_bidirectional_demo.py b/examples/stream_server/livekit_bidirectional_demo.py
index 80d785a..dfffcbe 100644
--- a/examples/stream_server/livekit_bidirectional_demo.py
+++ b/examples/stream_server/livekit_bidirectional_demo.py
@@ -48,6 +48,7 @@ def _shared_demo_parts() -> tuple[str, str, str, str, str]:
const DEFAULT_IMAGE_PATH = __DEFAULT_IMAGE_PATH__;
const DEFAULT_PROMPT = __PROMPT__;
const MAX_IMAGE_BYTES = 10 * 1024 * 1024;
+const CONTROL_HEARTBEAT_MS = 1000;
const CONTROL_TOPIC = "tf.control";
const STATUS_TOPIC = "tf.status";
const METRICS_TOPIC = "tf.metrics";
diff --git a/telefuser/entrypoints/cli/main.py b/telefuser/entrypoints/cli/main.py
index c5d2ada..8319de4 100644
--- a/telefuser/entrypoints/cli/main.py
+++ b/telefuser/entrypoints/cli/main.py
@@ -139,9 +139,30 @@ def serve(
@click.option("--livekit-url", default=None, type=str, help="LiveKit server URL")
@click.option("--livekit-api-key", default=None, type=str, help="LiveKit API key")
@click.option("--livekit-api-secret", default=None, type=str, help="LiveKit API secret")
-@click.option("--num-workers", default=1, type=int, help="Number of TeleFuser model workers")
-@click.option("--worker-gpu-map", default=None, type=str, help="GPU groups, for example '0,1;2,3'")
-@click.option("--queue-size", default=0, type=int, help="Maximum queued sessions; 0 rejects when busy")
+@click.option(
+ "--num-workers",
+ default=1,
+ type=int,
+ help="Number of model workers; the current in-process runtime requires 1",
+)
+@click.option(
+ "--max-sessions-per-worker",
+ default=None,
+ type=int,
+ help="Maximum retained sessions per model worker",
+)
+@click.option(
+ "--worker-gpu-map", default=None, type=str, help="GPU group for the current worker, for example '0,1,2,3'"
+)
+@click.option(
+ "--queue-size", default=0, type=int, help="Maximum queued sessions; 0 rejects when retained slots are full"
+)
+@click.option(
+ "--control-idle-timeout",
+ default=None,
+ type=float,
+ help="Seconds without control activity before a LingBot execution lease may yield",
+)
@click.option("--session-timeout", default=1800, type=int, help="Maximum session lifetime in seconds")
@click.option("--token-ttl", default=3600, type=int, help="LiveKit join token TTL in seconds")
@click.option("--controller-timeout", default=60, type=int, help="Seconds to keep a session after controller leaves")
@@ -150,7 +171,7 @@ def serve(
"--worker-mode",
type=click.Choice(["in-process", "process"], case_sensitive=False),
default="in-process",
- help="Worker isolation mode",
+ help="Worker isolation mode; the current runtime supports in-process only",
)
@click.option(
"--security-level",
@@ -172,8 +193,10 @@ def stream_serve(
livekit_api_key: str | None,
livekit_api_secret: str | None,
num_workers: int,
+ max_sessions_per_worker: int | None,
worker_gpu_map: str | None,
queue_size: int,
+ control_idle_timeout: float | None,
session_timeout: int,
token_ttl: int,
controller_timeout: int,
@@ -208,8 +231,10 @@ def stream_serve(
livekit_api_key=livekit_api_key,
livekit_api_secret=livekit_api_secret,
num_workers=num_workers,
+ max_sessions_per_worker=max_sessions_per_worker,
worker_gpu_map=worker_gpu_map,
queue_size=queue_size,
+ control_idle_timeout=control_idle_timeout,
session_timeout=session_timeout,
token_ttl=token_ttl,
controller_timeout=controller_timeout,
diff --git a/telefuser/pipelines/lingbot_world_fast/lease.py b/telefuser/pipelines/lingbot_world_fast/lease.py
new file mode 100644
index 0000000..6d36279
--- /dev/null
+++ b/telefuser/pipelines/lingbot_world_fast/lease.py
@@ -0,0 +1,254 @@
+"""Exclusive execution leases for time-sliced LingBot sessions."""
+
+from __future__ import annotations
+
+import threading
+import time
+from collections import deque
+from dataclasses import dataclass
+from typing import Literal
+
+ExecutionLeaseStatus = Literal["waiting", "queued", "active", "parked", "closing"]
+
+
+@dataclass(frozen=True)
+class ExecutionLeaseTransition:
+ """One externally observable lease state transition."""
+
+ session_id: str
+ status: ExecutionLeaseStatus
+
+
+@dataclass(frozen=True)
+class ExecutionLeaseSnapshot:
+ """Immutable state for one registered session."""
+
+ session_id: str
+ status: ExecutionLeaseStatus
+ last_activity_at: float | None
+ busy: bool
+ queue_position: int | None
+
+
+@dataclass
+class _ExecutionLeaseEntry:
+ idle_timeout: float
+ status: ExecutionLeaseStatus = "waiting"
+ last_activity_at: float | None = None
+ busy: bool = False
+ has_run_since_grant: bool = False
+ protect_next_grant: bool = False
+
+
+class ExecutionLeaseManager:
+ """Grant one model-execution lease and yield it at chunk boundaries."""
+
+ def __init__(self) -> None:
+ self._entries: dict[str, _ExecutionLeaseEntry] = {}
+ self._waiting: deque[str] = deque()
+ self._active_session_id: str | None = None
+ self._closed = False
+ self._condition = threading.Condition(threading.RLock())
+
+ def register(self, session_id: str, *, idle_timeout: float) -> None:
+ """Register a session that will request execution on its first control."""
+ if idle_timeout <= 0:
+ raise ValueError("idle_timeout must be positive")
+ with self._condition:
+ if self._closed:
+ raise RuntimeError("Execution lease manager is closed")
+ if session_id in self._entries:
+ raise ValueError(f"Execution lease session {session_id!r} already exists")
+ self._entries[session_id] = _ExecutionLeaseEntry(idle_timeout=float(idle_timeout))
+
+ def record_activity(
+ self,
+ session_id: str,
+ *,
+ now: float | None = None,
+ ) -> tuple[ExecutionLeaseTransition, ...]:
+ """Renew an active lease or queue a parked session for execution."""
+ observed_at = time.monotonic() if now is None else now
+ with self._condition:
+ entry = self._require_entry(session_id)
+ if entry.status == "closing":
+ return ()
+ entry.last_activity_at = observed_at
+ transitions: list[ExecutionLeaseTransition] = []
+ if entry.status in {"waiting", "parked"}:
+ entry.protect_next_grant = self._active_session_id is not None
+ entry.status = "queued"
+ if session_id not in self._waiting:
+ self._waiting.append(session_id)
+ transitions.append(ExecutionLeaseTransition(session_id, "queued"))
+
+ active_id = self._active_session_id
+ if active_id is not None and active_id != session_id:
+ active = self._entries[active_id]
+ if active.has_run_since_grant and not active.busy and self._is_expired(active, observed_at):
+ transitions.extend(self._park_active_locked())
+ transitions.extend(self._grant_next_locked())
+ self._condition.notify_all()
+ return tuple(transitions)
+
+ def wait_for_turn(self, session_id: str) -> bool:
+ """Block until ``session_id`` owns the lease or begins closing."""
+ with self._condition:
+ while True:
+ entry = self._entries.get(session_id)
+ if entry is None or entry.status == "closing" or self._closed:
+ return False
+ if self._active_session_id == session_id and entry.status == "active":
+ return True
+ self._condition.wait()
+
+ def begin_chunk(self, session_id: str) -> bool:
+ """Reserve the active lease until the submitted chunk completes."""
+ with self._condition:
+ entry = self._entries.get(session_id)
+ if entry is None or entry.status != "active" or self._active_session_id != session_id or entry.busy:
+ return False
+ entry.busy = True
+ entry.has_run_since_grant = True
+ return True
+
+ def finish_chunk(
+ self,
+ session_id: str,
+ *,
+ now: float | None = None,
+ ) -> tuple[ExecutionLeaseTransition, ...]:
+ """Release a chunk boundary and yield an expired lease when demand exists."""
+ observed_at = time.monotonic() if now is None else now
+ with self._condition:
+ entry = self._require_entry(session_id)
+ if not entry.busy:
+ raise RuntimeError(f"Execution lease session {session_id!r} has no chunk in flight")
+ entry.busy = False
+ transitions: list[ExecutionLeaseTransition] = []
+ if entry.status == "active" and self._waiting and self._is_expired(entry, observed_at):
+ transitions.extend(self._park_active_locked())
+ transitions.extend(self._grant_next_locked())
+ self._condition.notify_all()
+ return tuple(transitions)
+
+ def yield_if_idle(
+ self,
+ session_id: str,
+ *,
+ now: float | None = None,
+ ) -> tuple[ExecutionLeaseTransition, ...]:
+ """Yield an idle lease when demand arrived before the timeout elapsed."""
+ observed_at = time.monotonic() if now is None else now
+ with self._condition:
+ entry = self._require_entry(session_id)
+ transitions: list[ExecutionLeaseTransition] = []
+ if (
+ self._active_session_id == session_id
+ and entry.status == "active"
+ and entry.has_run_since_grant
+ and not entry.busy
+ and self._waiting
+ and self._is_expired(entry, observed_at)
+ ):
+ transitions.extend(self._park_active_locked())
+ transitions.extend(self._grant_next_locked())
+ self._condition.notify_all()
+ return tuple(transitions)
+
+ def abort_chunk(self, session_id: str) -> None:
+ """Clear an in-flight reservation after submission fails."""
+ with self._condition:
+ entry = self._entries.get(session_id)
+ if entry is not None:
+ entry.busy = False
+ self._condition.notify_all()
+
+ def deactivate(self, session_id: str) -> None:
+ """Stop a session from acquiring more work while cleanup drains."""
+ with self._condition:
+ entry = self._entries.get(session_id)
+ if entry is None:
+ return
+ entry.status = "closing"
+ self._remove_waiter_locked(session_id)
+ self._condition.notify_all()
+
+ def release(self, session_id: str) -> tuple[ExecutionLeaseTransition, ...]:
+ """Forget a cleaned session and grant any newly available lease."""
+ with self._condition:
+ entry = self._entries.pop(session_id, None)
+ if entry is None:
+ return ()
+ self._remove_waiter_locked(session_id)
+ if self._active_session_id == session_id:
+ self._active_session_id = None
+ transitions = tuple(self._grant_next_locked())
+ self._condition.notify_all()
+ return transitions
+
+ def snapshot(self, session_id: str) -> ExecutionLeaseSnapshot:
+ """Return the current lease state for diagnostics and tests."""
+ with self._condition:
+ entry = self._require_entry(session_id)
+ queue_position = None
+ if session_id in self._waiting:
+ queue_position = tuple(self._waiting).index(session_id) + 1
+ return ExecutionLeaseSnapshot(
+ session_id=session_id,
+ status=entry.status,
+ last_activity_at=entry.last_activity_at,
+ busy=entry.busy,
+ queue_position=queue_position,
+ )
+
+ def close(self) -> None:
+ """Wake every waiter and reject subsequent registrations."""
+ with self._condition:
+ self._closed = True
+ self._waiting.clear()
+ for entry in self._entries.values():
+ entry.status = "closing"
+ self._condition.notify_all()
+
+ def _grant_next_locked(self) -> list[ExecutionLeaseTransition]:
+ if self._active_session_id is not None:
+ return []
+ while self._waiting:
+ session_id = self._waiting.popleft()
+ entry = self._entries.get(session_id)
+ if entry is None or entry.status != "queued":
+ continue
+ entry.status = "active"
+ entry.has_run_since_grant = not entry.protect_next_grant
+ entry.protect_next_grant = False
+ self._active_session_id = session_id
+ return [ExecutionLeaseTransition(session_id, "active")]
+ return []
+
+ def _park_active_locked(self) -> list[ExecutionLeaseTransition]:
+ session_id = self._active_session_id
+ if session_id is None:
+ return []
+ entry = self._entries[session_id]
+ if entry.busy:
+ return []
+ entry.status = "parked"
+ self._active_session_id = None
+ return [ExecutionLeaseTransition(session_id, "parked")]
+
+ @staticmethod
+ def _is_expired(entry: _ExecutionLeaseEntry, now: float) -> bool:
+ return entry.last_activity_at is not None and now - entry.last_activity_at >= entry.idle_timeout
+
+ def _remove_waiter_locked(self, session_id: str) -> None:
+ try:
+ self._waiting.remove(session_id)
+ except ValueError:
+ pass
+
+ def _require_entry(self, session_id: str) -> _ExecutionLeaseEntry:
+ try:
+ return self._entries[session_id]
+ except KeyError as exc:
+ raise KeyError(f"Unknown execution lease session {session_id!r}") from exc
diff --git a/telefuser/pipelines/lingbot_world_fast/service.py b/telefuser/pipelines/lingbot_world_fast/service.py
index 21c0309..f30bfbc 100644
--- a/telefuser/pipelines/lingbot_world_fast/service.py
+++ b/telefuser/pipelines/lingbot_world_fast/service.py
@@ -23,6 +23,7 @@
from telefuser.utils.profiler import ProfilingContext4Debug
from .control import LingBotWorldFastControlBuilder, LingBotWorldFastControlContext
+from .lease import ExecutionLeaseManager, ExecutionLeaseTransition
from .pipeline import LingBotWorldFastPipeline
from .session import (
LingBotWorldFastDirectionCommand,
@@ -66,7 +67,7 @@
_VIDEO_OUTPUT_TYPES = frozenset({"chunk", "preview"})
_TERMINAL_OUTPUT_TYPES = frozenset({"done", "error"})
_MAX_INPUT_IMAGE_BYTES = 10 * 1024 * 1024
-_CONTROL_PREFETCH_DEPTH = 1
+_CONTROL_PREFETCH_DEPTH = 0
class LingBotWorldFastService:
@@ -94,6 +95,8 @@ def __init__(
self.output_queue_size = int(output_queue_size)
self.close_timeout = float(close_timeout)
self._sessions: dict[str, LingBotWorldFastSessionState] = {}
+ self._sessions_lock = threading.RLock()
+ self._lease_manager = ExecutionLeaseManager()
def start(self) -> None:
self.pipeline.warmup(self._warmup_session_config())
@@ -121,12 +124,16 @@ def _warmup_session_config(self) -> LingBotWorldFastSessionConfig:
)
def stop(self) -> None:
- for session_id in list(self._sessions.keys()):
+ with self._sessions_lock:
+ session_ids = list(self._sessions)
+ for session_id in session_ids:
self.close_session(session_id)
+ self._lease_manager.close()
self.pipeline.close()
def has_session(self, session_id: str) -> bool:
- return session_id in self._sessions
+ with self._sessions_lock:
+ return session_id in self._sessions
@staticmethod
def _load_image(config: dict) -> Image.Image:
@@ -180,13 +187,11 @@ def _frame_num_for_duration(max_duration_seconds: float, fps: int, chunk_size: i
return 4 * (latent_frames - 1) + 1
def create_session(self, config: dict) -> str:
- for stale_session_id, stale_state in list(self._sessions.items()):
+ with self._sessions_lock:
+ existing_sessions = list(self._sessions.items())
+ for stale_session_id, stale_state in existing_sessions:
if not stale_state.active:
self.close_session(stale_session_id)
- if self._sessions:
- raise RuntimeError(
- "LingBotWorldFastService supports one active session at a time; stop it before reconnecting"
- )
defaults = self.default_session_config
session_id = config.get("session_id") or str(uuid.uuid4())
@@ -220,6 +225,9 @@ def create_session(self, config: dict) -> str:
raise ValueError(f"max_duration_seconds must be positive, got {max_duration_seconds}")
if max_duration_seconds > self.max_generation_seconds:
raise ValueError(f"max_duration_seconds must not exceed {self.max_generation_seconds:g}")
+ control_idle_timeout = float(config.get("control_idle_timeout", defaults.get("control_idle_timeout", 10.0)))
+ if control_idle_timeout <= 0:
+ raise ValueError(f"control_idle_timeout must be positive, got {control_idle_timeout}")
frame_policy = str(config.get("frame_policy", defaults.get("frame_policy", "truncate")))
requested_frame_num = config.get("frame_num")
@@ -277,6 +285,7 @@ def create_session(self, config: dict) -> str:
),
show_control_hud=bool(config.get("show_control_hud", defaults.get("show_control_hud", True))),
benchmark_metrics=bool(config.get("benchmark_metrics", defaults.get("benchmark_metrics", False))),
+ control_idle_timeout=control_idle_timeout,
)
control_context = self.pipeline.control_context(session_config)
state = LingBotWorldFastSessionState(
@@ -284,10 +293,61 @@ def create_session(self, config: dict) -> str:
control_context=control_context,
output_queue=asyncio.Queue(maxsize=self.output_queue_size),
)
- self._sessions[session_id] = state
+ with self._sessions_lock:
+ if session_id in self._sessions:
+ raise ValueError(f"LingBotWorld session {session_id!r} already exists")
+ self._sessions[session_id] = state
+ try:
+ self._lease_manager.register(session_id, idle_timeout=control_idle_timeout)
+ except BaseException:
+ with self._sessions_lock:
+ self._sessions.pop(session_id, None)
+ raise
logger.info(f"LingBotWorld session created: {session_id}")
return session_id
+ def _session_state(self, session_id: str) -> LingBotWorldFastSessionState | None:
+ with self._sessions_lock:
+ return self._sessions.get(session_id)
+
+ def _ensure_execution_lease(
+ self,
+ session_id: str,
+ state: LingBotWorldFastSessionState,
+ *,
+ activate: bool = False,
+ ) -> None:
+ try:
+ self._lease_manager.snapshot(session_id)
+ except KeyError:
+ self._lease_manager.register(session_id, idle_timeout=state.config.control_idle_timeout)
+ if activate:
+ transitions = self._lease_manager.record_activity(session_id, now=0.0)
+ self._publish_lease_transitions(transitions)
+
+ def _publish_lease_transitions(self, transitions: tuple[ExecutionLeaseTransition, ...]) -> None:
+ stage_by_status = {
+ "queued": "lease_queued",
+ "active": "lease_granted",
+ "parked": "lease_parked",
+ }
+ for transition in transitions:
+ stage = stage_by_status.get(transition.status)
+ state = self._session_state(transition.session_id)
+ if stage is None or state is None:
+ continue
+ payload: dict[str, object] = {
+ "type": "status",
+ "stage": stage,
+ "timestamp": time.time(),
+ }
+ if transition.status == "queued":
+ try:
+ payload["queue_position"] = self._lease_manager.snapshot(transition.session_id).queue_position
+ except KeyError:
+ continue
+ self._put_output(state, payload)
+
@staticmethod
def _put_output(state: LingBotWorldFastSessionState, payload: dict) -> None:
if state.output_queue is None or state.loop is None:
@@ -810,6 +870,7 @@ def _next_realtime_control(
chunk_index: int,
emit_status: Callable[..., None],
block: bool,
+ idle_callback: Callable[[], None] | None = None,
) -> tuple[object, list[str] | None, float | None] | None:
"""Select one queued tap or snapshot a direction that remains held."""
while state.active:
@@ -839,8 +900,14 @@ def _next_realtime_control(
if not block:
return None
try:
- incoming = state.pending_inputs.get(block=True)
- except queue.Empty: # pragma: no cover - Queue.get blocks here
+ incoming = state.pending_inputs.get(
+ block=True,
+ timeout=0.25 if idle_callback is not None else None,
+ )
+ except queue.Empty:
+ if idle_callback is not None:
+ idle_callback()
+ continue
return None
if incoming.get("type") == "stop":
state.active = False
@@ -913,11 +980,45 @@ def _run_actor_worker_loop(
control_context: LingBotWorldFastControlContext,
control_builder: LingBotWorldFastControlBuilder,
emit_status: Callable[..., None],
+ *,
+ session_id: str | None = None,
) -> None:
"""Drive dynamic control ingress and ordered output through the shared actor graph."""
+ if session_id is None:
+ with self._sessions_lock:
+ session_id = next(
+ (candidate for candidate, current in self._sessions.items() if current is state),
+ f"direct-{id(state)}",
+ )
+ self._ensure_execution_lease(session_id, state, activate=True)
+
+ def yield_idle_lease() -> None:
+ transitions = self._lease_manager.yield_if_idle(session_id)
+ self._publish_lease_transitions(transitions)
+
+ first_item = self._next_realtime_control(
+ state,
+ control_context,
+ control_builder,
+ 0,
+ emit_status,
+ block=True,
+ idle_callback=yield_idle_lease,
+ )
+ if first_item is None:
+ return
+ if not self._lease_manager.wait_for_turn(session_id) or not state.active:
+ return
+ if not self._lease_manager.begin_chunk(session_id):
+ raise RuntimeError("LingBot execution lease changed before runtime initialization")
+
runtime_measurement = self._start_benchmark_measurement(state)
try:
- runtime = self.pipeline._create_initialized_session(state.config, progress_callback=emit_status)
+ try:
+ runtime = self.pipeline._create_initialized_session(state.config, progress_callback=emit_status)
+ except BaseException:
+ self._lease_manager.abort_chunk(session_id)
+ raise
finally:
runtime_facts = self._finish_benchmark_measurement(runtime_measurement)
state.generation_session = runtime
@@ -938,9 +1039,6 @@ def _run_actor_worker_loop(
runtime=self._runtime_metadata(runtime),
**({"measurement": {"name": "runtime_creation", **runtime_facts}} if runtime_facts is not None else {}),
)
- first_item = self._next_realtime_control(state, control_context, control_builder, 0, emit_status, block=True)
- if first_item is None:
- return
submitted = 0
controls_by_chunk: dict[int, list[str] | None] = {}
@@ -952,14 +1050,28 @@ def raise_scheduler_error() -> None:
if error is not None:
raise RuntimeError("LingBot streaming scheduler failed") from error
- def submit_chunk(item: tuple[object, list[str] | None, float | None]) -> None:
+ def submit_chunk(
+ item: tuple[object, list[str] | None, float | None],
+ *,
+ lease_reserved: bool = False,
+ ) -> None:
nonlocal submitted
+ if not lease_reserved:
+ if not self._lease_manager.wait_for_turn(session_id) or not state.active:
+ return
+ if not self._lease_manager.begin_chunk(session_id):
+ raise RuntimeError("LingBot execution lease changed before chunk submission")
deferred_control, applied_controls, control_received_at = item
- control = self.pipeline._resolve_control(deferred_control)
- self.pipeline._validate_control(runtime, control)
+ try:
+ control = self.pipeline._resolve_control(deferred_control)
+ self.pipeline._validate_control(runtime, control)
+ except BaseException:
+ self._lease_manager.abort_chunk(session_id)
+ raise
chunk_measurement = self._start_benchmark_measurement(state)
if not streaming_runtime.try_submit_chunk(streaming_session, submitted, control):
self._finish_benchmark_measurement(chunk_measurement)
+ self._lease_manager.abort_chunk(session_id)
raise RuntimeError("LingBot streaming ingress became unavailable after capacity check")
controls_by_chunk[submitted] = applied_controls
control_received_at_by_chunk[submitted] = control_received_at
@@ -975,7 +1087,7 @@ def submit_chunk(item: tuple[object, list[str] | None, float | None]) -> None:
state.chunk_started_at_monotonic[submitted] = time.monotonic()
submitted += 1
- submit_chunk(first_item)
+ submit_chunk(first_item, lease_reserved=True)
while state.active and runtime.current_chunk_index < runtime.chunk_count:
raise_scheduler_error()
outputs = streaming_runtime.poll_frames(streaming_session)
@@ -1053,6 +1165,12 @@ def submit_chunk(item: tuple[object, list[str] | None, float | None]) -> None:
else {}
),
)
+ if not streaming_runtime.wait_until_idle(streaming_session, timeout=self.close_timeout):
+ raise TimeoutError(
+ f"Timed out waiting for LingBot background work before yielding session {session_id!r}"
+ )
+ transitions = self._lease_manager.finish_chunk(session_id)
+ self._publish_lease_transitions(transitions)
admitted_prefetch = False
while (
@@ -1084,6 +1202,7 @@ def submit_chunk(item: tuple[object, list[str] | None, float | None]) -> None:
submitted,
emit_status,
block=True,
+ idle_callback=yield_idle_lease,
)
if item is None:
break
@@ -1094,7 +1213,7 @@ def submit_chunk(item: tuple[object, list[str] | None, float | None]) -> None:
streaming_runtime.wait_until_idle(streaming_session, timeout=0.05)
def _worker_loop(self, session_id: str) -> None:
- state = self._sessions.get(session_id)
+ state = self._session_state(session_id)
if state is None or state.output_queue is None or state.loop is None:
return
@@ -1123,7 +1242,7 @@ def _run_worker_loop(
self._emit_preview_frame(state)
control_context = state.control_context or self.pipeline.control_context(state.config)
control_builder = LingBotWorldFastControlBuilder(control_context)
- self._run_actor_worker_loop(state, control_context, control_builder, emit_status)
+ self._run_actor_worker_loop(state, control_context, control_builder, emit_status, session_id=session_id)
except Exception as exc:
logger.exception(f"LingBotWorld worker failed: session={session_id}, error={exc}")
self._put_output(
@@ -1137,6 +1256,7 @@ def _run_worker_loop(
)
finally:
state.active = False
+ self._lease_manager.deactivate(session_id)
try:
self._release_generation_session(state)
except Exception as exc:
@@ -1151,6 +1271,8 @@ def _run_worker_loop(
},
)
finally:
+ transitions = self._lease_manager.release(session_id)
+ self._publish_lease_transitions(transitions)
self._put_output(
state,
{
@@ -1161,8 +1283,9 @@ def _run_worker_loop(
},
)
self._put_output(state, {"type": "done"})
- if self._sessions.get(session_id) is state:
- self._sessions.pop(session_id, None)
+ with self._sessions_lock:
+ if self._sessions.get(session_id) is state:
+ self._sessions.pop(session_id, None)
def _runtime_metadata(self, runtime: LingBotWorldFastGenerationSession) -> dict[str, int]:
return {
@@ -1178,10 +1301,18 @@ def _runtime_metadata(self, runtime: LingBotWorldFastGenerationSession) -> dict[
}
def push_chunk(self, session_id: str, chunk: dict) -> None:
- state = self._sessions.get(session_id)
+ state = self._session_state(session_id)
if state is None or not state.active:
return
+ if chunk.get("type") == "stop":
+ state.active = False
+ self._lease_manager.deactivate(session_id)
+ self._stop_control_worker(state)
+ return
+ self._ensure_execution_lease(session_id, state)
received_at_monotonic = time.monotonic()
+ transitions = self._lease_manager.record_activity(session_id, now=received_at_monotonic)
+ self._publish_lease_transitions(transitions)
is_direction_action = chunk.get("type") in {"control", "control_state"} and self._update_direction_controls(
state,
chunk,
@@ -1199,7 +1330,7 @@ def push_chunk(self, session_id: str, chunk: dict) -> None:
self._wake_control_worker(state)
async def pull_chunks(self, session_id: str) -> AsyncGenerator[dict, None]:
- state = self._sessions.get(session_id)
+ state = self._session_state(session_id)
if state is None or state.output_queue is None:
return
@@ -1231,10 +1362,11 @@ def close_session(self, session_id: str, timeout: float | None = None) -> None:
effective_timeout = self.close_timeout if timeout is None else timeout
if effective_timeout <= 0:
raise ValueError(f"timeout must be positive, got {effective_timeout}")
- state = self._sessions.get(session_id)
+ state = self._session_state(session_id)
if state is None:
return
state.active = False
+ self._lease_manager.deactivate(session_id)
self._stop_control_worker(state)
worker = state.worker_thread
if worker is not None and worker.is_alive() and worker is not threading.current_thread():
@@ -1246,5 +1378,9 @@ def close_session(self, session_id: str, timeout: float | None = None) -> None:
return
if worker is None or not worker.is_alive():
self._release_generation_session(state)
- self._sessions.pop(session_id, None)
+ transitions = self._lease_manager.release(session_id)
+ self._publish_lease_transitions(transitions)
+ with self._sessions_lock:
+ if self._sessions.get(session_id) is state:
+ self._sessions.pop(session_id, None)
logger.info(f"LingBotWorld session closed: {session_id}")
diff --git a/telefuser/pipelines/lingbot_world_fast/session.py b/telefuser/pipelines/lingbot_world_fast/session.py
index da38e82..ccbe3f4 100644
--- a/telefuser/pipelines/lingbot_world_fast/session.py
+++ b/telefuser/pipelines/lingbot_world_fast/session.py
@@ -64,6 +64,7 @@ class LingBotWorldFastSessionConfig:
control_pitch_limit_degrees: float = 85.0
show_control_hud: bool = True
benchmark_metrics: bool = False
+ control_idle_timeout: float = 10.0
class LingBotWorldFastSessionStatus(str, Enum):
diff --git a/telefuser/service/livekit/config.py b/telefuser/service/livekit/config.py
index 58d072f..6a3db25 100644
--- a/telefuser/service/livekit/config.py
+++ b/telefuser/service/livekit/config.py
@@ -25,6 +25,12 @@ class LiveKitServeConfig(BaseSettings):
livekit_api_secret: str = Field(default="", description="LiveKit API secret")
num_workers: int = Field(default=1, ge=1, le=64, description="Number of TeleFuser LiveKit workers")
+ max_sessions_per_worker: int = Field(
+ default=1,
+ ge=1,
+ le=64,
+ description="Maximum retained sessions per model worker",
+ )
worker_gpu_map: str | None = Field(
default=None,
description="Semicolon-separated worker GPU groups, for example '0,1;2,3'",
@@ -35,6 +41,11 @@ class LiveKitServeConfig(BaseSettings):
)
queue_size: int = Field(default=0, ge=0, le=10000, description="Maximum queued sessions")
+ control_idle_timeout: float = Field(
+ default=10.0,
+ gt=0,
+ description="Seconds without control activity before a LingBot execution lease may yield",
+ )
session_timeout: int = Field(default=1800, ge=1, description="Maximum session lifetime in seconds")
token_ttl: int = Field(default=3600, ge=1, description="LiveKit token TTL in seconds")
controller_timeout: int = Field(
diff --git a/telefuser/service/livekit/main.py b/telefuser/service/livekit/main.py
index 4470cad..8a99ab8 100644
--- a/telefuser/service/livekit/main.py
+++ b/telefuser/service/livekit/main.py
@@ -25,8 +25,10 @@ def run_stream_server(
livekit_api_key: str | None = None,
livekit_api_secret: str | None = None,
num_workers: int | None = None,
+ max_sessions_per_worker: int | None = None,
worker_gpu_map: str | None = None,
queue_size: int | None = None,
+ control_idle_timeout: float | None = None,
session_timeout: int | None = None,
token_ttl: int | None = None,
controller_timeout: int | None = None,
@@ -44,8 +46,10 @@ def run_stream_server(
"livekit_api_key": livekit_api_key,
"livekit_api_secret": livekit_api_secret,
"num_workers": num_workers,
+ "max_sessions_per_worker": max_sessions_per_worker,
"worker_gpu_map": worker_gpu_map,
"queue_size": queue_size,
+ "control_idle_timeout": control_idle_timeout,
"session_timeout": session_timeout,
"token_ttl": token_ttl,
"controller_timeout": controller_timeout,
diff --git a/telefuser/service/livekit/multi_session_worker.py b/telefuser/service/livekit/multi_session_worker.py
new file mode 100644
index 0000000..d69d0eb
--- /dev/null
+++ b/telefuser/service/livekit/multi_session_worker.py
@@ -0,0 +1,140 @@
+"""Model worker that multiplexes retained LiveKit sessions over one pipeline."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+from telefuser.service.core.stream_pipeline_service import STREAM_MODE_BIDIRECTIONAL
+
+from .config import LiveKitServeConfig
+from .pipeline_adapter import LiveKitPipelineAdapter
+from .room_client import LiveKitRoomClient, RoomClient
+from .schemas import SessionStatus
+from .session_registry import SessionRecord
+from .token_service import LiveKitTokenService
+from .worker import LiveKitWorker as LiveKitSessionRunner
+from .worker import NullWorkerEventSink, WorkerEventSink
+
+
+class _SessionWorkerEventSink:
+ """Bind worker-level callbacks from one runner to its retained session."""
+
+ def __init__(self, owner: MultiSessionLiveKitWorker, session_id: str) -> None:
+ self._owner = owner
+ self._session_id = session_id
+
+ def on_worker_status(self, worker_id: str, status: str) -> None:
+ del worker_id
+ self._owner._on_session_worker_status(self._session_id, status)
+
+ def on_session_status(self, session_id: str, status: SessionStatus, error: str | None = None) -> None:
+ self._owner.event_sink.on_session_status(session_id, status, error)
+
+ def on_pipeline_session(self, session_id: str, pipeline_session_id: str) -> None:
+ self._owner.event_sink.on_pipeline_session(session_id, pipeline_session_id)
+
+ def on_session_finished(self, worker_id: str, session_id: str, error: str | None = None) -> None:
+ self._owner.event_sink.on_session_finished(worker_id, session_id, error)
+
+
+class MultiSessionLiveKitWorker:
+ """Load one model pipeline and retain multiple independent room sessions."""
+
+ def __init__(
+ self,
+ *,
+ worker_id: str,
+ config: LiveKitServeConfig,
+ pipeline_file: str,
+ token_service: LiveKitTokenService,
+ event_sink: WorkerEventSink | None = None,
+ pipeline_adapter: LiveKitPipelineAdapter | None = None,
+ room_client_factory: Callable[[], RoomClient] | None = None,
+ gpu_num: int = 1,
+ ) -> None:
+ self.worker_id = worker_id
+ self.config = config
+ self.pipeline_file = pipeline_file
+ self.token_service = token_service
+ self.event_sink = event_sink or NullWorkerEventSink()
+ self.pipeline_adapter = pipeline_adapter or LiveKitPipelineAdapter()
+ self.room_client_factory = room_client_factory or LiveKitRoomClient
+ self.gpu_num = gpu_num
+ self._sessions: dict[str, LiveKitSessionRunner] = {}
+ self._session_worker_statuses: dict[str, str] = {}
+ self._started = False
+
+ async def start(self, *, skip_validation: bool = False) -> None:
+ """Load the shared pipeline exactly once."""
+ if self._started:
+ return
+ self.pipeline_adapter.start(
+ self.pipeline_file,
+ skip_validation=skip_validation,
+ gpu_num=self.gpu_num,
+ )
+ if self.config.max_sessions_per_worker > 1 and self.pipeline_adapter.stream_mode != STREAM_MODE_BIDIRECTIONAL:
+ await self.pipeline_adapter.aclose()
+ raise RuntimeError("Multiple retained sessions require a BidirectionalService pipeline")
+ self._started = True
+ self.event_sink.on_worker_status(self.worker_id, "idle")
+
+ async def run_session(self, record: SessionRecord) -> None:
+ """Run one room session against the shared pipeline adapter."""
+ if not self._started:
+ raise RuntimeError(f"Worker {self.worker_id} is not started")
+ if record.session_id in self._sessions:
+ raise RuntimeError(f"Session {record.session_id} is already running")
+ if len(self._sessions) >= self.config.max_sessions_per_worker:
+ raise RuntimeError(f"Worker {self.worker_id} retained-session capacity is full")
+
+ self._session_worker_statuses[record.session_id] = "assigned"
+ runner = LiveKitSessionRunner(
+ worker_id=self.worker_id,
+ config=self.config,
+ pipeline_file=self.pipeline_file,
+ token_service=self.token_service,
+ event_sink=_SessionWorkerEventSink(self, record.session_id),
+ pipeline_adapter=self.pipeline_adapter,
+ room_client=self.room_client_factory(),
+ gpu_num=self.gpu_num,
+ )
+ self._sessions[record.session_id] = runner
+ try:
+ await runner.run_session(record)
+ finally:
+ self._sessions.pop(record.session_id, None)
+ self._session_worker_statuses.pop(record.session_id, None)
+ self._publish_aggregate_worker_status()
+
+ async def stop_session(self, session_id: str) -> None:
+ """Request one retained session to stop without affecting its peers."""
+ runner = self._sessions.get(session_id)
+ if runner is not None:
+ await runner.stop_session(session_id)
+
+ async def stop(self) -> None:
+ """Stop admission and close the shared pipeline after sessions drain."""
+ for session_id in tuple(self._sessions):
+ await self.stop_session(session_id)
+ self._started = False
+ await self.pipeline_adapter.aclose()
+ self.event_sink.on_worker_status(self.worker_id, "stopped")
+
+ def _on_session_worker_status(self, session_id: str, status: str) -> None:
+ if session_id not in self._session_worker_statuses:
+ return
+ self._session_worker_statuses[session_id] = status
+ self._publish_aggregate_worker_status()
+
+ def _publish_aggregate_worker_status(self) -> None:
+ statuses = set(self._session_worker_statuses.values())
+ aggregate = next(
+ (
+ status
+ for status in ("running", "draining", "starting_pipeline", "joining_room", "assigned", "starting")
+ if status in statuses
+ ),
+ "idle",
+ )
+ self.event_sink.on_worker_status(self.worker_id, aggregate)
diff --git a/telefuser/service/livekit/runtime.py b/telefuser/service/livekit/runtime.py
index f53e764..0c27cc1 100644
--- a/telefuser/service/livekit/runtime.py
+++ b/telefuser/service/livekit/runtime.py
@@ -8,6 +8,7 @@
from telefuser.service.security.security_validator import SecurityLevel
from .config import LiveKitServeConfig
+from .multi_session_worker import MultiSessionLiveKitWorker as LiveKitWorker
from .pipeline_adapter import LiveKitPipelineAdapter
from .scheduler import LiveKitScheduler, SchedulerAdmission
from .schemas import (
@@ -20,7 +21,6 @@
)
from .session_registry import TERMINAL_SESSION_STATUSES, SessionRecord, SessionRegistry
from .token_service import LiveKitTokenService
-from .worker import LiveKitWorker
from .worker_pool import InProcessLiveKitWorkerPool, WorkerPool
@@ -55,6 +55,7 @@ def __init__(
num_workers=config.num_workers,
gpu_groups=config.worker_gpu_groups(),
queue_size=config.queue_size,
+ max_sessions_per_worker=config.max_sessions_per_worker,
)
self.token_service = token_service or LiveKitTokenService(
api_key=config.livekit_api_key,
@@ -97,6 +98,7 @@ def create_session(self, request: SessionCreateRequest) -> CreateSessionResult:
room_name = f"tf-world-{session_id}"
session_config = dict(request.config)
session_config["session_id"] = session_id
+ session_config["control_idle_timeout"] = self.config.control_idle_timeout
if request.prompt is not None:
session_config["prompt"] = request.prompt
if request.image_path is not None:
@@ -191,10 +193,10 @@ def health(self) -> LiveKitHealthResponse:
status = "unhealthy"
elif workers_failed:
status = "degraded"
- running_statuses = {"joining_room", "starting_pipeline", "running", "draining"}
+ connected_statuses = {"starting_pipeline", "running", "draining"}
return LiveKitHealthResponse(
status=status,
- livekit_connected=any(worker.status in running_statuses for worker in self.scheduler.workers()),
+ livekit_connected=any(worker.status in connected_statuses for worker in self.scheduler.workers()),
**snapshot,
)
@@ -207,6 +209,8 @@ def metadata(self) -> dict:
"pipeline_file": self.pipeline_file,
"livekit_url": self.config.livekit_url,
"num_workers": self.config.num_workers,
+ "max_sessions_per_worker": self.config.max_sessions_per_worker,
+ "control_idle_timeout": self.config.control_idle_timeout,
"worker_mode": self.config.worker_mode,
"queue_size": self.config.queue_size,
**health.model_dump(),
@@ -265,12 +269,9 @@ def _finish_session(self, session_id: str, *, error: str | None = None) -> Sessi
return record
def _start_queued_session(self, admission: SchedulerAdmission) -> None:
- if admission.worker_id is None:
+ if admission.worker_id is None or admission.session_id is None:
return
- worker_state = next(worker for worker in self.scheduler.workers() if worker.worker_id == admission.worker_id)
- if worker_state.session_id is None:
- return
- session_id = worker_state.session_id
+ session_id = admission.session_id
try:
record = self.registry.assign_worker(session_id, admission.worker_id)
self.worker_pool.start_session(record)
diff --git a/telefuser/service/livekit/scheduler.py b/telefuser/service/livekit/scheduler.py
index 775393b..37beb5a 100644
--- a/telefuser/service/livekit/scheduler.py
+++ b/telefuser/service/livekit/scheduler.py
@@ -30,6 +30,8 @@ class WorkerState(BaseModel):
worker_id: str
status: WorkerStatus
gpu_ids: list[str] = Field(default_factory=list)
+ session_ids: list[str] = Field(default_factory=list)
+ session_capacity: int = 1
session_id: str | None = None
room_name: str | None = None
last_heartbeat_at: float
@@ -41,6 +43,7 @@ class SchedulerAdmission(BaseModel):
status: AdmissionStatus
worker_id: str | None = None
+ session_id: str | None = None
queue_position: int | None = None
reason: str | None = None
@@ -51,13 +54,22 @@ class _QueuedSession(BaseModel):
class LiveKitScheduler:
- """Simple FIFO scheduler with one active session per worker."""
-
- def __init__(self, *, num_workers: int, gpu_groups: list[list[str]] | None = None, queue_size: int = 0) -> None:
+ """FIFO scheduler with bounded retained-session capacity per worker."""
+
+ def __init__(
+ self,
+ *,
+ num_workers: int,
+ gpu_groups: list[list[str]] | None = None,
+ queue_size: int = 0,
+ max_sessions_per_worker: int = 1,
+ ) -> None:
if num_workers < 1:
raise ValueError("num_workers must be >= 1")
if queue_size < 0:
raise ValueError("queue_size must be >= 0")
+ if max_sessions_per_worker < 1:
+ raise ValueError("max_sessions_per_worker must be >= 1")
now = utc_timestamp()
groups = gpu_groups or [[] for _ in range(num_workers)]
@@ -69,25 +81,29 @@ def __init__(self, *, num_workers: int, gpu_groups: list[list[str]] | None = Non
worker_id=f"worker-{idx}",
status="idle",
gpu_ids=list(groups[idx]),
+ session_capacity=max_sessions_per_worker,
last_heartbeat_at=now,
)
for idx in range(num_workers)
}
self._queue_size = queue_size
self._queue: deque[_QueuedSession] = deque()
+ self._room_names: dict[str, str] = {}
self._lock = threading.RLock()
def assign(self, *, session_id: str, room_name: str) -> SchedulerAdmission:
- """Assign an idle worker or enqueue/reject the session."""
+ """Assign a worker with capacity or enqueue/reject the session."""
with self._lock:
- worker = self._first_idle_worker()
+ worker = self._first_available_worker()
if worker is not None:
worker.status = "assigned"
+ worker.session_ids.append(session_id)
+ self._room_names[session_id] = room_name
worker.session_id = session_id
worker.room_name = room_name
worker.error = None
worker.last_heartbeat_at = utc_timestamp()
- return SchedulerAdmission(status="assigned", worker_id=worker.worker_id)
+ return SchedulerAdmission(status="assigned", worker_id=worker.worker_id, session_id=session_id)
if self._queue_size == 0:
return SchedulerAdmission(status="rejected", reason="no_idle_worker")
@@ -105,21 +121,27 @@ def release_session(self, session_id: str) -> SchedulerAdmission | None:
self._queue = deque(item for item in self._queue if item.session_id != session_id)
return None
- worker.session_id = None
- worker.room_name = None
- worker.error = None
- worker.status = "idle"
+ worker.session_ids.remove(session_id)
+ self._room_names.pop(session_id, None)
+ worker.session_id = worker.session_ids[-1] if worker.session_ids else None
+ worker.room_name = self._room_names.get(worker.session_id) if worker.session_id is not None else None
+ worker_failed = worker.status in {"failed", "stopped"}
+ if not worker_failed:
+ worker.error = None
+ worker.status = "assigned" if worker.session_ids else "idle"
worker.last_heartbeat_at = utc_timestamp()
- if not self._queue:
+ if not self._queue or worker_failed:
return None
queued = self._queue.popleft()
worker.status = "assigned"
+ worker.session_ids.append(queued.session_id)
+ self._room_names[queued.session_id] = queued.room_name
worker.session_id = queued.session_id
worker.room_name = queued.room_name
worker.last_heartbeat_at = utc_timestamp()
- return SchedulerAdmission(status="assigned", worker_id=worker.worker_id)
+ return SchedulerAdmission(status="assigned", worker_id=worker.worker_id, session_id=queued.session_id)
def update_worker_status(self, worker_id: str, status: WorkerStatus) -> WorkerState:
"""Update a worker lifecycle status."""
@@ -154,23 +176,22 @@ def health_snapshot(self) -> dict[str, int]:
"""Return scheduler capacity counts."""
with self._lock:
workers = list(self._workers.values())
- busy_statuses = {"assigned", "joining_room", "starting_pipeline", "running", "draining"}
return {
"workers_total": len(workers),
- "workers_idle": sum(1 for worker in workers if worker.status == "idle"),
- "workers_busy": sum(1 for worker in workers if worker.status in busy_statuses),
+ "workers_idle": sum(1 for worker in workers if worker.status != "failed" and not worker.session_ids),
+ "workers_busy": sum(1 for worker in workers if bool(worker.session_ids)),
"workers_failed": sum(1 for worker in workers if worker.status == "failed"),
"queued_sessions": len(self._queue),
}
- def _first_idle_worker(self) -> WorkerState | None:
+ def _first_available_worker(self) -> WorkerState | None:
for worker in self._workers.values():
- if worker.status == "idle":
+ if worker.status not in {"failed", "stopped"} and len(worker.session_ids) < worker.session_capacity:
return worker
return None
def _worker_for_session(self, session_id: str) -> WorkerState | None:
for worker in self._workers.values():
- if worker.session_id == session_id:
+ if session_id in worker.session_ids:
return worker
return None
diff --git a/tests/server/conftest.py b/tests/server/conftest.py
index be19f95..79cb1a2 100644
--- a/tests/server/conftest.py
+++ b/tests/server/conftest.py
@@ -16,6 +16,13 @@
os.environ["TELEFUSER_SECURITY_LEVEL"] = "NONE"
+@pytest.fixture(autouse=True)
+def disable_proxy_for_local_server_tests(monkeypatch):
+ """Keep localhost integration requests out of process-level proxies."""
+ for name in ("HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "http_proxy", "https_proxy", "all_proxy"):
+ monkeypatch.delenv(name, raising=False)
+
+
@pytest.fixture(scope="session")
def pipeline_path():
"""Get the path to the fake pipeline."""
diff --git a/tests/unit/pipelines/lingbot_world_fast/test_execution_lease.py b/tests/unit/pipelines/lingbot_world_fast/test_execution_lease.py
new file mode 100644
index 0000000..2de50a5
--- /dev/null
+++ b/tests/unit/pipelines/lingbot_world_fast/test_execution_lease.py
@@ -0,0 +1,120 @@
+from __future__ import annotations
+
+from telefuser.pipelines.lingbot_world_fast.lease import ExecutionLeaseManager
+
+
+def test_execution_lease_yields_expired_busy_session_at_chunk_boundary() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=5.0)
+ manager.register("session-b", idle_timeout=5.0)
+
+ assert [transition.status for transition in manager.record_activity("session-a", now=0.0)] == [
+ "queued",
+ "active",
+ ]
+ assert manager.begin_chunk("session-a") is True
+
+ assert [transition.status for transition in manager.record_activity("session-b", now=6.0)] == ["queued"]
+ assert manager.snapshot("session-a").status == "active"
+
+ transitions = manager.finish_chunk("session-a", now=6.0)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [
+ ("session-a", "parked"),
+ ("session-b", "active"),
+ ]
+ assert manager.snapshot("session-b").status == "active"
+
+
+def test_execution_lease_yields_after_queued_waiter_reaches_idle_deadline() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=5.0)
+ manager.register("session-b", idle_timeout=5.0)
+ manager.record_activity("session-a", now=0.0)
+ manager.record_activity("session-b", now=1.0)
+
+ assert manager.yield_if_idle("session-a", now=4.99) == ()
+
+ transitions = manager.yield_if_idle("session-a", now=5.0)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [
+ ("session-a", "parked"),
+ ("session-b", "active"),
+ ]
+ assert manager.snapshot("session-a").status == "parked"
+ assert manager.snapshot("session-b").status == "active"
+
+
+def test_execution_lease_switches_idle_session_immediately_when_no_chunk_is_running() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=5.0)
+ manager.register("session-b", idle_timeout=5.0)
+ manager.record_activity("session-a", now=0.0)
+
+ transitions = manager.record_activity("session-b", now=5.0)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [
+ ("session-b", "queued"),
+ ("session-a", "parked"),
+ ("session-b", "active"),
+ ]
+
+
+def test_execution_lease_requeues_parked_session_at_fifo_tail() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=1.0)
+ manager.register("session-b", idle_timeout=1.0)
+ manager.record_activity("session-a", now=0.0)
+ manager.begin_chunk("session-a")
+ manager.record_activity("session-b", now=2.0)
+ manager.finish_chunk("session-a", now=2.0)
+
+ transitions = manager.record_activity("session-a", now=3.0)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [("session-a", "queued")]
+ assert manager.snapshot("session-b").status == "active"
+
+ assert manager.begin_chunk("session-b") is True
+ transitions = manager.finish_chunk("session-b", now=3.0)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [
+ ("session-b", "parked"),
+ ("session-a", "active"),
+ ]
+
+
+def test_execution_lease_guarantees_one_chunk_after_a_stale_waiter_is_granted() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=5.0)
+ manager.register("session-b", idle_timeout=5.0)
+ manager.record_activity("session-a", now=0.0)
+ manager.begin_chunk("session-a")
+ manager.record_activity("session-b", now=1.0)
+
+ manager.finish_chunk("session-a", now=10.0)
+ transitions = manager.record_activity("session-a", now=10.1)
+
+ assert [(transition.session_id, transition.status) for transition in transitions] == [("session-a", "queued")]
+ assert manager.snapshot("session-b").status == "active"
+
+ assert manager.begin_chunk("session-b") is True
+ transitions = manager.finish_chunk("session-b", now=10.2)
+ assert [(transition.session_id, transition.status) for transition in transitions] == [
+ ("session-b", "parked"),
+ ("session-a", "active"),
+ ]
+
+
+def test_execution_lease_deactivation_waits_for_explicit_cleanup_release() -> None:
+ manager = ExecutionLeaseManager()
+ manager.register("session-a", idle_timeout=1.0)
+ manager.register("session-b", idle_timeout=1.0)
+ manager.record_activity("session-a", now=0.0)
+ manager.record_activity("session-b", now=2.0)
+ assert manager.snapshot("session-b").status == "active"
+
+ manager.deactivate("session-b")
+ assert manager.snapshot("session-b").status == "closing"
+ assert manager.snapshot("session-a").status == "parked"
+
+ assert manager.release("session-b") == ()
diff --git a/tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py b/tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py
index 170b723..d4c94d8 100644
--- a/tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py
+++ b/tests/unit/pipelines/lingbot_world_fast/test_service_action_loop.py
@@ -83,7 +83,7 @@ def test_actor_worker_submits_control_and_emits_ordered_chunk() -> None:
with (
patch.object(service, "_next_realtime_control", return_value=first_control),
patch.object(service, "_put_output") as put_output,
- patch("telefuser.pipelines.lingbot_world_fast.service.time.monotonic", side_effect=[10.0, 12.0, 13.0]),
+ patch("telefuser.pipelines.lingbot_world_fast.service.time.monotonic", side_effect=[10.0, 12.0, 13.0, 14.0]),
):
service._run_actor_worker_loop(state, state.control_context, control_builder, emit_status)
@@ -91,6 +91,7 @@ def test_actor_worker_submits_control_and_emits_ordered_chunk() -> None:
streaming_runtime.try_submit_chunk.assert_called_once_with(
streaming_session, 0, pipeline._resolve_control.return_value
)
+ streaming_runtime.wait_until_idle.assert_called_once_with(streaming_session, timeout=service.close_timeout)
assert put_output.call_args.args[1]["index"] == 0
assert put_output.call_args.args[1]["frames"][0].size == (8, 8)
assert runtime.current_chunk_index == 1
@@ -165,7 +166,7 @@ def initialize(*_args: object, **_kwargs: object) -> LingBotWorldFastGenerationS
assert state.generation_session is runtime
-def test_actor_worker_prefetches_conditions_before_waiting_for_first_control() -> None:
+def test_actor_worker_waits_for_first_control_before_initializing_runtime() -> None:
pipeline = MagicMock()
runtime = LingBotWorldFastGenerationSession(config=_state().config, latent_f=1, chunk_size=1, cache_handle=7)
pipeline._create_initialized_session.return_value = runtime
@@ -175,20 +176,20 @@ def test_actor_worker_prefetches_conditions_before_waiting_for_first_control() -
service = LingBotWorldFastService(pipeline)
state = _state()
- def stop_after_prefetch(*_args: object, **_kwargs: object) -> None:
- assert pipeline._create_initialized_session.called
- assert streaming_runtime.create_session.called
+ def stop_before_control(*_args: object, **_kwargs: object) -> None:
+ assert not pipeline._create_initialized_session.called
+ assert not streaming_runtime.create_session.called
state.active = False
return None
- with patch.object(service, "_next_realtime_control", side_effect=stop_after_prefetch):
+ with patch.object(service, "_next_realtime_control", side_effect=stop_before_control):
service._run_actor_worker_loop(state, MagicMock(), MagicMock(), MagicMock())
- streaming_runtime.create_session.assert_called_once()
+ streaming_runtime.create_session.assert_not_called()
streaming_runtime.try_submit_chunk.assert_not_called()
-def test_actor_worker_prefetches_only_one_directional_chunk_ahead_of_output() -> None:
+def test_actor_worker_keeps_only_one_directional_chunk_in_flight() -> None:
pipeline = MagicMock()
runtime = LingBotWorldFastGenerationSession(
config=LingBotWorldFastSessionConfig(prompt="test", image=Image.new("RGB", (8, 8)), chunk_size=1),
@@ -221,7 +222,6 @@ def stop_wait(*_args: object, **_kwargs: object) -> bool:
assert streaming_runtime.try_submit_chunk.call_args_list == [
((streaming_session, 0, pipeline._resolve_control.return_value), {}),
- ((streaming_session, 1, pipeline._resolve_control.return_value), {}),
]
@@ -256,6 +256,18 @@ def test_release_stops_control_without_scheduling_stationary_generation() -> Non
assert state.pending_inputs.get_nowait() == {"type": "direction_control"}
+def test_stop_chunk_terminates_the_worker_without_becoming_an_explicit_control() -> None:
+ service = LingBotWorldFastService(MagicMock())
+ state = _state()
+ service._sessions["session-a"] = state
+
+ service.push_chunk("session-a", {"type": "stop"})
+
+ assert state.active is False
+ assert state.latest_explicit_control is None
+ assert state.pending_inputs.get_nowait() == {"type": "stop"}
+
+
def test_held_direction_supplies_a_nonblocking_prefetch_snapshot() -> None:
service = LingBotWorldFastService(MagicMock())
state = _state()
diff --git a/tests/unit/service/livekit/test_cli.py b/tests/unit/service/livekit/test_cli.py
index 20c5066..1495e76 100644
--- a/tests/unit/service/livekit/test_cli.py
+++ b/tests/unit/service/livekit/test_cli.py
@@ -29,6 +29,10 @@ def fake_run_stream_server(**kwargs):
"2",
"--worker-gpu-map",
"0;1",
+ "--max-sessions-per-worker",
+ "4",
+ "--control-idle-timeout",
+ "12.5",
"--queue-size",
"3",
],
@@ -41,5 +45,7 @@ def fake_run_stream_server(**kwargs):
assert captured["livekit_api_secret"] == "secret"
assert captured["num_workers"] == 2
assert captured["worker_gpu_map"] == "0;1"
+ assert captured["max_sessions_per_worker"] == 4
+ assert captured["control_idle_timeout"] == 12.5
assert captured["queue_size"] == 3
assert captured["skip_validation"] is True
diff --git a/tests/unit/service/livekit/test_config.py b/tests/unit/service/livekit/test_config.py
index e0f0627..725b885 100644
--- a/tests/unit/service/livekit/test_config.py
+++ b/tests/unit/service/livekit/test_config.py
@@ -29,3 +29,13 @@ def test_require_livekit_credentials_reports_missing_fields() -> None:
with pytest.raises(ValueError, match="livekit_api_key, livekit_api_secret"):
config.require_livekit_credentials()
+
+
+def test_multi_session_settings_load_from_environment(monkeypatch: pytest.MonkeyPatch) -> None:
+ monkeypatch.setenv("TELEFUSER_LIVEKIT_MAX_SESSIONS_PER_WORKER", "3")
+ monkeypatch.setenv("TELEFUSER_LIVEKIT_CONTROL_IDLE_TIMEOUT", "7.5")
+
+ config = LiveKitServeConfig()
+
+ assert config.max_sessions_per_worker == 3
+ assert config.control_idle_timeout == 7.5
diff --git a/tests/unit/service/livekit/test_demo.py b/tests/unit/service/livekit/test_demo.py
index aa35ac4..c8fce3c 100644
--- a/tests/unit/service/livekit/test_demo.py
+++ b/tests/unit/service/livekit/test_demo.py
@@ -23,6 +23,8 @@ def test_stream_demo_preserves_controls_and_uses_livekit_transport() -> None:
"{ rtcConfig: TURN_RTC_CONFIG }",
"topic: CONTROL_TOPIC",
'type: "control_state"',
+ "CONTROL_HEARTBEAT_MS = 1000",
+ "if (pressedControls.size > 0)",
'event: "reset"',
'event: "reset_pose"',
'type: "stop"',
diff --git a/tests/unit/service/livekit/test_multi_session_capacity.py b/tests/unit/service/livekit/test_multi_session_capacity.py
new file mode 100644
index 0000000..20972e9
--- /dev/null
+++ b/tests/unit/service/livekit/test_multi_session_capacity.py
@@ -0,0 +1,107 @@
+from __future__ import annotations
+
+from telefuser.service.livekit.config import LiveKitServeConfig
+from telefuser.service.livekit.runtime import LiveKitServeRuntime
+from telefuser.service.livekit.scheduler import LiveKitScheduler
+from telefuser.service.livekit.schemas import SessionCreateRequest
+
+
+class _TokenService:
+ def create_token(self, *, identity: str, room_name: str, role: str, **kwargs: object) -> str:
+ return f"{role}:{identity}:{room_name}"
+
+
+class _WorkerPool:
+ def __init__(self) -> None:
+ self.started: list[str] = []
+
+ async def start(self, *, skip_validation: bool = False) -> None:
+ return None
+
+ def start_session(self, record) -> None:
+ self.started.append(record.session_id)
+
+ async def stop_session(self, session_id: str) -> None:
+ return None
+
+ async def aclose(self) -> None:
+ return None
+
+
+def test_scheduler_uses_all_retained_slots_before_queueing() -> None:
+ scheduler = LiveKitScheduler(num_workers=1, max_sessions_per_worker=2, queue_size=1)
+
+ first = scheduler.assign(session_id="session-1", room_name="room-1")
+ second = scheduler.assign(session_id="session-2", room_name="room-2")
+ third = scheduler.assign(session_id="session-3", room_name="room-3")
+
+ assert first.status == second.status == "assigned"
+ assert third.status == "queued"
+ assert scheduler.workers()[0].session_ids == ["session-1", "session-2"]
+
+
+def test_scheduler_releases_one_slot_without_idling_other_sessions() -> None:
+ scheduler = LiveKitScheduler(num_workers=1, max_sessions_per_worker=2, queue_size=1)
+ scheduler.assign(session_id="session-1", room_name="room-1")
+ scheduler.assign(session_id="session-2", room_name="room-2")
+ scheduler.assign(session_id="session-3", room_name="room-3")
+
+ admission = scheduler.release_session("session-1")
+
+ assert admission is not None
+ assert admission.session_id == "session-3"
+ assert scheduler.workers()[0].session_ids == ["session-2", "session-3"]
+ assert scheduler.health_snapshot()["workers_busy"] == 1
+
+
+def test_scheduler_preserves_the_compatibility_room_for_a_remaining_session() -> None:
+ scheduler = LiveKitScheduler(num_workers=1, max_sessions_per_worker=2)
+ scheduler.assign(session_id="session-1", room_name="room-1")
+ scheduler.assign(session_id="session-2", room_name="room-2")
+
+ scheduler.release_session("session-1")
+
+ worker = scheduler.workers()[0]
+ assert worker.session_id == "session-2"
+ assert worker.room_name == "room-2"
+
+
+def test_scheduler_does_not_resurrect_a_failed_worker_when_a_session_finishes() -> None:
+ scheduler = LiveKitScheduler(num_workers=1, max_sessions_per_worker=2, queue_size=1)
+ scheduler.assign(session_id="session-1", room_name="room-1")
+ scheduler.assign(session_id="session-2", room_name="room-2")
+ scheduler.assign(session_id="session-3", room_name="room-3")
+ scheduler.fail_worker("worker-0", "worker failed")
+
+ admission = scheduler.release_session("session-1")
+
+ assert admission is None
+ worker = scheduler.workers()[0]
+ assert worker.status == "failed"
+ assert worker.error == "worker failed"
+ assert worker.session_ids == ["session-2"]
+ assert scheduler.health_snapshot()["queued_sessions"] == 1
+
+
+def test_runtime_starts_two_sessions_on_one_model_worker() -> None:
+ worker_pool = _WorkerPool()
+ runtime = LiveKitServeRuntime(
+ config=LiveKitServeConfig(
+ livekit_url="wss://livekit.example",
+ livekit_api_key="key",
+ livekit_api_secret="secret",
+ max_sessions_per_worker=2,
+ control_idle_timeout=8.0,
+ ),
+ pipeline_file="pipeline.py",
+ token_service=_TokenService(),
+ worker_pool=worker_pool,
+ )
+
+ first = runtime.create_session(SessionCreateRequest(identity="controller-1"))
+ second = runtime.create_session(SessionCreateRequest(identity="controller-2"))
+
+ assert first.admission.status == second.admission.status == "assigned"
+ assert worker_pool.started == [first.record.session_id, second.record.session_id]
+ assert runtime.registry.require(first.record.session_id).config["control_idle_timeout"] == 8.0
+ assert runtime.registry.require(second.record.session_id).worker_id == "worker-0"
diff --git a/tests/unit/service/livekit/test_multi_session_worker.py b/tests/unit/service/livekit/test_multi_session_worker.py
new file mode 100644
index 0000000..6ec9e69
--- /dev/null
+++ b/tests/unit/service/livekit/test_multi_session_worker.py
@@ -0,0 +1,227 @@
+from __future__ import annotations
+
+import asyncio
+import json
+
+import pytest
+
+from telefuser.service.core.stream_pipeline_service import STREAM_MODE_BIDIRECTIONAL, STREAM_MODE_SERVER_PUSH
+from telefuser.service.livekit.config import LiveKitServeConfig
+from telefuser.service.livekit.multi_session_worker import MultiSessionLiveKitWorker
+from telefuser.service.livekit.session_registry import SessionRecord
+
+
+class _TokenService:
+ def create_token(self, *, identity: str, room_name: str, role: str, **kwargs: object) -> str:
+ return f"{role}:{identity}:{room_name}"
+
+
+class _PipelineAdapter:
+ stream_mode = STREAM_MODE_BIDIRECTIONAL
+
+ def __init__(self) -> None:
+ self.start_calls = 0
+ self.closed_service = False
+ self.created: list[str] = []
+ self.pushed: list[tuple[str, dict]] = []
+ self.closed: list[str] = []
+ self.queues: dict[str, asyncio.Queue[dict | None]] = {}
+
+ def start(self, pipeline_file: str, *, skip_validation: bool = False, gpu_num: int = 1) -> None:
+ del pipeline_file, skip_validation, gpu_num
+ self.start_calls += 1
+
+ async def aclose(self) -> None:
+ self.closed_service = True
+
+ def create_session(self, config: dict) -> str:
+ session_id = str(config["session_id"])
+ self.created.append(session_id)
+ self.queues[session_id] = asyncio.Queue()
+ return session_id
+
+ def push_chunk(self, session_id: str, chunk: dict) -> None:
+ self.pushed.append((session_id, chunk))
+
+ async def pull_chunks(self, session_id: str):
+ while True:
+ chunk = await self.queues[session_id].get()
+ if chunk is None:
+ return
+ yield chunk
+
+ def close_session(self, session_id: str) -> None:
+ self.closed.append(session_id)
+
+
+class _RoomClient:
+ def __init__(self) -> None:
+ self.connected = asyncio.Event()
+ self.on_data = None
+ self.statuses: list[dict] = []
+ self.disconnected = False
+
+ async def connect(self, url: str, token: str, on_data) -> None:
+ del url, token
+ self.on_data = on_data
+ self.connected.set()
+
+ async def publish_video_track(self, name: str, width: int, height: int, *, fps: float = 16.0) -> None:
+ return None
+
+ async def publish_video_frame(self, frame_rgb, *, fps: float = 16.0) -> None:
+ return None
+
+ async def publish_audio_frame(self, pcm: bytes, *, sample_rate: int, channels: int) -> None:
+ return None
+
+ async def publish_status(self, payload: dict) -> None:
+ self.statuses.append(payload)
+
+ async def publish_metrics(self, payload: dict) -> None:
+ return None
+
+ async def disconnect(self) -> None:
+ self.disconnected = True
+
+ def emit_control(self, payload: dict, *, identity: str) -> None:
+ assert self.on_data is not None
+ self.on_data(json.dumps(payload), "tf.control", identity)
+
+
+class _EventSink:
+ def __init__(self) -> None:
+ self.worker_statuses: list[str] = []
+
+ def on_worker_status(self, worker_id: str, status: str) -> None:
+ del worker_id
+ self.worker_statuses.append(status)
+
+ def on_session_status(self, session_id: str, status: str, error: str | None = None) -> None:
+ return None
+
+ def on_pipeline_session(self, session_id: str, pipeline_session_id: str) -> None:
+ return None
+
+ def on_session_finished(self, worker_id: str, session_id: str, error: str | None = None) -> None:
+ return None
+
+
+def _record(session_id: str) -> SessionRecord:
+ return SessionRecord(
+ session_id=session_id,
+ room_name=f"room-{session_id}",
+ controller_identity=f"controller-{session_id}",
+ status="assigned",
+ worker_id="worker-0",
+ config={"session_id": session_id},
+ created_at=0,
+ updated_at=0,
+ )
+
+
+async def _wait_for(predicate, *, timeout: float = 1.0) -> None:
+ deadline = asyncio.get_running_loop().time() + timeout
+ while not predicate():
+ if asyncio.get_running_loop().time() >= deadline:
+ raise AssertionError("timed out waiting for condition")
+ await asyncio.sleep(0.01)
+
+
+def test_multi_session_worker_loads_one_pipeline_and_routes_two_rooms() -> None:
+ async def _run() -> None:
+ adapter = _PipelineAdapter()
+ rooms: list[_RoomClient] = []
+
+ def room_factory() -> _RoomClient:
+ room = _RoomClient()
+ rooms.append(room)
+ return room
+
+ worker = MultiSessionLiveKitWorker(
+ worker_id="worker-0",
+ config=LiveKitServeConfig(
+ livekit_url="wss://livekit.example",
+ livekit_api_key="key",
+ livekit_api_secret="secret",
+ max_sessions_per_worker=2,
+ ),
+ pipeline_file="pipeline.py",
+ token_service=_TokenService(),
+ pipeline_adapter=adapter,
+ room_client_factory=room_factory,
+ )
+ await worker.start(skip_validation=True)
+ tasks = [asyncio.create_task(worker.run_session(_record(session_id))) for session_id in ("a", "b")]
+ await _wait_for(lambda: len(adapter.created) == 2 and len(rooms) == 2)
+
+ rooms[0].emit_control({"type": "control_state", "controls": ["w"]}, identity="controller-a")
+ rooms[1].emit_control({"type": "control_state", "controls": ["j"]}, identity="controller-b")
+ await _wait_for(lambda: len(adapter.pushed) == 2)
+ for session_id in ("a", "b"):
+ await adapter.queues[session_id].put(None)
+ await asyncio.gather(*tasks)
+ await worker.stop()
+
+ assert adapter.start_calls == 1
+ assert adapter.created == ["a", "b"]
+ assert adapter.pushed == [
+ ("a", {"type": "control_state", "controls": ["w"]}),
+ ("b", {"type": "control_state", "controls": ["j"]}),
+ ]
+ assert sorted(adapter.closed) == ["a", "b"]
+ assert all(room.disconnected for room in rooms)
+ assert adapter.closed_service is True
+
+ asyncio.run(_run())
+
+
+def test_multi_session_worker_rejects_server_push_capacity_above_one() -> None:
+ async def _run() -> None:
+ adapter = _PipelineAdapter()
+ adapter.stream_mode = STREAM_MODE_SERVER_PUSH
+ worker = MultiSessionLiveKitWorker(
+ worker_id="worker-0",
+ config=LiveKitServeConfig(
+ livekit_url="wss://livekit.example",
+ livekit_api_key="key",
+ livekit_api_secret="secret",
+ max_sessions_per_worker=2,
+ ),
+ pipeline_file="pipeline.py",
+ token_service=_TokenService(),
+ pipeline_adapter=adapter,
+ )
+
+ with pytest.raises(
+ RuntimeError,
+ match="Multiple retained sessions require a BidirectionalService pipeline",
+ ):
+ await worker.start(skip_validation=True)
+
+ assert adapter.closed_service is True
+
+ asyncio.run(_run())
+
+
+def test_multi_session_worker_aggregates_runner_statuses() -> None:
+ event_sink = _EventSink()
+ worker = MultiSessionLiveKitWorker(
+ worker_id="worker-0",
+ config=LiveKitServeConfig(
+ livekit_url="wss://livekit.example",
+ livekit_api_key="key",
+ livekit_api_secret="secret",
+ max_sessions_per_worker=2,
+ ),
+ pipeline_file="pipeline.py",
+ token_service=_TokenService(),
+ pipeline_adapter=_PipelineAdapter(),
+ event_sink=event_sink,
+ )
+ worker._session_worker_statuses.update({"a": "assigned", "b": "assigned"})
+
+ worker._on_session_worker_status("a", "running")
+ worker._on_session_worker_status("b", "joining_room")
+
+ assert event_sink.worker_statuses[-1] == "running"
diff --git a/tests/unit/service/livekit/test_runtime.py b/tests/unit/service/livekit/test_runtime.py
index 809179a..ab8f185 100644
--- a/tests/unit/service/livekit/test_runtime.py
+++ b/tests/unit/service/livekit/test_runtime.py
@@ -87,6 +87,23 @@ def test_runtime_worker_callbacks_release_capacity() -> None:
assert runtime.scheduler.health_snapshot()["workers_idle"] == 1
+def test_runtime_reports_livekit_connected_only_after_room_connection() -> None:
+ config = LiveKitServeConfig(livekit_url="wss://livekit.example", livekit_api_key="key", livekit_api_secret="secret")
+ runtime = LiveKitServeRuntime(
+ config=config,
+ pipeline_file="pipeline.py",
+ token_service=FakeTokenService(),
+ worker_pool=FakeWorkerPool(),
+ )
+ runtime.create_session(SessionCreateRequest(identity="controller-1"))
+
+ assert runtime.health().livekit_connected is False
+
+ runtime.on_worker_status("worker-0", "starting_pipeline")
+
+ assert runtime.health().livekit_connected is True
+
+
def test_runtime_start_and_close_are_idempotent() -> None:
async def _run() -> None:
config = LiveKitServeConfig(