Skip to content

🤖 perf: frontend WebSocket transfers 73 MB and blocks the UI for minutes on large deployments #3960

Description

@ThomasK33

Summary

On a large deployment (246 live workspaces, 1,513 total in config.json, 100+ with sub-agent children), the single /orpc/ws connection transferred 73.34 MB and the UI stayed non-interactive for minutes. Investigated live (probe = headless Chromium loading the real frontend, attributing every WS frame to its oRPC procedure). Full report: docs/perf/websocket-traffic-investigation.md on branch ws-traffic-perf.

Measured on the live deployment (build 93244e0c7):

  • Fresh client boot: 8.6–9.1 MB, time-to-interactive 131–162 s.
  • Idle (no workspace selected): 1.2 MB/min.
  • Workflow workspace open: 5.2 MB/min (85 KB/s) steady state.

Attribution (boot, measured)

bytes frames procedure
4,488,126 4 projects/list (1.12 MB/call — embeds all 1,513 workspaces, 1,267 archived = 84% of payload; re-fetched on every config.onConfigChanged; 8 calls = 9.3 MB in one 4-min session)
2,893,100 1 workspace/activity/list (13,901 entries from never-pruned extensionMetadata.json; server took ~59 s to answer)
882,295 244 workspace/getSessionUsage (one RPC per workspace; getSessionUsageBatch already exists)
214,572 1 workspace/list
< 100 KB everything else (subscription deltas are well-behaved)

Compounding factors: no perMessageDeflate on the WS server (src/node/orpc/server.ts:1628ws defaults off, so all of this is uncompressed JSON), and the <App /> mount gate (AppLoader.tsx:219-276) waits on responses that queue behind this flood on one ordered socket — that's the multi-minute lockout.

Workflow layer: contributing (~14% while viewing a workflow workspace), not primary — WorkflowRunToolCall.tsx:1536-1544 polls the full growing run record (105 KB and counting) every 2 s even when the card is collapsed, and workflows.subscribe pushes the full record per durable write (O(N²) cumulative). Tracked with the anomaly issue below.

Recommended first move

Stop serializing dead workspaces (server-side only, protocol-compatible):

  1. projects.list: exclude archived workspaces (or drop embedded workspace bodies entirely — the frontend gets live metadata from workspace.list).
  2. Prune/tombstone extensionMetadata.json and scope workspaceService.getActivityList() (workspaceService.ts:12521-12599) to config-known live workspaces.

Expected: boot 8.6 MB → ~1.2 MB, idle −>90%, and removal of the ~59 s activity.list server scan that holds the boot gate. Proof-of-value: sandboxed dev server with this deployment's config.json/extensionMetadata.json, re-run the frame-attribution probe before/after (~20-line patch).

Follow-ups ranked in the report: permessage-deflate, getSessionUsageBatch at boot, scoped config-changed events, workflow deltas + poll removal, progressive hydration, transcript replay windowing.

Related: #3959


Generated with xum • Model: anthropic:claude-fable-5 • Thinking: xhigh • Cost: $1.01

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions