Skip to content

mount receive path fetches each file serially, so change-set latency scales with file count #406

Description

@khaliqgant

Summary

The mount's websocket receive path materialises a remote change set by fetching each file with its own server round trip, one at a time. Propagation latency therefore scales with the number of files in a change set rather than with its size in bytes. For the change-set shape this repository's own history actually produces, this is the dominant cost.

Measured evidence: docs/evidence/mount-latency-20260807/ · PR #405

The measurement

Change set n end-to-end median p95
Single small file (~300 B) 20 20.2 ms 161.7 ms
Repo-sized (11 files, ~14 KB) 20 216.7 ms 303.9 ms

Topology precondition for both rows: the relayfile server ran on the sender's own machine, so the sender→server leg was loopback, and the only network hop was server→receiver over a Tailscale LAN between two Macs. These are LAN best-case figures, not measurements of the hosted product path.

Splitting the legs isolates where the cost sits:

Leg Small file Repo-sized
A: sender→server (loopback) median 3.1 ms median 3.7 ms
B: server→receiver mount (LAN) median 15.8 ms median 212.4 ms

Leg B carries 212.4 ms of the 216.7 ms change-set median. Going from 1 file to 11 files is ~47× the bytes but ~10× the latency — the scaling factor tracks file count, not payload size.

Mechanism

On a remote change the daemon receives a websocket event and, per file:

  • applyWebSocketEventinternal/mountsync/syncer.go:3310
  • s.client.ReadFile(...)internal/mountsync/syncer.go:3320 — a synchronous HTTP round trip to the server
  • s.applyRemoteFile(...)internal/mountsync/syncer.go:3339
  • hash-compare and writeFileAtomic(localPath, remoteBytes, 0o644)internal/mountsync/syncer.go:6033-6051, implementation at syncer.go:8285

The websocket notification itself is cheap and arrives once. The expense is the N sequential ReadFile round trips that follow it, each paying full latency before the next begins. Nothing in this path batches or parallelises across the files of a single change set.

This also means the cost is paid again on a WAN deployment at WAN RTT per file — where the LAN measurement above used a ~4.5 ms RTT, a 50 ms RTT would put an 11-file change set well past half a second on this leg alone.

Suggested directions

Not prescribing an implementation, but the options seem to be:

  1. Bounded-concurrency fetch. Issue the per-file ReadFile calls with a small worker pool instead of serially. Lowest-risk change; turns N × RTT into roughly ceil(N/workers) × RTT. Ordering is already not guaranteed across files, and each file is published by an independent atomic rename, so concurrency does not change the visibility contract per path.
  2. Bulk read on the receive path. POST /v1/workspaces/{id}/fs/bulk already exists for the write direction. A read counterpart would collapse a change set into one round trip. Larger change, better asymptotics, and it needs a size cap so a big change set does not become one enormous response.
  3. Carry small file contents on the websocket event. Removes the follow-up fetch entirely for small files, at the cost of a fatter event and a size threshold to pick.

Any of these should be measured against the same harness so the before/after is comparable — harness/ in the evidence directory is reusable, and harness/assertions.py already encodes the gates.

Why it matters

Agents commit change sets, not single files. The single-file figure (20.2 ms) is the case that looks good and the case that rarely happens; the 11-file figure (216.7 ms) is the realistic one. The repository's own last 300 non-merge commits have a median of 6 files and a p75 of 11 files per commit, which is where the 11-file trial shape came from.

This also corrected a public claim: docs/multi-agent-collaboration-assessment.md previously asserted sub-200ms steady-state propagation, inferred by halving a round-trip measurement. That claim does not hold for realistic change sets, and PR #405 replaces it.

Scope note

Filing as a standalone engineering issue for triage. No readiness label is attached, so this does not dispatch.

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions