Fleet frugality: SSH ControlPersist 10m (PHNX-2582) + shared Linear request budget (PHNX-2310)

[muqsitnawaz]

Two disjoint fleet-frugality changes, one worktree, one commit each. Both reduce
repeated cost when the fleet fans out — SSH handshakes and Linear API requests.

PHNX-2582 — SSH ControlPersist raised to 10 minutes (c4bfb81)

The OpenSSH multiplex master only survives while it stays idle under
ControlPersist. The dominant repeating fleet touch is the daemon's 5-minute
service loop (CATCHUP_TICK_MS and the reap/sync services, all 5 * 60_000),
which is longer than the old 60s window — so every poll's master had already
died and each poll paid a fresh TCP+auth handshake (~100-300ms direct, up to
~500ms relayed; measured on the live fleet 2026-08-10). At 60s, 100% of a
5-minute poll's cost was connection setup.

• Extract the value as SSH_CONTROL_PERSIST_SECONDS (10 min), which clears the
  5-minute cadence with margin so a periodic --device/fan-out poll lands warm
  (~3-5ms).
• Bounded on purpose: a master reused after a host sleeps costs a ~45s
  ServerAlive teardown, and a wider window only widens that chance; fan-outs stay
  bounded regardless by their own per-peer timeoutMs.
• Test asserts the emitted option and that the value exceeds the 5-minute
  poll; the three call-site tests that pinned the 60s literal now track the
  constant.

The ticket's other cheap win — "pre-filter fan-out by last-known-reachable so a
dead box cannot bound a parallel sweep" — is already implemented:
isDialableDevice() (cli/src/lib/devices/registry.ts) gates every fleet
fan-out (remote-agents-json.ts, session/remote/*, feed/watch.ts,
channels/owner-forward.ts), and REMOTE_STDOUT_MAX_BYTES caps each peer's
capture. The agents serve --control channel the ticket named was removed under
RUSH-3001, and the ticket owner's decision was explicitly to tune
ControlPersist, not swap the transport — which is exactly this change.

PHNX-2310 — Shared per-key Linear request budget (de85a31)

~13 concurrent drain agents on one Linear API key, each a separate short-lived
process, had no shared view of request spend — so a drain could exhaust Linear's
2500 requests/hour limit and throttle ticket-status reads fleet-wide.
linear-cache.ts already caches reads and backs off after a 429; that is
reactive — by then the budget is gone.

• New cli/src/lib/linear-rate-limit.ts: a proactive, cross-process hourly
  budget keyed by the API key. fetchLinearIssuesPage reserves against it before
  each request; when the pool is spent it serves the last cached (stale) snapshot
  instead of forcing a 429 — the same graceful degradation the card already uses
  on any other failure.
• Lock-free on-disk state, mirroring usage-backoff.ts: one empty stamp file per
  request over a sliding one-hour window; count = readdir after sweeping
  elapsed stamps. Two concurrent reservers create different files and neither can
  clobber the other, so N processes coordinate with no shared mutable document to
  race on.
• Budget (2400) sits under the hard 2500 to leave headroom for a human on the key
  and absorb the small check-then-create overshoot. State is keyed by a hash
  of the key, so the raw credential never lands on disk and two keys spend
  independently.
• Tests exercise the real filesystem path: budget exhaustion + refill as stamps
  age out, per-key isolation, cross-invocation coordination, and that the raw key
  is never written into a path.

Verification

Full vitest suite on fleet workers via cli/scripts/test.sh --shard 6 — output
quoted in a comment below. Affected-area runs locally:
ssh-exec.test.ts + the two updated call-site suites (77 passed) and
linear-rate-limit.test.ts + linear-project-counts.test.ts +
linear-cache.test.ts + usage-backoff.test.ts (63 passed). tsc clean.

[muqsitnawaz]

Full-suite verification (cli/scripts/test.sh --shard 6)

Ran the whole vitest suite across 6 fleet workers. Both diff-affected areas are green; the only two failures are pre-existing under-load flakes in files this diff does not touch, and both pass cleanly in isolation (72/72):

• self-update.test.ts:1152 (ensureGlobalBinLinks foreign-path repair) — this is the already-filed flake PHNX-3477 ("flakes under load").
• pty-client.test.ts:131 (auto-detects a real bun build --compile executable) — env-dependent execPath detection under the sharded worker; bin resolved to .local/bin/node instead of the fixture.

$ bun x vitest run src/lib/self-update.test.ts src/lib/pty-client.test.ts
 Test Files  2 passed (2)
      Tests  72 passed (72)

Diff-area runs, all green:

ssh-exec.test.ts + cloud/factory.test.ts + browser/drivers/ssh.test.ts   77 passed
linear-rate-limit.test.ts + linear-project-counts.test.ts +
  linear-cache.test.ts + usage-backoff.test.ts                            63 passed

tsc clean. The required GitHub test check (single-runner, --retry=2) is the authoritative gate and is not subject to the sharded-under-load flake.

[muqsitnawaz]

Non-author review (code-reviewer subagent — prix-cloud is paused, #1767)

Verdict: CHANGES REQUESTED (3 non-blocking quality/doc-accuracy findings; the underlying mechanisms verified correct). Addressing all three in a follow-up commit:

1. SHOULD — ControlPersist rationale cites 5-min daemon services that never SSH (ssh-exec.ts, docs, changelog, test comment). The fixed 5-min daemon ticks (catchup/reap) don't touch SSH; the periodic SSH services (usage-sync/auth-sync) tick at 15 min. Fix: reground the rationale in the real dominant caller (repeated ad-hoc --device/fan-out touches arriving over minutes), drop the nonexistent-service claim.
2. SHOULD — "fleet-wide" overstates the budget's scope (changelog, docs/projects.md). State lives under getCacheDir() = per-machine, unsynced; two machines on one key each budget 2400 independently. Fix: reframe to per-machine + call out the cross-device aggregate as a known limitation (the module docstring already says "this box").
3. SHOULD — no test for the fail-open write-failure branch (linear-rate-limit.ts catch → true). Fix: add a test that forces the write to throw and asserts reserveLinearRequest still returns true.

Verified correct (not flagged): the reserve catch-returns-true fail-open matches usage-backoff.ts's sanctioned pattern; sliding-window budget races are bounded by the 2400<2500 headroom; createdMsOf rejects malformed names; per-key isolation via SHA-256; the PHNX-2582 isDialableDevice/REMOTE_STDOUT_MAX_BYTES cheap-win claim is true; no other file hardcodes the old 60s.

[muqsitnawaz]

Non-author review — final verdict: APPROVE (clear to merge)

The code-reviewer subagent re-verified commit 867e0ee0f against the diff and confirmed all three CHANGES-REQUESTED findings are resolved:

1. ControlPersist rationale — regrounded on the real dominant caller (repeated ad-hoc --device/fan-out bursts over minutes); the two residual stale copies in controlOpts() and the ssh-transport.md trade-offs section are fixed, and the rationale is now consolidated behind {@link SSH_CONTROL_PERSIST_SECONDS}. Only accurate 15-min usage-sync/auth-sync references remain.
2. Linear budget scope — changelog/docs/module docblock corrected to per-machine, with the cross-device aggregate named as a known limitation.
3. Fail-open branch — deterministic ENOTDIR test added (8/8 pass).

Verified independently: tsc clean; linear-rate-limit.test.ts 8/8. The two sshExec timeout-detection failures the reviewer observed are a pre-existing, diff-unrelated sandbox flake (they reproduce on unmodified origin/main and pass 25/25 on the CI-style runner) — not a regression.

Merging on green.