[world] Expose capabilities in deployment health checks - #3425
[world] Expose capabilities in deployment health checks#3425NathanColosimo wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 459e34b The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 183209ms → this run 177685ms (Δ -5524ms, -3%) 📜 Previous results (4)30d3802Mon, 10 Aug 2026 23:58:42 GMT · run logs
7084900Mon, 10 Aug 2026 23:16:58 GMT · run logs
a6f6825Mon, 10 Aug 2026 21:45:10 GMT · run logs
984728eMon, 10 Aug 2026 21:02:51 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
NathanColosimo
changed the title
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
984728e to
a6f6825
Compare
681274e to
78b2b1e
Compare
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Reviewed at 7084900. The direction is right — one capability shape for the local World and a probed target, Zod-inferred instead of hand-rolled parsing — and the suite is green as written. But the strictness semantics are a forward-compatibility trap that will fire on the very next capability addition, and I verified it experimentally.
The problem: .strict() turns "newer peer" into "unhealthy peer".
WorldCapabilitiesSchema is .strict(), and parseHealthCheckResponse does safeParse(response) → null on failure. I built the branch and ran the exact scenario:
capabilities: { hookRetention: { active: true }, slotEventIds: true }
→ safeParse success: false ("Unrecognized key: slotEventIds")
→ WHOLE RESPONSE DISCARDED (healthy / specVersion / encryptionPublicKey / hookResumeInputVersion all lost)
slotEventIds isn't hypothetical — #3389 (approved, in flight) adds exactly that field, and every world it touches will declare it. The moment both PRs are in, any deployment on the newer world reads as healthy: false ("timed out") to any prober on this build. Since routine deploys create SDK version skew between live deployments, this fires in normal operation: cross-deployment start() silently degrades to the legacy wire format, loses compression, loses the sealed-args key preload, and closes the parallel-resume gate — against a perfectly healthy target. The rejects invalid World capabilities test currently encodes this failure mode as correct.
There's also a tolerance regression on the other fields: the old parser kept healthy: true and skipped a malformed optional field (specVersion: "not-a-number"); the new safeParse discards the whole response (verified). And the PR description says "treating missing or invalid capabilities as unsupported" — the code treats them as unhealthy, which is a different and much worse thing.
Requested changes:
- Drop
.strict()fromWorldCapabilitiesSchemaand the innerhookRetentionobject. Zod's default strip semantics give exactly the behavior the description promises: unknown capability fields are ignored (fail-closed per capability), and a newer responder parses clean. - Make
capabilitiesunable to sink the response:capabilities: WorldCapabilitiesSchema.catch(undefined).optional()(or equivalent) so even a genuinely malformed capabilities object degrades to "unsupported" whilehealthy/encryptionPublicKey/versions survive. Consider the same per-field.catch(undefined)for the other optionals to restore the old parser's field-level tolerance — the probe is an optimization and should never manufacture failure. - Update the tests to encode tolerance: unknown capability key → known capabilities still surfaced; malformed optional field → field omitted,
healthykept. (The currentrejects …tests assert the trap.) - Coordinate with #3389: it adds
slotEventIdsto the interface this PR deletes frominterfaces.ts— whichever lands second must add the field to the schema, and the merge is a semantic conflict, not a textual one, in one direction.
Non-blocking:
- The schema's doc comments are solid but noticeably abbreviated from the interface's (the
hookResumeDedupper-lookup-attestation note and the fullerdeploymentAffinityrationale didn't survive). Worth carrying the operational detail over — the schema is now the only home those docs have. - Changeset (minor across core/world/workflow) is right for the new export.
The refactor itself is good and #3426 presumably needs it — with strip-plus-catch semantics this becomes a clean approve.
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
3 participants
Summary
This gives start() one capability shape for both the current World and a target deployment's World.
Stack
Testing