[e2e] Add Python e2e Test - #3369
Conversation
🦋 Changeset detectedLatest commit: 672209f The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 183209ms → this run 169135ms (Δ -14074ms, -8%) 📜 Previous results (4)bc7b027Tue, 11 Aug 2026 18:57:24 GMT · run logs
8a82563Tue, 11 Aug 2026 18:31:26 GMT · run logs
e3ee275Tue, 11 Aug 2026 17:44:14 GMT · run logs
a164e82Tue, 11 Aug 2026 01:39:35 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
Turns `packages/core/e2e/e2e.test.ts` into a cross-language conformance suite and adds `workbench/python` as its first non-JavaScript subject. Tests whose subject is the JavaScript implementation are marked `testJsOnly` / `describeJsOnly`; everything else is gated on a per-app `e2e-conformance.json` declaring which `workflows/99_e2e` fixtures the app implements, plus any individual tests its runtime cannot pass yet. Apps without that file — every existing one — are unaffected, so this is a no-op for the JS matrix. The Python app is built on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It runs against both `world-local` and a real Vercel deployment, the latter through the CBOR queue transport rather than HTTP. Baseline is 8 of 137 passing locally and 7 of 156 on Vercel; the rest skip, and the point of the lane is that they keep skipping for the declared reason. `workbench/python/README.md` records what is and is not ported, and the four gaps found on the way: the CBOR queue transport (vercel-py#265, #266), the Trusted Sources bypass header (vercel-py#278), `encr` payload decryption (vercel-py#279), and reading the slot-numbered `specVersion: 6` this repo's world-vercel now stamps (#3389). All four are fixed in the pinned rev. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
@vercel/workflow this PR is ready. One test is flaky due to the missing implementation of resilient start in Python, which is being added in vercel/vercel-py#284. This PR should be slightly updated after that fix, but I think the majority of this PR is ready for review. |
2 participants
Description
Turns
packages/core/e2e/e2e.test.tsinto a cross-language conformance suite and addsworkbench/pythonas its first non-JavaScript subject.The gate. Tests whose subject is the JavaScript implementation itself — the compiler,
@workflow/next, the CLI's TS codepaths — are markedtestJsOnly/describeJsOnly. Everything else is in scope for any language and is gated on a per-appe2e-conformance.json:languagejavascriptskips the JS-only testsfixturesworkflows/99_e2efixtures the app implementsunsupportedBoth axes are ratchets, which is the point — a fixture listed but missing from the deployed manifest is a hard failure, not a skip, and an
unsupportedentry naming a test that no longer exists fails too. An app can only move forward by editing the declaration, and the declaration is the thing review looks at.Apps that ship no
e2e-conformance.json— every existing one — are reported as "JavaScript, all fixtures present", so this is a no-op for the JS matrix. Format reference:packages/core/e2e/e2e-conformance.example.json.The Python app.
workbench/pythonis a bare ASGI app on vercel-py, pinned by git rev rather than a release so the suite reports on what is actually being built. It serves the.well-known/workflow/v1surface itself — the Python SDK does not claim to — and runs against bothworld-local(over HTTP) and a real Vercel deployment (over the CBOR queue transport, where the only HTTP route the driver touches ismanifest.json).Six fixtures are ported as a demo of the framework to start with:
addTen,nullByte,parallelSleep,promiseAll,sleepInLoop,sleeping.How did you test your changes?
Two CI lanes, both added here:
world-local)One baseline, not two: the extra 19 collected on Vercel are
e2e-agent.test.ts, which that lane picks up and skips whole, and the eighth local pass isdeploymentId: 'latest' is a no-op in non-Vercel worlds.E2E Python Conformanceis deliberately its own job rather than agetTestMatrixrow. Every step in the two local lanes is JS toolchain —pnpm dev/pnpm start, the canarynextpatch,dev.test.tsandlocal-build.test.ts, the per-frameworkDEV_TEST_CONFIGof compiler-generated paths — and none of it has a Python counterpart. Thee2e-vercel-prodlane, whose rows only need a URL, does carrypythonas an ordinary matrix row.PR Checklist - Required to merge
pnpm changesetwas run to create a changelog for this PR@vercel/workflowin a comment once the PR is ready, and the above checklist is complete