[e2e] Port 14 more fixtures to the Python workbench app - #3495
Draft
fantix wants to merge 8 commits into
Draft
Conversation
`5c77acb` is two commits past the old pin: vercel-py #256 and #258, the write and read halves of the workflow streaming API. That is what makes `get_writable()` exist at all, so it is the prerequisite for the stream fixtures in the next commit rather than a routine bump. Relocked the documented way — `rm uv.lock && uv lock --no-config`, twice, then `grep -c 'source = { git' uv.lock` to confirm all 8 sibling packages still resolve from the checkout rather than silently falling back to PyPI. The lock moves by exactly the rev, as it should. Local suite unchanged at `8 passed | 129 skipped` on this commit alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Takes the local lane from `8 passed | 129 skipped` to `23 passed | 114 skipped` of 137. Every fixture here was picked because the Python SDK grew the API it needs, not because it looked easy: - **Races** (`promiseRace`, `promiseAny`, `sleepWinsRace`, `stepWinsRace`, `promiseRaceStressTest`). `Promise.race` / `Promise.any` have no asyncio spelling that takes bare awaitables, so two shared helpers supply them. Both resolve ties by argument order rather than by `asyncio.wait`'s set iteration order, which would be a latent replay divergence the first time two steps landed in the same turn. - **Streams** (`outputStream`, `outputStreamInsideStep`, `utf8Stream`) — unlocked by the pin in the previous commit. Both spellings are covered because they take different paths through the SDK: a handle the workflow body creates and passes into a step's arguments, and `get_writable()` called inside the step itself. - **Retries** (`errorRetrySuccess`, `errorRetryDisabled`, `errorRetryFatal`, `errorFatalCatchable`) via `get_step_metadata().attempt`, `max_retries=0` and `FatalError`. - **Child runs** (`spawnWorkflowFromStep`) and **`metadataFromHelper`**. Four new `unsupported` entries, and they are two defects, not four: - `sleepWinsRaceWorkflow` / `stepWinsRaceWorkflow` pick the right winner every time. They are the only two tests in the suite that bound a run's elapsed time from *above*, which makes them the only two that notice the ~5s `world-local` waits before redelivering a first delivery the app 500'd because it could not read the run row yet. Same upstream gap as the `resilient start` entry already there; all three come out together. - The two `FatalError` tests fail on the half Python cannot do. The step lifecycle is right — one attempt, run fails on it — but a thrown error loses its identity across the event log, so the body catches a `RuntimeError` and the run's `errorCode` is `RuntimeError` rather than `USER_ERROR`. What is left is now blocked on named APIs rather than on porting effort: hooks (19 tests, a shape difference), `setAttributes` (9, no equivalent), distributed abort, `getWorkflowMetadata`, `RetryableError`, and the two fixtures that call `start()` from a workflow body, which the Python sandbox denies by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests🐘 Local Postgres (1 failed)express-stable-quickjs (1 failed):
E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
❌ 🐘 Local Postgres
✅ 🪟 Windows
✅ 🌐 Cross-language Conformance
✅ vercel-multi-region
|
Contributor
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
fantix
marked this pull request as draft
August 12, 2026 18:10
`23 passed | 114 skipped` → `25 passed | 112 skipped`, and the more useful
number is the other one: the "hooks need a design decision upstream" bucket
was 19 tests and is 15, because the premise was wrong. `HookEvent`
implements `__await__` *and* `__aiter__`/`__anext__`, so
`for await (const p of hook)` ports to `async for payload in hook`
unchanged. The real constraint is narrower — `Hook.set_result` calls
`hook_cls(**raw)`, so the payload's structural TypeScript type becomes a
declared dataclass, loose in exactly the fields the fixture leaves optional.
Two of the four pass and two are exempted, and both exemptions are defects
this suite found rather than ones predicted before it ran:
- `hookWithSleepWorkflow` stalls on its first payload. `hook_created`,
`wait_created`, `hook_received` land; the step the body calls next never
does, with no error and a 200 on the delivery. Both neighbours rule out
the obvious causes — `hookWithSleepFinalStepWorkflow` takes the same hook
twice before its one step and passes, `sleepWithSequentialStepsWorkflow`
puts the same never-completing `sleep('1d')` in front of three sequential
steps and passes — which leaves the transition only this fixture makes:
resuming from `hook_received` straight into a new step suspension.
- `hookTokenReuseLoopWorkflow` never frees the token. vercel-py flushes
`hook_created` and `hook_disposed` from one task group with no ordering,
which is what upstream #2777 fixed on the TypeScript side.
`sleepWithSequentialStepsWorkflow` has no hook in it and was only ever
bucketed with them because it is their control. It is ported next to them
for the same reason: it is what makes the two failures above specific.
One translation trap is worth the comment it got in the fixture file:
`using hook` is not `try/finally`. A Python workflow body unwinds through a
`_SuspendException` on every suspension, so a `finally` around an `await`
runs once per turn instead of once at scope exit — disposing a hook there
deletes the suspension before the orchestrator can flush its `hook_created`,
and the run stalls with no hook for the driver to resume. Cost one debugging
round; should cost the next reader none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 194368ms → this run 171365ms (Δ -23003ms, -12%) 📜 Previous results (3)1310bd0Thu, 13 Aug 2026 15:32:20 GMT · run logs
bb847d2Wed, 12 Aug 2026 20:53:06 GMT · run logs
8bde3f6Wed, 12 Aug 2026 19:14:18 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
`24 passed | 113 skipped` → `26 passed | 111 skipped`, and two `unsupported` entries come out: `sleepWinsRaceWorkflow` and `stepWinsRaceWorkflow` pass on this rev. The pin is a throwaway branch on vercel-py, `pin/resilient-start-on-streaming-main` (`4b70f10`) — `main` at 5c77acb with #282/#283/#284 cherry-picked on top. The rebase is the point: the stack was written against 02173de, before the streaming API, so pinning its own head would take `get_writable()` away and break three stream fixtures. One cherry-pick conflict, in `_events_create_impl`'s id minting, resolved toward main's `self._new_id(...)` helper. Repin to plain `main` when the stack merges; `pyproject.toml` carries the whole story next to the `rev`. What it fixes is the thing that made the two race fixtures flaky: the run's first delivery no longer 500s on a run row that has not landed yet, so it no longer costs the ~5s the world waits before redelivering, which is the entire budget those two tests measure against. `resilient start` itself still fails, one layer deeper, and the reason string now says which layer. The run *is* bootstrapped from `run_started`; it is then rejected by its own model: 2 validation errors for NonFinalWorkflowRun input.bytes / input.str input_value={'__type': 'Uint8Array', … 'ZGV2bFtbMV0sMTIzXQ=='} That base64 is `devl[[1],123]` — the fixture's own input. `RunInput.input` is typed `Any` (`world.py:117`), so the `Uint8Array` envelope the local world's JSON queue transport wraps `bytes` in is never decoded, and `NonFinalWorkflowRun.input` (`world.py:220`) wants `bytes | str`. Local world only: the Vercel world's CBOR transport carries bytes natively, and the stack's own unit tests use fakes with no transport at all. So this lane is the only place it is visible, which is what the lane is for. Two things the upstream rebase will also need, noted here because this one hit them: the stack's test doubles predate the streaming API and fail to instantiate (`FakeWorld` and `_World` are missing `streams_get`, `streams_write`, `streams_close`, `streams_list`, `streams_get_info`, `streams_get_chunks`), 18 unit tests' worth, all one cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was true when written and stopped being true when the first hook fixture reached the Vercel lane. `encp` is not only the format one run uses to write to another — it is what every hook payload arrives as there, because the driver resumes from outside the run with no symmetric key, so `@workflow/world-vercel` seals to the run's public key instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`2edb224` is the stack rebased onto a main that already carries the
streaming API, so it no longer needs the throwaway branch this repo was
pinned to. `26 passed | 111 skipped` → `27 passed | 110 skipped`.
`resilient start: addTenWorkflow completes when run_created returns 500`
passes. That is the last of the three `unsupported` entries this defect
was costing, and the one written specifically to inject the condition —
so the exemption list is down from six to five and the run-row section of
the README is now history rather than a gap.
One of the stack's four commits exists because this lane found it:
`world-local` smuggles `bytes` through its JSON queue as a
`{__type: "Uint8Array"}` envelope, the file store already spoke that
dialect and the queue did not, and nothing noticed until
`runInput.input` became the first queue payload to carry bytes. Upstream
`d966b24`. Only this lane could see it — the Vercel world's queue is CBOR
and carries bytes natively, and the stack's unit tests build payloads
with no transport at all.
Also narrows gap 16 with an experiment the stable base made cheap.
`hookWithSleepWorkflow`'s stall does **not** need the concurrent
`sleep('1d')`: deleting that line changes nothing, the log is still
`hook_created`, `hook_received`, then no `step_created`. Together with
`sleepWithSequentialStepsWorkflow` passing (a pending wait does not block
step creation) and `hookWithSleepFinalStepWorkflow` passing (a step
created after two `hook_received` events is fine), what is left is one
shape: a step created after one `hook_received` in a body that would then
re-await the same hook. The reason string in `e2e-conformance.json` now
says exactly that, and says it is an observed shape rather than a root
cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…ecific
Repin to `93979e4`: the resilient-start stack has merged, and this is one
commit above `main` for `workflow: read sealed encp payloads`.
`encp` is what unblocks hooks on the deployed lane. Every hook payload
arrives sealed to the run's public key there, because the driver resumes
from outside the run with no symmetric key, so a runtime that cannot read
`encp` fails any hook fixture outright rather than skipping it. Upstream
derives the X25519 keypair from the same run key material `encr` already
resolves, so it needed no new key plumbing — the diff is two files, and
its 121 unit tests pass.
It buys zero tests here, and the reason is a worse finding than the one it
fixed. With `encp` out of the way `hookWithSleepFinalStepWorkflow` runs on
the Vercel lane, and locally it started flaking: a 120s timeout in 1 of 7
full-suite runs (5/5 in isolation, 6/6 after), with the same signature as
`hookWithSleepWorkflow`'s deterministic stall — a hook payload delivered
and the step that follows it never created.
That reframes the stall. It looked like a property of one body shape
("a step after one hook_received in a body that re-awaits the hook");
`hookWithSleepFinalStepWorkflow` differs only in that its step comes after
the *second* payload, so if it can fail too then both are the same race
and one of them merely usually wins it. Not confirmed — no event log was
captured for the failing run — and the reason string says so.
So the exemption stays, and it now costs more than a test: with every hook
fixture that carries a payload exempted, nothing in the suite exercises
`encp` at all. The fix for it rests on upstream's unit tests until the
stall is fixed, which makes that stall the thing standing between this
lane and verifying two upstream changes rather than one.
`27 passed | 110 skipped`, unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3369 — review that one first; this PR's diff is against its branch.
Local lane:
8 passed | 129 skipped→26 passed | 111 skippedof 137.Commits
5c77acb— two commits past the old pin: Fix workflow stack traces (no more evalmachine) #256/Bug: ERR_REQUIRE_ESM when using @workflow/world-postgres in Next.js App Router #258, the write and read halves of the streaming API.get_writable()did not exist before them.HookEventimplements__aiter__/__anext__, sofor await (const p of hook)ports toasync for payload in hookunchanged.hookWithSleepFinalStepWorkflow— Vercel lane only,encp. See below.encp.The pin is not on vercel-py
mainIt is
pin/resilient-start-on-streaming-main(4b70f10), a throwaway branch:mainat 5c77acb with vercel-py #282/#283/#284 (resilient start) cherry-picked on top. The stack was written against 02173de, before the streaming API, so pinning its own head would takeget_writable()away and break three stream fixtures. One conflict, in_events_create_impl's id minting, resolved toward main'sself._new_id(...).pyproject.tomlcarries the reasoning next to therev. Repin tomainwhen the stack merges.It buys two exemptions:
sleepWinsRaceWorkflowandstepWinsRaceWorkflowwent from failing ~half of all runs to passing, because the run's first delivery no longer 500s on a run row that has not landed, and so no longer costs the ~5s the world waits before redelivering — the entire budget those two tests measure.Six
unsupportedentries, five upstream causesresilient startRunInput.inputis typedAny, so theUint8Arrayenvelope the local world's JSON queue wrapsbytesin reachesNonFinalWorkflowRun.input, which wantsbytes | str. Local world only — Vercel's queue is CBOR and the stack's unit tests use fakes with no transport.hookWithSleepFinalStepWorkflowencp. On Vercel the driver resumes a hook as an external client with no symmetric key, so the payload is sealed to the run's public key; vercel-py readsencrand notencp, and the run fails on its first payload. Exempting it costs a local pass —unsupportedis not lane-aware — but the alternative was leaving the Vercel lane red on a gap this repo cannot close.hookWithSleepWorkflowhook_receivedstraight into a new step. Both neighbours pass, which is what makes it specific.hookTokenReuseLoopWorkflowhook_created/hook_disposedflushed with no ordering — the #2777 shape.FatalErrorThree of those five were found by running this suite, which is the argument for the lane.
What is left
Everything still skipped names a missing API rather than porting effort: hooks (15 —
metadataonBaseHook.wait(), where the wire model already carries the field, andgetConflict()),setAttributes(9), abort (3), and single names likeRetryableErrorandgetWorkflowMetadata.Testing
26 passed | 111 skipped (137)onworld-local, two consecutive full-suite runs. No changeset —.changeset/python-workbench-app.mdon the base branch already covers this app.🤖 Generated with Claude Code