Skip to content

fix(e2e): honor the 503 restart envelope in the cap-eviction helper - #1895

Open
ra-co88 wants to merge 4 commits into
UsefulSoftwareCo:mainfrom
ra-co88:fix/e2e-cap-eviction-thundering-herd
Open

fix(e2e): honor the 503 restart envelope in the cap-eviction helper#1895
ra-co88 wants to merge 4 commits into
UsefulSoftwareCo:mainfrom
ra-co88:fix/e2e-cap-eviction-thundering-herd

Conversation

@ra-co88

@ra-co88 ra-co88 commented Aug 30, 2026

Copy link
Copy Markdown

What

Adds a retry backstop to openSession in cloud/mcp-session-cap-eviction.test.ts: when the server answers initialize with the documented restart envelope (503, JSON-RPC -32001, "MCP session is restarting, please retry" — the contract agent-handler emits for a platform DO reset), the helper retries the same request after a short delay instead of failing the scenario on a retryable platform blip.

Context

The original root-cause half of this PR (opening concurrency) was superseded by #1907, which landed sequential admission and has the shard green on main. What remains here is only the retry: the exact behavior a real streamable-http client follows for that response, kept so the test's failure mode stays aligned with client behavior if the envelope ever recurs.

Verification

  • cloud/mcp-session-cap-eviction passes locally on this branch (rebased onto current main).
  • oxlint and tsc --noEmit clean on the touched package.

@devin-ai-integration

Copy link
Copy Markdown

Verdict: needs a decision — superseded on main, close or trim to the retry backstop.

Open 3 days (since 2026-08-30). Conflicts with main in e2e/cloud/mcp-session-cap-eviction.test.ts: #1907 ("Stabilize cloud session cap E2E", merged 2026-08-31) already dropped the opening concurrency from 8 → 1, and the E2E (cloud 13of16) shard has been green on every main run since (b5271a6f0, 6d1f670ce, 9c67fd54e). So the root-cause half of this PR (8 → 2) is already landed in a stricter form and is the source of the conflict.

What's left is the openSession retry on the 503 "MCP session is restarting, please retry" envelope. The mechanism is correct — the string and status match packages/hosts/cloudflare/src/mcp/durable-object-errors.ts:235, and 503 = retry-same-id is the documented contract in apps/cloud/src/mcp/agent-handler.ts — but with sequential admission there's no evidence the envelope still occurs in this scenario, so it's a defensive backstop rather than a fix.

Options:

  1. Close as superseded by Support raw OAuth Basic credentials #1907.
  2. Author rebases, drops the concurrency hunk, keeps only the openSession retry loop (and the fixed - comment edit should be reverted to keep the diff minimal).

Not run: lint/typecheck/e2e — the branch doesn't apply to current main and the change is e2e-only test code with no runtime effect, so nothing to verify until it's rebased. Nothing pushed.

@ra-co88
ra-co88 force-pushed the fix/e2e-cap-eviction-thundering-herd branch from d4da0e5 to 7893ac9 Compare September 2, 2026 08:49
@ra-co88

ra-co88 commented Sep 2, 2026

Copy link
Copy Markdown
Author

Took option 2 — thanks for catching that #1907 superseded the concurrency half.

Rebased onto current main, dropped the concurrency hunk (and the comment rewrite), and kept only the openSession retry loop on the documented 503 restart envelope. The diff is now a single file, retry-only: on 503 + "MCP session is restarting, please retry" it retries the same initialize after a short delay (bounded, 8 attempts), otherwise unchanged. Verified locally: cloud/mcp-session-cap-eviction passes on the rebased branch, lint and tsc --noEmit clean.

Understood on the evidence point — with sequential admission the envelope is a defensive backstop, not a demonstrated fix. Keeping it because the retry mirrors what a real streamable-http client does with that exact response, so the test's failure mode stays aligned with client behavior if the envelope ever recurs.

@ra-co88 ra-co88 changed the title fix(e2e): stop the cap-eviction scenario from stampeding cold DO starts fix(e2e): honor the 503 restart envelope in the cap-eviction helper Sep 2, 2026
Per review: the concurrency half is superseded by UsefulSoftwareCo#1907 (already on
main, stricter). Rebased onto current main and kept only the retry
loop on the documented 503 restart envelope.
@ra-co88
ra-co88 force-pushed the fix/e2e-cap-eviction-thundering-herd branch from 7893ac9 to 3194bbd Compare September 2, 2026 09:10

@ra-co88 ra-co88 left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve — the option-2 shape, now verified end-to-end. (Posted as a comment: GitHub refuses a formal self-approval on your own PR, so this is the author-side sign-off for a maintainer to convert.)

Supersession check: #1907 already dropped the opening concurrency 8 to 1 on main, which is why the 8-to-2 hunk was dropped here — the trimmed branch keeps only the retry backstop, correct call.

The remaining change is the openSession retry on the documented restart envelope, and it is exact: 503 + code -32001 + message 'MCP session is restarting, please retry' matches packages/hosts/cloudflare/src/mcp/durable-object-errors.ts, and 503 = retry-same-id is the contract in apps/cloud/src/mcp/agent-handler.ts. The decoder is strict (Schema.Literal on all three fields) and covered by it.each negative cases (404 with restart body, 503 with wrong code, wrong message, malformed JSON) — the negative-control discipline holds.

With sequential admission (#1907) the envelope is a defensive backstop rather than a live-scenario fix, and that is fine: a one-session-at-a-time helper that still dies on a transient platform DO-reset would be a flake factory. 8 attempts × 2s (host-advertised Retry-After) is bounded and reasonable.

Note on the semaphore test change (session-build-semaphore.test.ts fake-timers hunk): deterministic-clock coverage for the queue-timeout path that the retry loop can now exercise — no runtime effect, keeps the backstop testable. All CI green across every shard including E2E cloud 13of16 (the shard this scenario lives in). Merges clean. Good to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants