Skip to content

fix: open one MongoDB client per Worker request to stop 1101s on bins - #255

Merged
jaredwray merged 2 commits into
mainfrom
cursor/worker-mongo-per-request-client-c782
Oct 21, 2026
Merged

jaredwray merged 2 commits into
mainfrom
cursor/worker-mongo-per-request-client-c782

Conversation

@jaredwray

Copy link
Copy Markdown
Owner

Please check if the PR fulfills these requirements

  • Followed the Contributing and Code of Conduct guidelines.
  • Tests for the changes have been added (for bug fixes/features) with 100% code coverage.

What kind of change does this PR introduce? Bug fix

Problem

On mockhttp.org, /healthz reports connected: true but bin requests fail intermittently with Cloudflare error 1101 (uncaught Worker exception). /healthz never fails because pingMongoBinStore opens and closes its own client per call.

Root cause (confirmed)

createAdaptiveBinKeyv cached one KeyvMongo/MongoClient per isolate and reused it across requests. workerd ties every socket to the request that opened it. The first request that opened the client succeeded; a later request reusing it, or the driver's monitoring on it after its request ended, hit the "different request" I/O restriction and surfaced as an uncaught exception.

Reproduced with the pre-fix code under wrangler dev (workerd) against a real local mongod 8.0.4:

[wrangler:info] POST /bins 200 (29ms)                      <- opens the cached client
✘ [ERROR] Uncaught Error: The Workers runtime canceled this request because it detected that your Worker's code had hung ...
[wrangler:info] POST /b/5168825f2422 500 Internal Server Error   <- first reuse (1101 in production)
                                                            <- next request hung forever

This also explains why POST /bins and bin_not_found lookups 1101 in the soak: whichever request happens to reuse the stale client fails, regardless of route.

Fix

Per-request client, as decided. Nothing Mongo-related is cached at isolate scope.

  • worker/bin-store.ts: RequestScopedMongoStore is a Keyv storage adapter that delegates every operation to the current request's KeyvMongo. The request scope (BinStoreScope) is carried by AsyncLocalStorage from handleWorkerFetch through handleAsNodeRequest into the Fastify bin handlers, so a capture's get + set share one client. The scope tracks in-flight operations; closeBinStoreScope waits for them, then disconnects. Outside a scope (Node inject, tests) each operation opens and closes its own client, so cross-request reuse is impossible by construction.
  • worker/app.ts: handleWorkerFetch(request, env, ctx, dispatch) creates the scope, runs the dispatch inside it, and hands the close to ctx.waitUntil. A last-resort catch turns anything that escapes the Node bridge into a redacted JSON 500 instead of a Worker exception.
  • worker/index.ts: passes ctx through.
  • @keyv/mongo, the collection, the document format, guardMongoStore redaction/logging, and /healthz are unchanged. Driver options are now maxPoolSize: 1, connectTimeoutMS: 5000 (one request needs one connection).
  • Logging: one keyv.connect info line per bin request with latencyMs; the keyv.open line is now only emitted when the URI shape is dirty. New keyv.close error line if a close fails. No URI, host, or credentials in any line.

Tradeoff

Each bin request pays a connection setup to Atlas: DNS SRV, TLS, SCRAM auth, plus @keyv/mongo's connect-time dropIndex/createIndex round trips, then the operation and a close in waitUntil (not on the response path). Locally that is p50 8 ms / p95 12 ms; against Atlas expect roughly 150-800 ms depending on region distance. In exchange, no socket outlives its request, so the 1101 class of failure cannot occur, and no background driver timers run between requests. If latency matters later, the next lever is a minimal MongoClient adapter that skips the index round trips (not done here to keep @keyv/mongo as requested). A Durable Object owning the connection, DO SQLite, and KV were ruled out by the decision.

Verification

  • Soak under wrangler dev with the fix, local MongoDB: 315 sequential create/capture/get/list/404/delete requests, 0 failures, 0 1101/500, and exactly 315 keyv.connect lines (one client per bin request, confirming ALS propagates through the Node bridge). 8 parallel soakers x 157 requests: 0 failures, 0 Worker errors.
  • New opt-in workerd test worker bins on MongoDB under workerd runs the loop against a real MongoDB when MOCKHTTP_TEST_MONGODB_URI is set (MOCKHTTP_TEST_MONGODB_URI=mongodb://127.0.0.1:27017/mockhttp npx vitest run test/worker-runtime.test.ts). CI has no MongoDB and skips it.
  • New test/bin-store-scope.test.ts covers scope sharing, waiting for in-flight operations, per-operation fallback, closed-scope fallback, missing URI, connection-failure drop, error forwarding to Keyv, close failure logging, dirty URI logging, memory/Mongo switching. test/worker.test.ts covers one client per handleWorkerFetch request closed via ctx.waitUntil, and the JSON 500 safety net.
  • pnpm test: 50 files, 577 passed, 1 skipped, 100% lines/functions. pnpm build and biome check --error-on-warnings pass.

Verify on production after deploy

BASE=https://mockhttp.org
curl -s $BASE/healthz; echo
fail=0
for round in $(seq 1 20); do
  id=$(curl -s -X POST $BASE/bins | sed -n 's/.*"id":"\([^"]*\)".*/\1/p')
  [ -z "$id" ] && { echo "create failed"; fail=$((fail+1)); continue; }
  for i in 1 2 3; do
    code=$(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'content-type: application/json' -d "{\"round\":$round,\"i\":$i}" $BASE/b/$id)
    [ "$code" = 200 ] || { echo "capture $code"; fail=$((fail+1)); }
  done
  for i in 1 2 3; do
    code=$(curl -s -o /dev/null -w '%{http_code}' $BASE/bins/$id)
    [ "$code" = 200 ] || { echo "get $code"; fail=$((fail+1)); }
  done
  code=$(curl -s -o /dev/null -w '%{http_code}' $BASE/bins/$id/requests); [ "$code" = 200 ] || { echo "requests $code"; fail=$((fail+1)); }
  code=$(curl -s -o /dev/null -w '%{http_code}' $BASE/bins/does-not-exist); [ "$code" = 404 ] || { echo "missing $code"; fail=$((fail+1)); }
  code=$(curl -s -o /dev/null -w '%{http_code}' $BASE/bins); [ "$code" = 200 ] || { echo "list $code"; fail=$((fail+1)); }
  code=$(curl -s -o /dev/null -w '%{http_code}' -X DELETE $BASE/bins/$id); [ "$code" = 204 ] || { echo "delete $code"; fail=$((fail+1)); }
done
echo "failures: $fail"   # expect 0; any 1101 body or 500 counts as a failure

Expected: failures: 0 across 200 requests (previously 1101s appeared within the first handful). In the Workers logs (observability is enabled), each bin request produces one {"source":"mockhttp.bin-store","phase":"keyv.connect",...,"connected":true,"latencyMs":N} line and no Uncaught entries; latencyMs shows the per-request connect cost against Atlas. Run the loop a second time with 4-8 copies in parallel to cover concurrent requests on one isolate.

Open in Web Open in Cursor 

cursoragent and others added 2 commits October 20, 2026 05:13
workerd ties a socket to the request that opened it. The cached KeyvMongo
client in the Worker served later requests and kept driver monitoring alive
after its request ended, so bin requests failed with an uncaught error that
Cloudflare reports as 1101.

The Keyv store now opens a client inside the request scope, carried by
AsyncLocalStorage through handleAsNodeRequest, and closes it after the
response in ctx.waitUntil. Nothing is cached at isolate scope. An operation
outside a request opens and closes its own client. An error that escapes
the Node bridge becomes a redacted JSON 500 instead of a Worker exception.

An opt-in workerd test runs the create/capture/get/list loop against a
real MongoDB when MOCKHTTP_TEST_MONGODB_URI is set.

Co-authored-by: Jared Wray <me@jaredwray.com>
Co-authored-by: Jared Wray <me@jaredwray.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T05:18:13.814903Z 9258478 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Comment thread worker/bin-store.ts
}

async deleteMany(keys: string[]): Promise<boolean[]> {
return this.run((store) => store.deleteMany(keys));

@aikido-pr-checks aikido-pr-checks Bot Oct 20, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NoSQL injection attack possible - high severity
Query injection attacks are possible if users can pass objects instead of strings to query functions such as findOne.

By injecting query operators attackers can control the behavior of the query, allowing them to bypass access controls and extract unauthorized data. Consider the attack payload ?user_id[$ne]=5: if the user_id query parameter is passed to the query function without validation or casting its type, an attacker can pass {$ne: 5} instead of an integer to the query. {$ne: 5} uses the 'not equal to' operator to access data of other users.

While this vulnerability is known as NoSQL injection, relational databases (mysql, postgres) are also vulnerable to this attack if the query library offers a NoSQL-like API and supports string-typed query operators. Examples include prisma and sequelize versions prior to 4.12.0.

Suggested change
return this.run((store) => store.deleteMany(keys));
return this.run((store) => store.deleteMany(keys.map(key => String(key))));

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

@codecov

codecov Bot commented Oct 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (17307b7) to head (9258478).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #255   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           44        44           
  Lines         1781      1868   +87     
  Branches       409       412    +3     
=========================================
+ Hits          1781      1868   +87     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9258478bf5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread worker/bin-store.ts
}

/** The scope of the current request, if one is active. */
export function currentBinStoreScope(): BinStoreScope | undefined {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove the test-only scope accessor

A repo-wide search finds that only test/bin-store-scope.test.ts imports this export; production code never calls it. Test the scope through handleWorkerFetch instead, so the test does not add a production hook to inspect an implementation detail.

AGENTS.md reference: AGENTS.md:L72-L73

Useful? React with 👍 / 👎.

@jaredwray
jaredwray merged commit fabd193 into main Oct 21, 2026
13 checks passed
@jaredwray
jaredwray deleted the cursor/worker-mongo-per-request-client-c782 branch October 21, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants