Repository navigation
fix: open one MongoDB client per Worker request to stop 1101s on bins - #255
Conversation
workerd ties a socket to the request that opened it. The cached KeyvMongo client in the Worker served later requests and kept driver monitoring alive after its request ended, so bin requests failed with an uncaught error that Cloudflare reports as 1101. The Keyv store now opens a client inside the request scope, carried by AsyncLocalStorage through handleAsNodeRequest, and closes it after the response in ctx.waitUntil. Nothing is cached at isolate scope. An operation outside a request opens and closes its own client. An error that escapes the Node bridge becomes a redacted JSON 500 instead of a Worker exception. An opt-in workerd test runs the create/capture/get/list loop against a real MongoDB when MOCKHTTP_TEST_MONGODB_URI is set. Co-authored-by: Jared Wray <me@jaredwray.com>
Co-authored-by: Jared Wray <me@jaredwray.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
| } | ||
|
|
||
| async deleteMany(keys: string[]): Promise<boolean[]> { | ||
| return this.run((store) => store.deleteMany(keys)); |
There was a problem hiding this comment.
NoSQL injection attack possible - high severity
Query injection attacks are possible if users can pass objects instead of strings to query functions such as findOne.
By injecting query operators attackers can control the behavior of the query, allowing them to bypass access controls and extract unauthorized data. Consider the attack payload ?user_id[$ne]=5: if the user_id query parameter is passed to the query function without validation or casting its type, an attacker can pass {$ne: 5} instead of an integer to the query. {$ne: 5} uses the 'not equal to' operator to access data of other users.
While this vulnerability is known as NoSQL injection, relational databases (mysql, postgres) are also vulnerable to this attack if the query library offers a NoSQL-like API and supports string-typed query operators. Examples include prisma and sequelize versions prior to 4.12.0.
| return this.run((store) => store.deleteMany(keys)); | |
| return this.run((store) => store.deleteMany(keys.map(key => String(key)))); |
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #255 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 44 44
Lines 1781 1868 +87
Branches 409 412 +3
=========================================
+ Hits 1781 1868 +87 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9258478bf5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
|
|
||
| /** The scope of the current request, if one is active. */ | ||
| export function currentBinStoreScope(): BinStoreScope | undefined { |
There was a problem hiding this comment.
Remove the test-only scope accessor
A repo-wide search finds that only test/bin-store-scope.test.ts imports this export; production code never calls it. Test the scope through handleWorkerFetch instead, so the test does not add a production hook to inspect an implementation detail.
AGENTS.md reference: AGENTS.md:L72-L73
Useful? React with 👍 / 👎.
Please check if the PR fulfills these requirements
What kind of change does this PR introduce? Bug fix
Problem
On mockhttp.org,
/healthzreportsconnected: truebut bin requests fail intermittently with Cloudflare error 1101 (uncaught Worker exception)./healthznever fails becausepingMongoBinStoreopens and closes its own client per call.Root cause (confirmed)
createAdaptiveBinKeyvcached oneKeyvMongo/MongoClientper isolate and reused it across requests. workerd ties every socket to the request that opened it. The first request that opened the client succeeded; a later request reusing it, or the driver's monitoring on it after its request ended, hit the "different request" I/O restriction and surfaced as an uncaught exception.Reproduced with the pre-fix code under
wrangler dev(workerd) against a real localmongod8.0.4:This also explains why
POST /binsandbin_not_foundlookups 1101 in the soak: whichever request happens to reuse the stale client fails, regardless of route.Fix
Per-request client, as decided. Nothing Mongo-related is cached at isolate scope.
worker/bin-store.ts:RequestScopedMongoStoreis a Keyv storage adapter that delegates every operation to the current request'sKeyvMongo. The request scope (BinStoreScope) is carried byAsyncLocalStoragefromhandleWorkerFetchthroughhandleAsNodeRequestinto the Fastify bin handlers, so a capture'sget+setshare one client. The scope tracks in-flight operations;closeBinStoreScopewaits for them, then disconnects. Outside a scope (Nodeinject, tests) each operation opens and closes its own client, so cross-request reuse is impossible by construction.worker/app.ts:handleWorkerFetch(request, env, ctx, dispatch)creates the scope, runs the dispatch inside it, and hands the close toctx.waitUntil. A last-resortcatchturns anything that escapes the Node bridge into a redacted JSON 500 instead of a Worker exception.worker/index.ts: passesctxthrough.@keyv/mongo, the collection, the document format,guardMongoStoreredaction/logging, and/healthzare unchanged. Driver options are nowmaxPoolSize: 1,connectTimeoutMS: 5000(one request needs one connection).keyv.connectinfo line per bin request withlatencyMs; thekeyv.openline is now only emitted when the URI shape is dirty. Newkeyv.closeerror line if a close fails. No URI, host, or credentials in any line.Tradeoff
Each bin request pays a connection setup to Atlas: DNS SRV, TLS, SCRAM auth, plus
@keyv/mongo's connect-timedropIndex/createIndexround trips, then the operation and a close inwaitUntil(not on the response path). Locally that is p50 8 ms / p95 12 ms; against Atlas expect roughly 150-800 ms depending on region distance. In exchange, no socket outlives its request, so the 1101 class of failure cannot occur, and no background driver timers run between requests. If latency matters later, the next lever is a minimalMongoClientadapter that skips the index round trips (not done here to keep@keyv/mongoas requested). A Durable Object owning the connection, DO SQLite, and KV were ruled out by the decision.Verification
wrangler devwith the fix, local MongoDB: 315 sequential create/capture/get/list/404/delete requests, 0 failures, 0 1101/500, and exactly 315keyv.connectlines (one client per bin request, confirming ALS propagates through the Node bridge). 8 parallel soakers x 157 requests: 0 failures, 0 Worker errors.worker bins on MongoDB under workerdruns the loop against a real MongoDB whenMOCKHTTP_TEST_MONGODB_URIis set (MOCKHTTP_TEST_MONGODB_URI=mongodb://127.0.0.1:27017/mockhttp npx vitest run test/worker-runtime.test.ts). CI has no MongoDB and skips it.test/bin-store-scope.test.tscovers scope sharing, waiting for in-flight operations, per-operation fallback, closed-scope fallback, missing URI, connection-failure drop, error forwarding to Keyv, close failure logging, dirty URI logging, memory/Mongo switching.test/worker.test.tscovers one client perhandleWorkerFetchrequest closed viactx.waitUntil, and the JSON 500 safety net.pnpm test: 50 files, 577 passed, 1 skipped, 100% lines/functions.pnpm buildandbiome check --error-on-warningspass.Verify on production after deploy
Expected:
failures: 0across 200 requests (previously 1101s appeared within the first handful). In the Workers logs (observability is enabled), each bin request produces one{"source":"mockhttp.bin-store","phase":"keyv.connect",...,"connected":true,"latencyMs":N}line and noUncaughtentries;latencyMsshows the per-request connect cost against Atlas. Run the loop a second time with 4-8 copies in parallel to cover concurrent requests on one isolate.