Cluster, step 1 of 5: a gateway in front of independent daemons - #102
Closed
Annactswell wants to merge 26 commits into
Closed
Annactswell wants to merge 26 commits into
Annactswell wants to merge 26 commits into
Conversation
…t until the next garbage collection acquireSingleWriterLock's handle was discarded at boot. better-sqlite3 closes a handle whose object is garbage collected, and a closed handle releases the exclusive file lock — so the "one ledger, one daemon" guarantee lasted only until the first collection: measured 2026-09-11 with the fake executor, a second daemon on the same ledger started and listened seconds after the first. The handle is now kept in main.ts and closed on shutdown; the second instance dies at boot naming the conflict, as intended.
…paths the gateway imports without the executor The gateway in front of several daemons wants four small self-contained pieces of the daemon — the constant-time token compare, the per-name serialization queue, the single-writer file lock and the bounded shutdown — and nothing else. Four subpath entries expose exactly those; deliberately not the package root, whose import graph loads dockerode, execa and the AWS SDK into any process that touches it. The lock's busy sentence becomes the caller's: the gateway takes the same lock over its own database file and must name its own variable in the exit. auth.ts reads the cookie jar structurally so it type-bundles on its own, without @fastify/cookie's request augmentation in its graph.
…id, inside the name's slot when a create is in flight The gateway keeps no directory of where sandboxes live — a second copy of a fact the nodes' ledgers already hold, and every copy drifts. When its cache has no answer it asks every node this one read-only question in parallel and routes to the node that says yes. Three steps make the answer truthful about a create in flight. A row that exists answers at once, whatever its state: a restoring sandbox has a row, and waiting for its slot would hold the answer for the whole restore, long past the gateway's two-second patience. No row while the name's slot is busy waits its turn and looks again — the daemon creates first and writes the row second, both under the slot, so a gateway retrying a create whose answer was lost finds the sandbox on the node that built it and never places a second copy. No row and a free slot is a plain no. By id there is no slot to wait on and none is needed: nobody can ask about an id before the create that minted it has answered. The in-flight test parks the fake executor's create to hold the acquire mid-build; removing the wait turns it red.
…nd where it can be reached DORMICE_GATEWAY_ENDPOINT makes a daemon a node of a fleet. Set, it POSTs /checkIn to the gateway every DORMICE_CHECK_IN_INTERVAL_SECONDS with its id, the address the gateway may forward to (DORMICE_NODE_ENDPOINT, defaulting to its own loopback — right when gateway and node share a machine), its build and a fresh reading: CPU, memory, swap, the data disk, the ledger's census by state. Unset, the daemon is the whole platform by itself, as before. The token is the one DORMICE_API_TOKEN gateway and nodes share. Who knows the truth speaks: the node knows what it runs and where it lives, so the node reports and the gateway only listens. The gateway learns of a node from its first check-in — no registration verb, no nodes file — and reads two missed check-ins as down; the interval travels in every check-in so both ends measure with the same number. Failures are logged on the change, never every tick, and are never fatal: the gateway is the fleet's front door, not the node's reason to live. The shared host schema is split into its named parts (host reading, data disk, state counts) so getHostMetrics and the check-in describe the machine with one vocabulary; readHostReading is the one function both read through.
…e placed, existing ones are found by asking, everything else is forwarded raw A fleet is N daemons that know nothing of each other behind one gateway. The gateway holds no sandbox state: it keeps the nodes that have checked in (one table, written by the nodes themselves — no registration verb, no nodes file), what each last reported (memory), a cache of where names were last found, and a per-name slot. When the cache has no answer it asks every node "do you hold this sandbox?" in parallel, two seconds, and routes to the one that says yes; two yeses are a 409 it refuses to guess about; no yes while a node is silent is a 503 with Retry-After, because a name that lives only on a silent node must not be built a second time elsewhere. A new name is placed on the emptiest node by active density per core, skipping nodes that are down, above the CPU limit, at the active ceiling or below the data-disk floor, with each pick counted against its node until the next reading. The forwarding plane, error dialects and placement come from the first cut by file (branch router-v1), minus the directory, the claims, the reconciler and the actor header: one token for the whole fleet, and the node trusts the gateway whole. Faces in this cut: the native sandbox verbs, the E2B control plane by name or id, envd by header, the bare signed-URL door as an honest 501, and the gateway's own checkIn / listNodes / removeNode. Daemon-addressed verbs answer 501 until the configuration authority moves here; the sandbox port proxy joins with the sandbox domain. Only the fleet token opens the door in this cut; minted keys arrive with the key table. Reverse-proved: without the name slot twenty simultaneous acquires build several copies; without the placement counter a burst lands on one node; without the 404 re-check a destroy behind the gateway's back leaves a stale entry.
… holds a placement slot The placement counter exists so a burst inside one interval is counted against a node before its next reading shows it. It was only ever reset by the reading, so a sandbox placed and destroyed within the same interval — a short job, or the exam's churn — kept holding a slot the reading would never show; with the exam's two-per-node gate that shut placement for a whole interval after a handful of fast tests. Each node now remembers the ids its creates answered with since the last reading; a destroy of one of them takes the placement off the count, and a create the node itself refused is uncounted at once. A hop's 502/503/504, or no answer, still counts until the reading: the sandbox may exist.
…at joins and dies, all driven over the wire The setup boots node A exactly as before (every existing suite is untouched) and, outside docker mode, a gateway plus nodes B and C that share its token and check in every second. The exam speaks only the SDK, the official e2b package and plain fetch; direct calls to a node stage what the gateway must then find: a sandbox built behind its back is routed at once (no reconcile to wait for), a name on two nodes is a 409 naming both, a destroy behind its back is caught by the 404 re-check. Placement, the active gate and its Retry-After, five simultaneous acquires, streaming through two hops, envd and the bare signed door are graded; a node booted by the test itself joins at its first check-in, and when it dies its sandboxes 502 and a new name is a 503 naming it until removeNode. A second gateway on the same database file dies at boot naming the conflict, as does a second daemon on node A's ledger.
…ngeset for the fleet wire The unit is copied into place by hand until install.sh learns the two roles; it carries no docker dependency, because the gateway never touches a container.
… at the door, and a check-in that moves or shares an endpoint is said in the log forwardCreate took every 2xx id into placedIds and paid every node-side 4xx back to placedSinceCheckIn, wakes included. A wake was never counted in the first place, so destroying one inside the same interval uncounted a real placement and let the active gate admit one sandbox more than the node's reading allows. The callers now say whether pick() chose the node (a placement) or the name was found on it (a wake); only a placement moves the count either way. Reverse-proved: without the flag the new test reads b's count as 0 with a placement still in flight. nameOf and the E2B metadata.name are judged by the shared sandboxNameSchema before any node is asked. A node's 400 to lookupSandbox read as silence, so a 129-character name came back as a 503 with Retry-After — retry forever — instead of the 400 it deserves. Fleet.checkIn reports the endpoint a node moved from; the check-in route warns on a move and when two nodes report one endpoint. Both are misconfigurations (two machines sharing DORMICE_NODE_ID, whose default is node-1; a DORMICE_NODE_ENDPOINT naming the wrong machine) whose symptom downstream is a 409 on every name, and the check-in is the only place that sees them. e2b.ts: the `dialect` helper that existed only to be overridden is gone.
…OINT: the loopback default names the gateway's own machine Left unset, the node reports http://127.0.0.1:<DORMICE_PORT> as where it can be reached, and a remote gateway dials its own daemon (or nothing) for this node: every sandbox placed "here" lands there, and every lookup then finds it twice — a 409 on every name, from one forgotten variable. Refused at boot, naming the variable and the address to write, whenever DORMICE_GATEWAY_ENDPOINT is not loopback. An explicit value is the operator's word and is taken as written. The loopback test tolerates an unparsable URL: zod runs the object-level refinements even when a field failed, and the field's own error ("must be a full http(s) URL") must be the one that shows, not an Invalid URL thrown from inside the rule.
…swers at once when the slot is free tryRun-then-run did the same thing in three lines. run() executes the task immediately when nothing holds the key and queues it behind the holder otherwise — exactly "a plain no when the slot is free, wait for the acquire when it is not". SKIPPED is no longer imported here.
…nstead of followed The gateway hands a node's endpoint to undici as the request's origin when it forwards, and undici refuses an origin that carries a path (UND_ERR_INVALID_ARG, measured). Its own lookup joins `<endpoint>/lookupSandbox` as a string and works with a path. So a node reporting http://10.0.0.7:80/dormice was found by every lookup and reached by no forward. The shared wire now has `endpointSchema`: a trailing slash is dropped (the two ends must agree byte for byte, and `//lookupSandbox` is a 404 the gateway reads as silence), a path or query is a 400 at /checkIn, and DORMICE_NODE_ENDPOINT refuses the same at the daemon's boot, where the operator is looking. The node's check-in used fetch's default redirect handling. A front that answers plain http with a 308 to https (a Caddy binding the gateway's domain) would be followed across origins, and the Fetch standard drops Authorization on the way, so the gateway answered 401 and the log said "wrong token" where the address was wrong. The check-in no longer follows: it reports the 3xx and its Location, and names the variable to fix. Found by the second review pass of 2026-09-14; each fix reverse-proved (the new test goes red with the fix removed).
forwardCapture opened with `if (res.destroyed) return null`; forwardStream did not. A caller of a streamed verb (execCommand) that hangs up during the lookup round — up to two seconds when a node is slow — has a destroyed response whose 'close' has already fired, so the abort wired to 'close' would never come, and the node would run the command to its end for nobody. Same first line now, and the doc comment that already claimed the two followed one rule is true. Found by the second review pass of 2026-09-14; reverse-proved.
…d a second reporter under one id are refused while the node is live Three ways the fleet could end up with one name on two nodes, or a placement past every gate, found by the second review pass of 2026-09-14. The cache is a hint for every verb but the two creators: a stale entry costs a reader one 404 that evicts it, but acquireSandbox and the E2B named create are create-or-wake on the node, so a cache hit forwarded as a wake to a node that has since deleted the row (an E2B deadline kill is the scanner's routine, five minutes by default) rebuilt the sandbox there — past placement's four gates, uncounted in placedSinceCheckIn, every re-create of an expired name pinned to its first node. The creators now confirm a cache hit with that one node by id (Finder.byName confirm): found is a wake, absent evicts and asks the fleet, silent is the same 503 a silent stranger earns. One RTT per warm acquire. removeNode on a node still checking in deleted its row; a name of its acquired before its next check-in was placed elsewhere, then the node re-added itself and the name was on two nodes — a 409 an operator clears by hand. It is now a 409 up front, naming what to do: stop the daemon, wait two of its intervals (what "down" means), then remove. A check-in that changed a node's endpoint inside the previous reporter's own interval was written through. Two daemons sharing one DORMICE_NODE_ID (the default is node-1) flipped the endpoint at every check-in; a lookup asked whichever was current, a name on the other read as new and was built again, on two nodes, with no 409 ever. Such a check-in is now refused (409) naming both addresses; the first reporter keeps the id; a node that really moved is taken an interval later. The 503 for a silent node now says the node may be busy building that very name (its lookup waits for the name slot, and a cold create outlasts the gateway's two seconds), and the shared lookup comment no longer promises that a retry finds a sandbox still being built.
Node holds a written head until the first body byte. A node stream that opens and then waits for its first event looked, through the gateway, like a node that had not answered — measured: a 300ms pause before the first byte held the status line for 300ms. The head now goes out when the node's did.
…over a few milliseconds The sampler was primed right before the first check-in, so its first delta spanned the sliver between the two calls and read near 0 or near 100 by luck; a freshly restarted node could sit out its first interval on a number that meant nothing. Unprimed, the first reading is null — "no interval yet" — which placement lets through as unknown, as it was written to.
…s why in words undici's request() ignores an abort signal while the socket is still connecting, so a node whose host drops the SYN (a deleted VM, a closed security group — the case removeNode exists for) held every question for the dispatcher's 10s connect timeout, not the promised two seconds; and the creator's confirmation runs inside the name's slot, so queued acquires of a name cached there paid that in series. fetch honours the signal mid-connect (measured: request 10 500ms, fetch 2 001ms against 192.0.2.1); it follows no redirect here either. causeOf now prefers string codes: a DOMException's numeric legacy code rendered a timed-out node as "did not answer (23)". One causeOf, in lookup.ts, shared with forward.ts. The refused-connection test moved off port 9, which fetch refuses as a "bad port" without dialling.
clientGone was checked only at placement, after the lookup round: a creator queued behind a slow create or destroy whose client had given up still confirmed with the cached node — a question inside the slot, paid in series by every abandoned request behind a stuck name. It is now the first thing inside the slot on both faces, and still checked after the round of questions. The fake node learned a slow destroy for the test.
…talled gateway unit first A machine that runs both (the single-machine install is a fleet of one) upgraded with install.sh rebuilt only the server, CLI and console: the gateway's dist stayed on the older commit and ran it at its next restart, against a daemon whose check-in it may no longer parse. The gateway is now in the build filter, and a dormice-gateway unit that is running or enabled is restarted before the daemon so its first check-in lands on the new one.
The non-200 branch cut the gateway's answer at 200 characters. The gateway's longest refusal, the 409 naming both endpoints of a shared node id, runs to about 260 — the operator saw the diagnosis and lost the remedy. Cut at 400 now: whole enough for every sentence the gateway writes, short enough that a front's HTML error page does not flood the log.
It claimed the daemon runs its other verbs unserialized; the daemon holds a slot per name for thirteen of the fifteen. The gateway's own reason for taking the slot only for acquire and destroy — the two verbs whose outcome it acts on — stands on its own and is now the one stated.
A node deletes rows on its own (an E2B deadline kill is the scanner's routine) and tells no gateway, so the cache kept an entry for every sandbox ever created through the process for as long as it lived — an unnamed E2B create a minute is half a million dead entries a year. A hundred thousand entries, generous next to a fleet's live population; past it the entry nobody asked about for longest goes, and a hit moves an entry to the young end. A wrongly evicted entry costs the one round of questions any miss costs.
…names the way out A node whose DORMICE_NODE_ID changed checks in as a new node while its old id keeps its row; both are asked, both answer from the same address, and every name there was a 409 telling the operator to destroy one copy — of a sandbox that exists once. The finding now carries the endpoints, and when they are one the sentence says so: remove the id that no longer checks in (or correct a DORMICE_NODE_ENDPOINT that names the wrong machine). Nothing is collapsed or healed; the fleet's list is wrong and the operator is told exactly how.
…hen the failure changes fetch says "fetch failed" and keeps ECONNREFUSED, ENOTFOUND or the TLS error in cause; the log carried only the first half. And a failure was logged once per streak: a gateway that was unreachable and then, up again, refused this node as a twin of another (409) never made the log — the streak had already been announced. What is wrong is now compared with the numbers blanked, so a change of failure is one more line and a 409 that says 3s ago, then 4s ago, is still one.
…_NODE_ID The gateway tells nodes apart by it, and node-1 is what every other unconfigured node says too: the second to check in was refused as a twin at every check-in, or — when the first had been silent for an interval — taken for it having moved, and the first's names placed again elsewhere. The same rule as DORMICE_NODE_ENDPOINT, for the same reason: refused at boot, where the operator is looking. Beside its gateway, or alone, the default still serves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. The first of five steps toward a clustered Dormice: one gateway in front of N independent daemons. Not for merge on its own. The series lands together, and the production cut-over is a hand-run
install.sh, not the one-click updater.What this step adds
packages/gateway: the fleet's single door. It holds no sandbox state. Nodes check in every 15 s; a sandbox is found by asking every node in parallel (lookupSandbox); placement picks by (active + in-flight) / cores behind CPU, in-flight and data-disk gates. Two nodes holding one name answer 409 naming them; a silent node makes a new name answer 503 with Retry-After rather than risk a second copy.lookupSandboxverb; a check-in loop that runs only whenDORMICE_GATEWAY_ENDPOINTis set (unset means standalone, behaviour unchanged); the ledger lock handle held for the life of the process.e2e/src/gateway.test.ts: a black-box exam with a gateway in front of two fake-executor nodes.deploy/dormice-gateway.service, hand-installed for now.install.shroles come in a later step.Validated
pnpm testgreen in every package.The series
install.shroles and an image registry.Review commit by commit. Each message carries its reasoning, and the code comments carry the design.