diff --git a/docs/1.guide/7.integrations/1.celld.md b/docs/1.guide/7.integrations/1.celld.md index b23555f..da9e810 100644 --- a/docs/1.guide/7.integrations/1.celld.md +++ b/docs/1.guide/7.integrations/1.celld.md @@ -12,22 +12,25 @@ title: celld celld uses only a small part of the S3 API, but conditional writes must work correctly. [`mountx/s3`](/transports/s3) supports these requests and can serve a celld bucket from an `FsDriver`. ::note -This setup was tested in August 2026 with celld 0.2.0 and `mountx/s3` from `main`, after the [conditional `PUT` fix](/transports/s3#conditional-writes). The test used one host, one gateway process, two celld nodes, and a [node-fs](/guide/drivers/built-in#node-fs) driver with a local directory. +This setup was tested in September 2026 with celld 0.5.0 and `mountx/s3` from `main`. The test used one Linux host, one gateway process, one or two celld nodes, and a [node-fs](/guide/drivers/built-in#node-fs) driver with a local directory. :: ## What celld needs from a bucket -celld's [fencing document](https://github.com/denoland/celld/blob/main/docs/fencing.md) lists three requirements. Only one node may own a cell at a time. A record in the bucket controls that ownership. +celld's [storage guarantees](https://github.com/denoland/celld/blob/main/docs/guarantees.md) list four requirements. Only one node may own a cell at a time. A record in the bucket controls that ownership. | Requirement | Request | What `mountx/s3` does | | ------------------------------ | ------------------------------- | --------------------------------------------------------- | | Create only if missing | `PUT` with `If-None-Match: *` | Uses `O_CREAT\|O_EXCL`; the losing request receives `412` | | Update only if unchanged | `PUT` with `If-Match: ""` | Compares and writes under a per-key `S3Session` lock | | Read a completed write at once | `GET` after a successful `PUT` | Uses the driver's read-after-write behavior | +| Read part of an object | `GET` with `Range` | Reads the range from the file and answers `206` | -celld also sends `ListObjectsV2`, `GET`, ranged `GET`, `PUT`, `HEAD`, and `DELETE` requests. It uses path-style URLs and SigV4 signatures. Its list requests use `prefix` and `start-after`. It does not use multipart uploads. +celld checks these itself. `celld diagnose`, and every node before it serves, sends four conditional writes and one ranged read to objects under `probe/`. A node stops when the bucket ignores a condition or returns a wrong range. `mountx/s3` passes these checks. -celld adds `x-amz-meta-litestream-timestamp` to replication segments. The gateway stores only [`x-amz-meta-mtime`](/transports/s3#mtime-not-much-else), but celld does not read the Litestream header back. +celld also sends `ListObjectsV2`, `HEAD`, `DELETE`, `DeleteObjects`, and multipart upload requests. It uses path-style URLs and SigV4 signatures. Its list requests use `prefix`, `delimiter`, `start-after`, `max-keys`, and `continuation-token`. It uses multipart uploads for replication segments of 5 MiB or more and for large R2 writes. + +celld adds user metadata to some objects: `x-amz-meta-litestream-timestamp` on replication segments, `x-amz-meta-sha256` on deployment assets, and retention and schema metadata on telemetry. The gateway stores only [`x-amz-meta-mtime`](/transports/s3#mtime-not-much-else). celld does not read the first and the last back. Without the `sha256` value, `celld deploy` uploads an asset again even when the bucket already has it. For R2 bindings, see [Known problems](#r2-bindings-lose-object-metadata). ::warning **Use exactly one gateway process for each fleet bucket.** Conditional writes are protected only inside one `S3Session`. If two `createS3Server()` processes use the same directory, each process has its own lock. Two conflicting writes could then both succeed. Point every celld node at the same gateway process. @@ -51,7 +54,7 @@ await using server = await createS3Server(createNodeFsDriver("./bucket"), { console.log(server.url); // http://127.0.0.1:9100 ``` -Pass the same credentials to celld through the standard AWS environment variables. `celld deploy` also needs [esbuild](https://esbuild.github.io) on `PATH`: +Pass the same credentials to celld through the standard AWS environment variables. `celld deploy` also needs [esbuild](https://esbuild.github.io) on `PATH` for Worker code: ```sh export AWS_ACCESS_KEY_ID=celld AWS_SECRET_ACCESS_KEY=celld-secret AWS_REGION=us-east-1 @@ -70,36 +73,48 @@ CELLD_WATCH=./state celld \ --advertise 127.0.0.1:8081 ``` -You must set both `--endpoint` and `--region`. The endpoint is not AWS, so celld cannot find the region from it. `CELLD_WATCH` is the node's local working directory. It is not the bucket directory. +Set `--endpoint` for every command. With an endpoint, celld uses path-style URLs. `--region` defaults to `AWS_REGION`, then `us-east-1`. When the gateway sets `region`, it must be the same value. `CELLD_WATCH` is the node's local working directory. It is not the bucket directory. + +To check the bucket before you start a node: + +```sh +celld diagnose --bucket s3://cells --endpoint http://127.0.0.1:9100 --region us-east-1 +``` ## What works - **Deploy:** `celld deploy` bundles the Worker. It writes `deploy/current.json` and the versioned objects. +- **Bucket checks:** `celld diagnose` and the startup checks report the conditional writes and the ranged read as supported. - **Activation:** When no node owns a requested cell, one node gets the ownership record and starts the isolate. -- **Replication:** celld copies the cell's SQLite data to the bucket as LTX segments under an epoch prefix. -- **Restore:** You can stop a node, delete its `CELLD_WATCH` directory, and start it again. It restores the cell from the bucket, and the Durable Object keeps its previous state. +- **Replication:** celld copies the cell's SQLite data to the bucket as LTX segments under an epoch prefix. Large segments use multipart uploads. +- **Restore:** You can stop a node, delete its `CELLD_WATCH` directory, and start it again. It restores the cell from the bucket, and the Durable Object keeps its previous state. A large cell restores in paged mode, which reads database pages from the bucket with ranged `GET` requests. - **Two nodes:** The nodes find each other through leases in the bucket. If a node loses the race for a cell, it sends the request to the owner. - **Fencing:** When nodes race for a new cell, only one becomes the owner. The others receive `412` and follow the winning owner. +- **Handoff:** A node that stops with `SIGTERM` uploads its databases and releases its ownership records. +- **Crash recovery:** After a node is killed, another node takes its cells once the lease expires. It recovers the dead node's log from the bucket, and the Durable Objects keep their state. +- **Workers KV:** Reads, writes, metadata, lists, and deletes. +- **R2 bindings:** Object bytes, ranged reads, conditional reads and writes, paged and delimited lists, multipart uploads, and batch deletes. Object metadata does not work; see below. +- **`celld cell list`:** Including `--limit` and `--after`. ## Known problems -### A reader may see an incomplete `PUT` +### R2 bindings lose object metadata -The gateway writes directly to the target object. It does not write a temporary file and rename it later. See [the write-in-place caveat](/transports/s3#the-write-in-place-caveat). Real S3 does not let readers see a partly written `PUT`, but this gateway can. +celld serves an `r2_buckets` binding from the fleet bucket, under `r2//`. It stores the R2 record of an object in the `x-amz-meta-celld-r2` header, and the content headers (`Content-Type`, `Content-Language`, `Content-Disposition`, `Content-Encoding`, `Cache-Control`) as headers. -celld can hit this problem when a node loses a conditional write and immediately reads the winning record: +The gateway does not store them. When a Worker reads the object back: -``` -celld ownership read failed: decode s3://cells/cells/Counter:5a8beec…/own.json: - EOF while parsing a value at line 1 column 0 -→ Worker failed: rejected: Error: route failed: ResolveFailed -``` +- `httpMetadata` is empty, except `contentType`, which is `application/octet-stream`. +- `customMetadata` has only `mtime`, which comes from the gateway's own `x-amz-meta-mtime`. +- `checksums` is empty. -In one test, two nodes requested 100 different new cells at the same time. This error happened four times. The request failed, but the nodes did not create two owners or different copies of the data. See [issue #20](https://github.com/pithings/mountx/issues/20). +The bytes, the size, the ETag, and the upload time are correct. -### `celld diagnose` only checks access +### A reader may see an incomplete `PUT` + +The gateway writes directly to the target object. It does not write a temporary file and rename it later. See [the write-in-place caveat](/transports/s3#the-write-in-place-caveat). Real S3 does not let readers see a partly written `PUT`, but this gateway can. -The bucket test in `celld diagnose` sends one list request. It can print `ok bucket` even when a bucket does not support conditional writes. Use this result only to confirm that celld can reach the bucket. It does not confirm that the bucket is safe for a fleet. +With celld 0.2.0, a node that lost a conditional write and immediately read the winning record could read it half-written, and that request failed. The celld 0.5.0 test did not show this failure, but the gateway behavior has not changed. ## Not tested @@ -107,6 +122,8 @@ The bucket test in `celld diagnose` sends one list request. It can print `ok buc - Long runs, many cells, or low-memory conditions. - Crash safety for the directory used by the driver. The gateway passes writes to the driver and does not call `fsync` itself. - Drivers other than node-fs. The [memory driver](/guide/drivers/built-in#memory) keeps the whole fleet in RAM and loses it when the process exits, so it is not suitable for a fleet bucket. +- Hosts other than one Linux 6.18 host with an ext4 directory, including older kernels still used by LTS distributions. +- celld features not listed under [What works](#what-works), such as Workflows, Containers, and telemetry retention. ## Next diff --git a/test/integrations/celld.test.ts b/test/integrations/celld.test.ts index 53f1346..32605f1 100644 --- a/test/integrations/celld.test.ts +++ b/test/integrations/celld.test.ts @@ -54,16 +54,17 @@ * ## The known rough edge, and why this file is not flaky * * A `PUT` writes the object in place, with no temporary file and no rename, so - * a reader can catch one half-written (issue #20). celld meets it because the - * loser of a compare-and-swap immediately re-reads the record it lost, and gets - * `EOF while parsing a value at line 1 column 0` when it lands mid-write. That - * fails the *request* — `route failed: ResolveFailed` — and it does not produce - * two owners. + * a reader can catch one half-written. celld 0.2.0 met it because the loser of + * a compare-and-swap immediately re-read the record it lost, and got + * `EOF while parsing a value at line 1 column 0` when it landed mid-write. That + * failed the *request* — `route failed: ResolveFailed` — and it did not produce + * two owners. celld 0.5.0 has not shown it (200 raced cells, none failed), but + * the gateway still writes in place, so the window is still there. * * So the race case asserts what the fence actually guarantees and tolerates what * is known to be missing: no cell may ever answer `1` twice (that is two * owners), and most cells must come out clean. It does not demand that every - * request succeed, because #20 says some will not. + * request succeed, because a half-written read can still fail one. * * Every spawn is bounded, every wait has a deadline, and no literal control * character appears in this file. @@ -463,9 +464,9 @@ describe.skipIf(!usable)("a celld fleet with mountx/s3 as its bucket", () => { } /* Not every request has to succeed: a loser re-reads the record it lost and - can catch the winner's `PUT` half-written (issue #20), which fails that - one request without ever handing out a second owner. Four in a hundred, - measured. + can catch the winner's `PUT` half-written, which fails that one request + without ever handing out a second owner. Four in a hundred on celld + 0.2.0; none in two hundred on 0.5.0. This is also the line that catches issue #19, and it is worth knowing why it rather than the two-owners check above. When the fence is gone,