Blob storage: `Blob` state machine with a pluggable gRPC data plane by rjhuijsman · Pull Request #109 · reboot-dev/reboot

rjhuijsman · 2026-07-23T15:41:41Z

Before this change, Reboot had no first-class way to store large binary objects: state machines hold protobuf state, which is unsuited to multi-megabyte payloads, so applications had nowhere to put user uploads like images or videos.

This PR introduces the Blob state type as the control plane for one immutable-once-committed binary object: its state holds only metadata (content type, size, upload progress, lifecycle, authorization) while the bytes travel directly between the client and a data plane. Data is uploaded to the data plane in parts, each via PUT to a URL minted by the control plane - compatible with S3 as a data plane, if we ever want that. Uploads are resumable, sizes are enforced against the real bytes at commit time, and never-committed blobs expire automatically.

The data plane is a plain gRPC service, BlobDataPlane (data_plane.proto — no Reboot options, implementable by anything), discovered via REBOOT_BLOB_DATA_PLANE_URL. Locally (rbt dev run / rbt serve run / unit tests) a filesystem-based data plane server is run out-of-the-box.

Authorization: Blob has two mechanisms that combine:

Blob creation is always application-mediated; it can't be done directly from a frontend, creation must go through a backend call. The backend can then use whatever existing auth mechanism they'd like. Assuming the Blob is given a random ID, knowing that state ID acts as a capability: to upload or download you must first know the ID.
Blobs can be created with an uploader_id and downloader_ids; if given these will limit uploads and downloads to only users whose IDs are in those lists.

Reviewer hint: review commits in-order.

TESTED: with new unit tests, and by manually running rbt dev run.

aviator-app · 2026-07-23T15:41:44Z

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This pull request is currently open (not queued).

How to merge

To merge this PR, comment /aviator merge or add the mergequeue-ready label.

See the real-time status of this PR on the Aviator webapp.

Use the Aviator Chrome Extension to see the status of your PR within GitHub.

Copilot

Pull request overview

Introduces first-class blob storage with a Reboot control plane, pluggable gRPC data plane, local filesystem implementation, and browser SDK.

Changes:

Adds blob protocols, lifecycle, authorization, multipart storage, and tests.
Integrates local data-plane startup into CLI and test harnesses.
Demonstrates attachments through the chat-room example and React hooks.

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
`tests/reboot/std/blob/v1/BUILD.bazel`	Configures blob tests.
`tests/reboot/std/blob/v1/blob_tests.py`	Tests blob lifecycle and authorization.
`tests/reboot/examples/chat-room/serve_expected_output.txt`	Updates structured message output.
`tests/reboot/examples/chat-room/BUILD.bazel`	Adds frontend blob packages.
`reboot/std/react/package.json`	Exports React blob helpers.
`reboot/std/react/BUILD.bazel`	Builds React blob helpers.
`reboot/std/react/blob/package.json`	Configures the blob submodule.
`reboot/std/react/blob/index.tsx`	Implements browser upload/download helpers.
`reboot/std/react/blob/BUILD.bazel`	Builds the React blob module.
`reboot/std/BUILD.bazel`	Includes the blob TypeScript API.
`reboot/std/blob/v1/package.json`	Configures the blob module.
`reboot/std/blob/v1/index.ts`	Exposes Node.js blob integration.
`reboot/std/blob/v1/BUILD.bazel`	Builds blob Python and TypeScript libraries.
`reboot/std/blob/v1/blob.py`	Implements the Blob control plane.
`reboot/std/blob/v1/_store.py`	Implements filesystem blob storage.
`reboot/std/blob/v1/_proxy.py`	Proxies application byte routes.
`reboot/std/blob/v1/_http.py`	Serves filesystem upload/download requests.
`reboot/std/blob/v1/_filesystem_server.py`	Hosts the local data plane.
`reboot/std/blob/v1/_data_plane.py`	Creates data-plane connections.
`reboot/routing/cors_settings.py`	Exposes upload ETags through CORS.
`reboot/examples/chat-room/frontend/web/src/App.tsx`	Adds attachment UI and uploads.
`reboot/examples/chat-room/frontend/web/src/App.module.css`	Styles attachment UI.
`reboot/examples/chat-room/frontend/web/package.json`	Adds blob frontend dependencies.
`reboot/examples/chat-room/frontend/reboot-non-react-web/src/main.ts`	Handles structured messages.
`reboot/examples/chat-room/frontend/mobile/src/App.tsx`	Handles structured messages.
`reboot/examples/chat-room/frontend/.tests/type_check.sh`	Installs local blob packages.
`reboot/examples/chat-room/backend/tests/chat_room_servicer_test.py`	Updates message assertions.
`reboot/examples/chat-room/backend/src/main.py`	Registers the blob library.
`reboot/examples/chat-room/backend/src/chat_room_servicer.py`	Creates attachment blobs.
`reboot/examples/chat-room/api/chat_room/v1/chat_room.proto`	Defines attachment APIs.
`reboot/examples/chat-room/.tests/serve_test.sh`	Updates response validation.
`reboot/cli/common/BUILD.bazel`	Builds data-plane startup support.
`reboot/cli/common/blob_data_plane.py`	Spawns the local data plane.
`reboot/cli/commands/serve.py`	Starts blobs under `serve run`.
`reboot/cli/commands/dev.py`	Starts blobs under `dev run`.
`reboot/cli/commands/BUILD.bazel`	Adds CLI blob dependencies.
`reboot/BUILD.bazel`	Packages blob runtime artifacts.
`reboot/aio/tests.py`	Runs a data plane in tests.
`reboot/aio/http.py`	Adds PUT route support.
`reboot/aio/BUILD.bazel`	Adds test-harness blob dependencies.
`rbt/std/BUILD.bazel`	Packages generated blob APIs.
`rbt/std/blob/v1/package.json`	Configures generated blob modules.
`rbt/std/blob/v1/data_plane.proto`	Defines the gRPC data-plane contract.
`rbt/std/blob/v1/BUILD.bazel`	Generates blob protocol bindings.
`rbt/std/blob/v1/blob.proto`	Defines Blob state and methods.
`documentation/docs/learn_more/testing.md`	Updates testing examples.
`documentation/docs/learn_more/define/protobuf.mdx`	Updates protobuf examples.
`documentation/docs/learn_more/call/from_within_your_app.mdx`	Updates internal-call examples.
`documentation/docs/learn_more/call/from_react.mdx`	Updates React examples.
`documentation/docs/learn_more/applications.mdx`	Documents blob registration and PUT routes.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-14T11:32:15Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-09-14T12:31:29Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Before this change, an application's custom HTTP routes (`application.http`) could only be registered for `GET`, `POST`, and `OPTIONS`; the docs listed the `PUT`/`DELETE`/... gap as a known limitation. This blocked serving a plain-HTTP upload endpoint, where `PUT` is the natural verb. Add `application.http.put(...)`, a sibling of the existing `post(...)` that forwards `methods=["PUT"]` to the underlying FastAPI route (the route-capture machinery already supports arbitrary methods; only the public sugar was missing). Update the custom-HTTP-routes documentation to list `PUT` among the supported methods. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Before this change, `Application.run()` invoked each library's `pre_run(application)` hook, but the `Reboot` in-process test harness (`reboot.aio.tests`) did not. A library that performs application setup in `pre_run` — for example, registering custom HTTP routes — therefore behaved differently under test than in a real run, and its routes were simply absent when brought up via the harness. Call `library.pre_run(...)` for every library in `Reboot.up()`, before deciding whether a local Envoy is needed, mirroring what `Application.run()` does. Libraries must already tolerate being `pre_run` more than once (a test may bring the same application up again after a `down`). This harness behavior is covered by unit tests introduced in a later commit (`reboot/std: add a `Blob` state machine with a gRPC blob data plane`): `blobs_tests.py` brings an application up with the `BlobsLibrary`, whose `pre_run` hook connects to the blob data plane and registers the byte-proxying HTTP routes the tests then exercise — which succeeds only when the harness has invoked `pre_run`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Reboot had no first-class way to store large binary objects: state machines hold protobuf state, which is unsuited to multi-megabyte payloads, so applications had nowhere to put user uploads like images or videos. Add `rbt.std.blobs.v1.Blob`, a state machine that is the *control plane* for one immutable-once-committed binary object. Its state holds only metadata — content type, expected/maximum size, upload progress, lifecycle status — while the bytes live in a *data plane* and travel directly between the client and that data plane via URLs minted per part. Uploads are resumable (parts are idempotent by number), sizes are enforced against the real bytes at commit time, and blobs that are never committed expire automatically. The data plane is a gRPC service, `BlobDataPlane` (`data_plane.proto`), deliberately free of Reboot options so that anything can implement it; the control plane discovers it via `REBOOT_BLOB_DATA_PLANE_URL` and calls it to provision uploads, mint URLs, finalize objects, and delete bytes. Keeping the implementation behind a bare gRPC URL means developers can write their own data planes to fit any environment. A data plane whose URLs are not directly reachable by clients (e.g. when run adjacent to an `rbt dev run` that's used over an `ngrok` tunnel) asks, via its `Configuration`, for certain requests to the app's HTTP server (under the reserved `/__/reboot/blob/`) to be forwarded to it; the `Blob` library then registers reverse-proxying routes on the application, so a single application origin serves both control plane and bytes and no second port needs exposing. A data plane whose URLs _are_ directly reachable (e.g. presigned S3) asks for nothing and is never in the application's path. This commit contains just one implementation of a data plane: a filesystem-based server. It binds loopback only and relies on the forwarded-path proxying above, so it works anywhere a Reboot app may be deployed. The `reboot.aio.tests.Reboot` test harness runs the filesystem data plane in-process for every test, so applications using `reboot.std.blobs` work in unit tests out of the box — which is also how this commit is tested. A later commit has `rbt dev run`/`rbt serve run` provide the same data plane for local runs. Authorization model: blob creation is application-mediated (the application enforces quota and size policy), after which the blob's framework-generated random ID acts as a capability (only callers who know the ID can call, and thus read or write, the blob). In addition the backend may limit the identities of callers by setting `uploader_id` and `downloader_ids` at create-time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

With the blob data plane behind a gRPC interface, an application that uses `reboot.std.blobs` needs a data-plane service to talk to — but local development must keep working out of the box, with no external service to configure. Have `rbt dev run` and `rbt serve run` start the open-source filesystem data-plane server as a background subprocess whenever `REBOOT_BLOB_DATA_PLANE_URL` is not already set (in Reboot Cloud, or via `--env`, it is — and then nothing is spawned), and point the application at it on localhost. The server picks its own ports and reports them through a ready file only once both its gRPC and HTTP endpoints are listening, so there is no port-allocation race and the application can never observe a half-started data plane. Blob bytes live under the application's state directory, so `rbt dev expunge` removes them along with the rest of the state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Uploading a file to a `Blob` from the browser means driving the whole multipart protocol — fetching upload instructions, `PUT`ting each part to its URL, reporting ETags, committing, and polling for the result — plus resuming after a dropped connection. Before this change an application author had to write all of that by hand against the generated client. Add `@reboot-dev/reboot-std-react/blobs`: - `useBlobUpload()` — the dead-simple case: given a blob id (from an application RPC, since creation is app-mediated) and a `File`, it uploads every part directly to the data plane, resumes already- confirmed parts, reports progress, and commits. - `BlobUploader` — the same machinery for bytes that don't come from a `File` (media recorders, transforms), with explicit `putPart`/`commit` and a `writable()` stream. - `useBlobDownloadUrl()` — resolves to a URL for a committed blob (e.g. for an `<img src>`), plus a re-exported reactive `useBlob` so any participant — not just the uploader — can render live progress. The part-`PUT` response carries the part's ETag, which the browser must read to report it back; expose the `etag` response header through Envoy's CORS configuration so cross-origin uploads (including direct-to-S3 uploads on the Cloud) can see it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The chat-room example only sent text, so it demonstrated nothing about storing binary data. Give it file attachments, wired the way a real app would: message-first, with attachment bytes uploaded after the message is already visible. `Send` now takes a list of attachment descriptors; the servicer checks each against a per-attachment size limit, creates a `Blob` per attachment, embeds the blob ids in the immediately-published message, and returns them. The web frontend then uploads each file into its blob via `useBlobUpload`, and every participant renders the attachment off its reactive blob status — a progress bar while it uploads (visible to everyone, since progress lives on the blob's state), the image once committed. No end-user auth here, so blobs are created with an empty `owner_id` (anyone in the room may upload). The documentation snippets that embed the chat-room proto are regenerated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

An `Application` has always taken both `servicers` and `legacy_grpc_servicers`, but a `Library` could only contribute the first, so a library offering an interface that predates Reboot -- or one shared with something that does not speak Reboot -- had no way to bring it along. Its users had to be told to pass the servicer themselves, which is exactly the coupling a library exists to remove. `AbstractLibrary.legacy_grpc_servicers()` now sits beside `servicers()`, defaulting to none so that existing libraries are unaffected, and `Application` collects it the same way. Both lists go through the same validation as before, so a Reboot servicer offered here is still refused with the message pointing at `servicers()`.

Envoy removes `x-reboot-caller-id` from traffic whose caller IDs it does not trust, which is what lets `is_app_internal` authorizers believe the header. The removal was attached to a single route: the one matching `content-type` exactly `application/grpc`. A request arriving as `application/grpc+proto`, which gRPC permits and grpc-core sends, misses that match and falls through to one of the per-method prefix routes, which carried no removal at all -- nor did the `/` route, nor the websocket and HTTP routes. The header arrives intact, and an application ID is not a secret: it is derived from the application's name. What a caller may claim about itself should not depend on which route their request happens to match, so the removal now belongs to the route configuration, which covers every route in it. This does not arise on Reboot Cloud, where the public listener is configured to trust caller IDs because the proxies in front of it ensure they are truthful. It does arise under `rbt serve run`. While here, servers offer themselves on loopback rather than on every local network address. Envoy reaches them from the same host, so loopback is enough, except in the two cases where the caller is outside it: Envoy running in a Docker container, and Reboot Cloud, where the platform reaches a server from outside its pod. Binding every address is not itself much of an exposure -- anything on the machine can already read the application's database off disk -- but there is no reason to offer what nothing needs.

`meta.json` was read back as a bare `dict` and passed around as one, so every reader spelled its keys out by hand -- `meta.get("committed", False)`, `meta["upload_id"]`, `part["size"]` -- and a mistake in any of them was a runtime `KeyError` rather than something `mypy` could catch. The shape was also only discoverable by reading whichever writer happened to produce it. `BlobMetadata` and `PartRecord` now carry it, and `from_json` / `to_json` confine the `dict` to the one boundary where JSON makes it the right shape. The on-disk format is unchanged: absent fields are still omitted rather than written as `null`, so a store written by an earlier build reads back identically.

The download handler took care to read part files off the event loop, but the upload handler beside it wrote them from it: `f.write(chunk)` and `os.fsync()` ran inline while a part -- megabytes of it -- came off the wire, stalling every other request the worker was serving. `aiofiles`, already a dependency of this repo, now backs both byte paths. The download loop keeps the same one-hop-per-chunk cost it already paid and just says so more directly; the upload loop stops blocking. `fsync` stays an explicit `asyncio.to_thread`, since `aiofiles` does not wrap it. Deliberately not applied to `_store.py`. Its `asyncio.to_thread(sync)` calls each carry a whole operation -- `complete()` reads every part and digests it -- so one hop is amortized across all of that work. Per-call `await`s there would turn a single hop into thousands and break up sequences that are easier to reason about whole.

…tion The data plane was a program of its own: a gRPC server, an HTTP server, and a `meta.json` per blob, started as a subprocess and reached over localhost. The application then reverse-proxied byte traffic to it, so every uploaded and downloaded megabyte went through the application's Python on its way somewhere else in the same process tree. Being alone was also the only thing that made it correct: one `asyncio.Lock` ordered a part's publication against a commit, which holds exactly while one process serves every request. It is now three ordinary parts of the application that uses it. The `BlobDataPlane` gRPC service is a legacy gRPC servicer, so the control plane still speaks the one interface every data plane implements and Reboot routes to it by name -- no address to configure, and none that could be, since the servers hosting it are not running when an application's environment is composed. Its byte endpoints are routes on the application's own HTTP server, so bytes arrive on the origin they were always going to, with no hop in between. And its metadata is `StoredBlob`, a state machine. That last one is what earns the change rather than merely tidying it. A part's bytes are written by whichever of a replica's servers served the upload, and whether that part is *in* the object is now decided in one place all of them share, so the answer no longer depends on a lock local to one process. A part is written under a name nothing reads and published only once `StoredBlob` agrees it belongs; one that arrives after the commit is refused, and its bytes are dropped rather than left on top of what the recorded ETag describes. Completion no longer re-reads the parts either: it derives the object's ETag from the digests taken while the bytes were being written, so there is no window in which a part could change under it. Starting it moves with it. It was started from three places -- `rbt dev run`, `rbt serve run` and the test harness -- each with its own idea of where bytes go, its own "unless already configured" check, and its own teardown, and the two CLI paths needed a ready file, a port-reporting protocol and a separate copy of the crypto root keys just to sign URLs with. A data plane inside the application is started by the application, once, from `Application.run()`, which already knows where the run's state goes -- so blobs land beside it without anyone computing a path, and the server subprocesses `server_managers` forks never start one because they leave `Application.run()` earlier. Being started there is also what puts the two checks where the decision to store bytes locally is actually made. Reboot Cloud sets a data-plane URL of its own, so the check against it stays honest in one line rather than three. And the filesystem data plane keeps bytes on the disk of the replica running it, which a second replica cannot see: an application spread across replicas would serve a download or a 404 depending on where the request landed, and an upload's parts could go somewhere completion never looks. `LocalConfigExtractor` already parses `REBOOT_REPLICA_CONFIG` to plan placement, so it now offers the count, and anything above one is refused. Servers are not counted and do not matter: every server of a replica shares that replica's disk, so `rbt dev run`'s two and `rbt serve run`'s one per core are as fine as they always were. Cloud's own check does not subsume this one -- a customer replicating an application themselves is exactly the case that would otherwise fail quietly. `_proxy.py` goes, and with it the forwarded-path machinery it existed for: a data plane inside the application has nothing to forward to. The standalone server goes. Splitting `data_plane_py` out of `blob_py` is what makes the runtime's side of this legal: the data plane stores and serves bytes and does not know what a `Blob` is, so it can sit below `//reboot/aio` where `blob_py` -- which does know -- cannot.

Review of the `Blob` API found three places where a field's default value was carrying meaning. `uploader_id` used the empty string to mean "anyone who knows this blob's ID may upload", on the state, on `Create` and on `Info`, which asks a reader to know that an empty string is a decision rather than an omission. All three are `optional` now, and the authorizer asks whether the field is there rather than whether it is empty. `downloader_ids` is `downloaders`, since it holds a `Downloaders` whose own field is already `user_ids`: `downloaders.user_ids` says once what `downloader_ids.user_ids` said twice. And `DataPlaneCompleteUploadResponse` carries a `oneof`. Completion either finished the object or did not, and the pair of an `etag` that was "empty when `error` is set" alongside an `optional error` left both of those states expressible at once.

Both of the blob tests' waiting helpers read in a loop with a 50ms sleep between attempts: one for the upload session the `BeginUpload` workflow provisions, one for a blob reaching a status. Reviewing them pointed out what the repo already asks for -- `reactively()`, which returns when the state changes instead of on the next tick. Neither loop takes a deadline, before or after: the test's own timeout already covers a condition that never becomes true, and a second, lower-level one would only add a way to fail under load.

The `Application` snippet carried three lines about blob data planes, which review found dense for a page whose subject is `Application` -- the TypeScript tab beside it shows the same construction in four lines. The example says why the library is there and no more. What the comment was the only home for has somewhere better to be. The page never said what `libraries` is at all, though the snippet has always passed one, so it says so now, and the data plane and `REBOOT_BLOB_DATA_PLANE_URL` are described there in prose rather than inside the code the reader is meant to be reading.

Review found the extracted range one line short: it stopped at `return;` on line 200 and left behind the `}` that closes `if (aborted !== undefined) {` on line 201, so the rendered block opened a branch it never closed. Nothing was going to catch that. `markdown-autodocs` copies the range out verbatim without parsing it, and Docusaurus renders an unbalanced fenced block as the inert text it treats every fenced block as. The regenerated snippet is committed alongside the range, since changing the range alone does not rewrite the body.

Drop Reboot Cloud implementation details from two comments: the `data_plane_proto` comment in `rbt/std/blob/v1/BUILD.bazel` named the Cloud facilitator as the thing that hosts the data plane via `legacy_grpc_servicers`, when any Reboot application can, and the `_CONFIGURATION_RETRY_SECONDS` comment in `blob.py` named who spawns the data plane, when all that matters is that it is normally already running at the address `REBOOT_BLOB_DATA_PLANE_URL` names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8vHXG2KLHDUruv8642AXY

- `data_plane.proto`: remove `forwarded_paths`, `ForwardedPath`, `HttpMethod` and `http_port` from `Configuration`. Before this change the option survived the removal of the byte-route proxying it configured, guarded by a refusal at startup; the feature was never released, so nothing needs the option or the fallback. - `reboot.aio.tests.Reboot.up()`: run `pre_run` for a Node.js application's Python-native libraries, as `NodeApplication.run()` does. Before this change the harness skipped every library's `pre_run` under Node.js, so a library refusing to serve such an application (the blob library, without a data plane URL) was refused under `rbt dev run` but silently started in tests. - `tests/reboot/std/blob/v1/blob_tests.ts`: show that a Node.js application registering the blob library without `REBOOT_BLOB_DATA_PLANE_URL` is refused at startup, with the message naming the variable. - `.claude/rules/python-annotate-return-types.md`: record that every function gets a return type annotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8vHXG2KLHDUruv8642AXY

Before this change, `Send` was declared a bare `transaction: {}`, which `main` no longer accepts: since the rebase, every transaction has to declare whether it takes the lock on its own state `exclusive` or `shared`. `Send` appends the message to its own state, so it takes the lock exclusive, as the generator's own error message suggests for a transaction that writes its state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D8vHXG2KLHDUruv8642AXY

rjhuijsman force-pushed the rjh.blob-storage-v2 branch from 1574235 to 6f904f0 Compare August 3, 2026 15:49

rjhuijsman commented Aug 3, 2026

View reviewed changes

rjhuijsman force-pushed the rjh.blob-storage-v2 branch from 842a135 to b0c684d Compare August 24, 2026 15:47

rjhuijsman commented Aug 25, 2026

View reviewed changes

rjhuijsman force-pushed the rjh.blob-storage-v2 branch from 263b706 to 23d4ee1 Compare August 25, 2026 12:17

rjhuijsman commented Aug 25, 2026

View reviewed changes

Comment thread documentation/docs/learn_more/call/from_react.mdx Outdated

rjhuijsman force-pushed the rjh.blob-storage-v2 branch 2 times, most recently from ef66db2 to c306853 Compare August 25, 2026 13:10

rjhuijsman marked this pull request as ready for review August 25, 2026 13:17

rjhuijsman self-assigned this Aug 25, 2026

rjhuijsman requested review from benh and a balanced review from Copilot August 25, 2026 13:18

Copilot started reviewing on behalf of rjhuijsman August 25, 2026 13:18 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes