Repository navigation
Conversation
Accept streamed file bodies with storage backpressure. Node stages on the target filesystem and publishes by rename; reserved staging paths support exact-key cleanup after killed writers. R2 uses bounded multipart parts for generated unknown-length bodies and retains the prior object on failure. Restores the reviewed streaming foundation from its preserved archive. Verified 20 file-store tests, platform and deployment typechecks, and scoped lint; checked current Cloudflare API references. Preserve original errors and cleanup chains.
Stream one ordered NDJSON artifact to durable storage and transient position-addressed readers. Keep metadata publication behind complete file writes, renew active leases, and preserve durable recording when live appends fail. Expose authenticated attach-only byte reads without active-run UI changes. Abort failed durable readers, stop heartbeat races, and wait for artifact publication before clean live EOF; cover lifecycle, backpressure, ownership, retries and retention with focused tests.
# Conflicts: # packages/gateway/src/dump/types.ts
Empty heartbeats use the last acknowledged offset independently of the ordered data writer. This keeps a live run from expiring while durable reads or append acknowledgements are pending, without changing bytes or data offsets. Verify long native-reader gates and concurrent heartbeat positions; stream end still awaits the in-flight heartbeat.
…eline-live-recording
This was referenced Oct 1, 2026
# Conflicts: # README.md # packages/gateway/__tests__/control-plane/dump_test.ts # packages/gateway/__tests__/dump/run-sink_test.ts # packages/gateway/__tests__/repo/dump-store_test.ts # packages/gateway/__tests__/scheduled_test.ts # packages/gateway/__tests__/vitest.setup.ts # packages/gateway/src/control-plane/dump.ts # packages/gateway/src/dump/run-sink.ts # packages/gateway/src/dump/store-contract.ts # packages/gateway/src/dump/types.ts # packages/gateway/src/repo/dump-store.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #476; remains draft until that dependency lands.
Streams encoded pipeline diagnostics into durable storage and the same bytes into a temporary LogStream. The portable writer now lives in
@floway-dev/dumpas one functionalcreateRunRecorderfactory. Gateway binds its storage/publication/live/background ports and retains admission, IDs, attribution, drain, HTTP measurement, SQL adapters and authenticated routes.Event batches stay ordered; producers follow durable backpressure. Lost acknowledgements retry at the same byte position, while live-service failure preserves durable recording. Independent empty heartbeats keep active streams alive during quiet upstream periods or slow writes. The metadata callback drains owned work and flushes the encoded tail before sampling completion time. Encoding/storage/metadata failures preserve original errors and abort owned writing.
Streamed FileStore writes publish atomically on Node and R2. Staged-file leases remain active until storage completes. Durable storage and metadata publication precede clean live EOF; publication failure retains the artifact and rejects with its original error. The authenticated live route attaches to an existing per-key stream with byte offsets and framed completion. Active-request discovery and realtime dashboard UI remain deferred.
Read FileStore contracts/implementations,
packages/dump/src/recorder.ts, the thin gatewaydump/run-sink.ts, SQL storage/leases, then the authenticated live route.Validation at
83254c30b: all eight exact-head CI checks passed. All selected local root checks passed, including 631 files / 6862 tests with four workers, all typechecks, genuine installers (106 passed, 4 host skips), repository checks and Web build/output assertions. The ordinary local full run hit an unchanged UI test timeout; source/assertions/timeouts remain unchanged and the complete reduced-concurrency suite passed. Focused tests cover native publication gates/rejection, metadata-factory abort, exact concurrent UTF-8 bytes, backpressure, retry, heartbeat and authenticated readers. Independent extraction review found no remaining issues; earlier native workerd storage probes remain applicable to unchanged adapters.