feat: chunk-aligned edge caching of object bytes via the Cache API

[alukach]

Implements #188.

What

Ranged object GETs on public products are served through a worker-controlled, per-PoP chunk cache built on the Cache API (the same mechanism as the Source API metadata cache, extended to data):

1. Resolve + authorize (unchanged — the gateway pipeline runs on every request, hit or miss).
2. Normalize the client's range to 4 MiB aligned blocks; look each block up in the Cache API and fetch misses from the backend as ranged GETs, caching them as 200s (the Cache API refuses 206s) with an immutable TTL.
3. Assemble the client's exact range and respond 206 — buffered for a single-chunk read, streamed for a multi-chunk read.
4. Only ranged reads are cached. Full-object GETs and bytes=0- bulk transfers bypass to the direct stream. Spans > 32 MiB also bypass, bounding subrequests and memory.

How

• Integration point: ChunkCachingBackend wraps multistore's WorkerBackend (ProxyBackend::forward). Key enabler: GETs are presigned as query-string SigV4 with Range unsigned and pass-through, so one presigned URL serves all chunk sub-fetches with different Range headers — no re-signing, no multistore changes, and authz/analytics/error mapping stay byte-identical.
• Ranged-only eligibility (eligible): object GET, carries a Range header, non-conditional, on a product whose anonymous view is is_public(). Cloud-native geospatial reads (COG tiles, GeoParquet footers, PMTiles directories, Zarr chunks) are all ranged and carry the cross-client reuse the cache exists for; full-object GETs are bulk downloads (aws s3 cp, rclone) — throughput-bound, single-touch, worst-case for assemble-then-respond — so they stay on the direct stream. bytes=0- (open-ended from zero) is a full-object transfer wearing a Range header and bypasses the same way. Private bytes can never enter or leave the cache.
• Range-sliced hits (cache_get_chunk): a warm mid-chunk read asks the Cache API for just the requested bytes of the cached block (cache.get with a Range header → Cloudflare returns a server-side-sliced 206) instead of pulling the whole 4 MiB into the isolate and slicing there. Falls back to slicing in-wasm if a runtime returns the full 200. The chunk stays 4 MiB, so a cold miss still over-fetches the whole block — that over-fetch is deliberate read-ahead for adjacent tiles/blocks (COG grids, vsicurl scans); only the warm read-out, which has no prefetch value, is trimmed.
• Streamed multi-chunk assembly: a range spanning >1 chunk is assembled through a bounded-concurrency ordered stream (futures::stream…buffered(N) → Response::from_stream, the same primitive the direct S3 passthrough already streams through) so the first bytes flush after the first chunk resolves instead of after the whole span buffers. Single-chunk reads keep the buffered path (exact HIT/MISS, no stream setup). Ordered emission + per-chunk length/ETag validation mean no chunk is ever yielded before it's validated; the trade is that once bytes have flushed, a later-chunk failure truncates the response (client retry) rather than bypassing.
• Cache keys carry content identity only: https://{host}/.chunk-cache/v1/{account}/{product}/{key}?etag=…&cs=…&i=N. ETag in the key makes overwrites self-invalidating; chunk size in the key makes tuning self-invalidating; auth material never appears.
• Consistency: per-object meta (ETag + length + the origin's entity headers, 60 s TTL) is learned via a 1-byte probe (bytes=0-0); every chunk fetch carries If-Match. An overwrite mid-read produces a 412 → meta invalidated → passthrough. Mixed-generation stitching is structurally impossible. Weak/missing ETags bypass. A cached chunk whose length disagrees with the object meta is poison — evicted and re-fetched (or, on a range-sliced 206, detected via the Content-Range total).
• Header fidelity: the probe captures the origin's entity headers into the meta entry, and a cache-served response replays them (content-type, content-encoding, last-modified, etag, …) before overriding only range framing + x-cache. A HIT is header-identical to the direct path.
• Failure posture: any cache error on the single-chunk/setup path degrades to a direct backend fetch; cache.puts ride ctx.waitUntil and never block or fail the response.

Observability & rollout

• x-cache: HIT | MISS | BYPASS + x-cache-chunks on responses. Single-chunk reads report exact hits/total (e.g. 1/1); multi-chunk streamed reads report stream/N and x-cache: MISS — headers must flush before the chunks resolve, so per-chunk hit precision isn't available there (a conservative floor; the majority small footer/tile reads are single-chunk and reported exactly). Streamed 206s use Transfer-Encoding: chunked (no Content-Length; Content-Range is present).
• Analytics gains append-only blob10 = cache_status, normalized to the exact HIT/MISS/BYPASS tokens so a CDN-fronted backend's own x-cache can't pollute the taxonomy.
• CHUNK_CACHE_ENABLED gates everything: staging + PR previews on, prod off. Flip the prod var after a staging soak; kill switch = flip it back.

Performance characteristics

Chunk caching trades bandwidth (over-fetching whole chunks on a cold miss — deliberate prefetch) for reuse (origin offload on repeated reads). This revision (2559768) keeps that gain while removing the two losses the first cut measured — warm read amplification and large-range TTFB — by range-slicing hits and streaming multi-chunk assembly, and by caching only ranged reads. A/B vs the staging baseline (no chunk cache) on a public 133 MB GeoParquet, warm medians (proxy-perf/scripts/staging_cache_scenarios.py):

scenario	first cut	this revision	what changed
origin fetch ms, repeated 64 KiB	9.8× less	~8× less	offload preserved
warm TTFB, 64 KiB ranged GET	1.0× (even)	~1.8–2.5× faster	range-sliced hit — cache now beats origin RTT on a warm ranged read
geopandas bbox windowed read	0.7×	~1.0× (parity)	warm read amplification gone
throughput, 16 MiB ranged GET	0.7×	~0.7–0.8×	streaming fixed TTFB, not sustained MB/s

The wins: origin offload holds (a HIT skips the S3 fetch entirely — worker-side server-timing: backend collapses from ~130 ms cold to single digits warm), and the two latency-bound scenarios that matter for cloud-native ranged access — warm TTFB and the real windowed read — are now a clear win and parity respectively, where the first cut was even/slower. The Cache API entry is shared across every request at a PoP, so hot ranges (parquet footer, COG header, popular tiles) are edge-served for every client after the first.

The remaining weak spot: large-range throughput stays a touch below origin (~0.7–0.8×) — the cache adds a re-serve hop that a straight origin stream doesn't, and streaming fixes first-byte latency but not sustained MB/s on a one-shot large read. This is the least relevant axis for the target workload (latency-bound ranged reads), and true bulk transfers (full-object, bytes=0-) now bypass entirely, so they keep origin throughput.

Verification

• 19 native unit tests over the pure range/chunk/key math (tests/chunk_cache.rs, incl. bytes=0- → From(0) which the bypass keys on); analytics blob10 + token-normalization tests (tests/analytics.rs).
• 10 integration tests (tests/test_integration.py), running in CI with CHUNK_CACHE_ENABLED=true, including the four new behaviors: full-object GET bypasses, bytes=0- bypasses, a multi-chunk range streams byte-identically to the same slice fetched directly, and overlapping suffix ranges (bytes=-100 then bytes=-8) resolve from the same cached tail chunk with a warm HIT (footer reuse).
• Exercised end-to-end under wrangler dev (miniflare's real Cache API) and on the live PR preview:
  • cold chunk → MISS, warm re-read → HIT with backend;dur in single digits (origin skipped)
  • single-chunk warm HIT reports x-cache-chunks: 1/1; 8 MiB multi-chunk read reports stream/2 with Transfer-Encoding: chunked and a correct Content-Range, assembling byte-identically
  • full-object GET and bytes=0- → BYPASS; suffix-range footer reuse confirmed; past-EOF → BYPASS to origin 416; conditional If-None-Match → 304 via direct path

Open follow-ups (from the issue)

• HEAD responses riding a short-TTL cache (the meta entry already exists to build on).
• Extending to private products (authz already runs per request; needs a cache-key hygiene review first).
• Request collapsing for cold-object bursts (same class as Federated credential cache is per-isolate; consider cross-instance caching (KV/DO) + single-flight #148).
• A workload-shape bypass for isolated small cold reads was considered and deferred — it would forfeit the cold over-fetch's prefetch value; revisit only if telemetry shows single-touch cold reads dominate a real dataset.

🤖 Generated with Claude Code

[claude]

Claude finished @alukach's task in 4m 30s —— View job

---

Review complete

• Gather context (diff, changed files, CLAUDE.md)
• Review src/chunk_cache.rs (core logic)
• Review src/lib.rs, src/analytics.rs, src/config.rs changes
• Review tests
• Post findings

---

✅ No blocking issues — safe to merge.

Walked through the range/chunk math, the cache-key construction, the ETag/If-Match consistency story, the buffered vs. streamed response paths, and the eligible/chunk_plan gating in src/lib.rs. Checked for the usual suspects (integer under/overflow in the range math, off-by-ones in chunk_bounds/chunk_index_range, cache-key collisions from the percent-encoding, header-injection via replayed origin headers, and conditional-request handling) — all held up, and the 19+10 unit/integration tests cover the sharp edges well (suffix-overflow, inverted ranges, ETag rotation, multi-chunk streaming).

Two minor, non-blocking observations:

• src/chunk_cache.rs:507-525 (resolve_chunk): when a backend chunk fetch succeeds but fails length validation (full.len() as u64 != expected, line 514), the error propagates up through serve_chunked's ? and the whole request falls back to inner.forward on the original client request — meaning the backend gets hit twice (once inside resolve_chunk, once on bypass) for that read. Rare (only on a genuinely inconsistent backend response), but worth knowing if you ever see doubled origin requests correlated with BYPASS in a metrics dump.
• src/analytics.rs:196-206 (cache_status_token call site): when chunk_plan is None (private product, feature disabled, HEAD, etc.) the wrapper never touches x-cache, so if the underlying backend ever emits its own literal x-cache: HIT (unlikely for S3, but not impossible for another object_store backend), it would pass through cache_status_token unfiltered and get misattributed to the chunk cache in blob10. Not reachable with the current S3-only backend, so purely a forward-looking note given the PR discussion about supporting more backends.

Nothing here blocks merge — both are edge cases already implicitly acknowledged by the PR's own "fails closed" / conservative-floor design philosophy.

[github-actions]

🚀 Latest commit deployed to https://source-data-proxy-pr-189.source-coop.workers.dev

• Date: 2026-07-19T20:45:41Z
• Commit: 027de30

[alukach]

Design note: making edge caching conditional per product

Follow-up question that came up: could edge caching be opt-in per product rather than "all public products"? Short answer — it's cheap on the proxy, because the decision point already has the product object in hand. The chunk_plan gate (src/lib.rs, ~L269) already fetches the full SourceProduct (edge-cached via PRODUCT_CACHE_SECS) and branches on it:

get_or_fetch_product(...).await.ok()
    .filter(|p| p.is_public())
    .map(|_| CachePlan::new(&host, &parts.path, ctx.clone()))

Making it product-conditional = adding one predicate right here. No new fetch, no extra round-trip. The real question is where the opt-in bit comes from, not the plumbing.

Three ways to source the decision

A. Env-var allow/deny list — proxy-only, shippable now.
Add CHUNK_CACHE_PRODUCTS (or _ACCOUNTS), parse to a set, gate is_public() && allowlisted(account, product). ~10 lines, no schema change, no other service.

• Control: per-product / per-account on/off; trivially invertible to opt-out (default-on with a kill-list).
• Change latency: a wrangler vars push + redeploy (the deploy pipeline already runs wrangler secret bulk) — minutes, ops-managed.
• Good for: the staging→prod canary, piloting named products, or yanking a misbehaving one. Not self-service.

B. A product flag from the Source API — the "real" opt-in.
The gate already deserializes SourceProduct, and serde ignores unknown fields, so once the API exposes a field (e.g. metadata.edge_cache: true, or a tags array), the proxy adds it to the struct and the gate becomes p.is_public() && p.edge_cache_opted_in() — a few lines here.

• Control: self-service per-product, set by product owners.
• Change latency: no deploy; effective within the product-cache TTL (mind the known product/data-connection cache staleness).
• Cost: lives in the control plane (the source.coop API — separate repo/team): add the field, expose it on the product GET, give owners a way to set it. Heavier, cross-team lift; the proxy side stays trivial.

C. Per-product policy (not just on/off).
CachePlan is already the per-request carrier, so per-product parameters (chunk size, max span, TTL, cache-full-objects, tail-only) could thread through it from the product object. Same data path, more surface. Moderate; only worth it if we want tuning, not a toggle.

Level of control achievable

Control	Difficulty
Per-product on/off	Easy (A or B)
Per-account on/off	Easy (A or B)
Opt-in or opt-out (default either way)	Trivial, both directions
Toggle without a deploy	Only via B (product metadata), TTL-bounded
Per-prefix / object-pattern	Moderate (gate sees the path; needs match logic)
Per-product tuning (chunk size / TTL / span / full-object)	Moderate (Option C)

The one real dependency

Today the product's metadata holds only mirror/connection routing — there's no free-form tags field to hang a flag on. So a self-service per-product opt-in (Option B) needs a Source API schema addition. Encouragingly, the product already carries per-product control bits the proxy reads (disabled, visibility), so an edge_cache flag is the same mechanism as those — a well-trodden path on the control-plane side, not a novel one.

Why it's low-risk either way

The gate already fetches the product, already branches, and CachePlan = None is an existing, tested "cache fully off" state that the wrapper backend cleanly delegates through (every ineligible request exercises it). A new predicate composes with is_public() and adds no new risk surface. Cache keys are per-product-path, so a partial rollout can't contaminate products that aren't opted in.

Bottom line: ops-managed named lists are a few hours entirely within this repo (Option A). Self-service per-product opt-in is also easy on the proxy but gated on a small Source API schema change (Option B) — and we can ship A now and migrate to B later without reworking the gate, since both are just predicates in the same spot.