From c15bda7759496b74cb3754be9531d103de4fd376 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:22:35 +0200 Subject: [PATCH 01/32] Add the plan for block-granular Caterva2 reads Analysis of what it would take to give C2Array the block-granular reads blosc2.open(url, lazy=True) got in #701, and where the work goes. Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 314 ++++++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 plans/cat2-block-granularity.md diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md new file mode 100644 index 000000000..4e6addb51 --- /dev/null +++ b/plans/cat2-block-granularity.md @@ -0,0 +1,314 @@ +# Block-Granular Reads For Caterva2 And `C2Array` + +Analysis and plan — nothing implemented. Written 2026-08-17, after +[plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs +(merged as PR #701). + +## The question + +`blosc2.open(url, lazy=True)` now fetches only the blocks a slice touches, and +gets 5-17x against an object store for the shapes that suit it. `C2Array`, the +other remote path, still fetches whole chunks: it reads them through Caterva2's +`api/chunk` endpoint, which is indexed by chunk number and has no notion of a +byte range. What would it take to give it the same thing, and where should the +work go — into blosc2, or into the Caterva2 server? + +**Verdict: into the client, mostly.** Caterva2 already serves byte ranges +properly for stored datasets, including authenticated ones, so no new endpoint +is needed. What is missing is a `read_range` on `C2Array`, a way to tell which +datasets support it, and a connection pool — plus two small server changes that +make the arrangement honest rather than lucky. + +## What was verified + +Against `https://cat2.cloud/demo` and against a local server run from +`~/ironArray/caterva2` (`CATERVA2_SECRET=c2sikrit cat2-server`, serving +`_caterva2/state/public/`). + +### Caterva2 serves real byte ranges, for free + +`fetch_data` (`caterva2/services/server.py:581`) returns `FileResponse` when the +whole of a stored dataset is asked for: + +```python +if ( + whole + and not isinstance( + array, blosc2.LazyArray | hdf5.HDF5Proxy | blosc2.NDField | blosc2.CTable + ) + and not filter + and inner_key is None +): + return FileResponse( + abspath, filename=abspath.name, media_type="application/octet-stream" + ) +``` + +Starlette's `FileResponse` implements RFC 7233 on its own: `Accept-Ranges: +bytes`, 206 with `Content-Range`, and `_handle_single_range` does +`await file.seek(start)` and reads only `end - start` bytes. So Caterva2 +inherits ranges from that one `return`. Measured on `kevlar-tomo.b2nd` +(14.44 MB, 10 chunks of ~1.44 MB, 47 blocks each), over the network to the demo: + +| request | median | bytes | +|---|---|---| +| full fetch | 1.654 s | 14.44 MB | +| **range 32 KB** | **0.085 s** | 0.03 MB | +| range 1.4 MB | 0.228 s | 1.44 MB | +| `api/chunk`, one chunk | 0.234 s | 1.37 MB | + +A small range costs 5% of the full fetch, and a large one costs what the same +bytes cost through `api/chunk`. Nothing is materialized per request. + +**Multipart works too**: `Range: bytes=0-31, 1000-1063, 5000-5099` returns 206 +`multipart/byteranges` (566 bytes for the three spans). No object store offers +that, and `block_plan` already computes exactly the coalesced range list it +wants — see phase 4. + +**Authentication composes.** On the local server, a `@personal` dataset uploaded +by a logged-in user answers a ranged request with 206 when the cookie is sent and +401 without it. The `FileResponse` branch runs after `split_and_resolve(path, +user)`, so ranges and auth are orthogonal. This is the fact that decides the +design: fsspec's HTTP filesystem cannot carry that cookie, and `C2Array` already +holds it. + +### The branch is narrow, and missing it is silent + +Everything else returns `StreamingResponse`, which has no range support at all. +Measured, on both servers: + +``` +GET api/fetch/@public/kevlar-tomo.b2nd?slice_=0:1 Range: bytes=0-31 + -> 200, 1,364,328 bytes, no Content-Range, no Accept-Ranges + +GET api/fetch/@personal/doubled.b2nd (a lazy expression) Range: bytes=0-0 + -> 200, 2,423,617 bytes +``` + +So lazy expressions, HDF5-backed datasets, `.b2z`/`.h5` members, `field=` and +`filter=` queries, and peer/provider datasets answer a one-byte request with the +whole body. `api/download` is the same. And clients do not defend themselves: +fsspec's HTTP filesystem, asked for 32 bytes of the first case, **returned +1,364,328 bytes in 2.7 s** and reported no error. A byte-range reader pointed at +such a dataset degrades into N full downloads while appearing to work. + +### `api/info` already says which is which + +A stored dataset reports `chunks`, `blocks` and `schunk`; a computed one reports +`expression` and `operands` and none of those: + +``` +@personal/personal.b2nd keys=['blocks', 'chunks', 'dtype', 'mtime', 'schunk', 'shape'] +@personal/doubled.b2nd keys=['dtype', 'expression', 'mtime', 'operands', 'shape'] +``` + +`C2Array` fetches `api/info` at construction already, so the discriminator costs +nothing. It is necessary but **not sufficient**: an HDF5 leaf or a `.b2z` member +may well report a geometry and still be served by `StreamingResponse`. The +authority has to be the status code of the first range read — see phase 2. + +### The prototype + +A ~40-line adapter, with no changes to blosc2 at all, reading a local +authenticated dataset block by block: + +``` +opened over Range+auth: (600, 600) float64 chunks=(300, 600) blocks=(30, 600) blocks/chunk=10 + server honours ranges (probe): True + opening cost: 4 requests, 253 bytes +slice [10:12, 30:40] via blocks: 2 requests, 121,214 bytes correct=True +the same data as C2Array does it: 1 request, 1,211,720 bytes +``` + +Ten blocks per chunk, ten times fewer bytes. End-to-end over the network, on the +demo's `kevlar-tomo.b2nd` with blosc2's shipped 1 MB threshold and no patching: +**1.364 MB in 1.477 s (chunks) against 0.015 MB in 0.269 s (blocks)**. + +It worked because `FsspecNDSource` never uses fsspec directly past construction: +it asks its filesystem object for `isdir`, `ukey` and `cat_file(path, start=, +end=)`, which is a three-method interface anything can implement. The prototype +supplied one over `httpx` and monkeypatched `blosc2.core._import_fsspec` to hand +it over. That monkeypatch is the only part that needs a supported replacement — +see phase 3. + +### `C2Array` builds a new HTTP client per request + +`_xget` calls `_httpx().get(...)`, and `_httpx()` returns the *module*, so every +request opens a connection and negotiates TLS. On the same chunk over the +network: + +``` +C2Array.get_chunk(0) 0.886 s (1.35, 0.89, 0.81) +same URL, pooled client 0.513 s (1.20, 0.51, 0.42) +``` + +~0.37 s per request of pure setup. Block mode issues *more, smaller* requests, +which is exactly what per-request handshakes punish. `aget_chunk` already keeps a +reused `AsyncClient`; the sync path never got the same treatment. + +## What not to build + +**A block endpoint** (`api/block/{path}?nchunk=&nblock=`). It duplicates what +Range already does for stored data, and for computed data it solves the wrong +problem: a block only exists as bytes when the array is stored in blocks, so for +a lazy expression or an HDF5 proxy the server must compute a whole chunk before +it can hand one over — network saved, server work unchanged. Caterva2 already has +the better primitive there, and it is finer than blocks: `slice_` asks for +exactly the region wanted and computes only that. `C2Array.__getitem__` uses it +already. + +The rule that falls out, and that the plan follows throughout: + +> **Blocks where the bytes already exist in blocks; `slice_` where they must be +> computed.** The split is exactly the `FileResponse` / `StreamingResponse` line +> the server already has. + +## Plan + +### Phase 1 — Pool the HTTP client in `C2Array` (caterva2) + +Smallest change, largest certain payoff, and independent of everything else. +`_xget`/`_xpost` should share a module-level `httpx.Client` (thread-safe, with +`limits=` set) instead of calling `httpx.get`. Measured: ~0.37 s per request +saved over a WAN link, on the *existing* chunk path. + +Care: the client must not capture auth headers globally (they are per call), and +a long-lived client needs `timeout` and connection limits configured. Closing it +at interpreter exit is nice but not required. + +**Do this first even if the rest is never built.** + +### Phase 2 — A capability check on the client (caterva2) + +`C2Array` gains a private `_ranges_ok` state with three values: unknown, yes, no. + +- From `api/info`, already fetched at construction: no `chunks`/`blocks`/`schunk` + (i.e. an `expression` dataset) means **no** without any request. +- Otherwise unknown, resolved by the first range read: 206 with a `Content-Range` + means **yes**; a 200 means **no**, and the source falls back to whole chunks for + the life of the object. + +The fallback must be permanent and silent-safe: never retry ranges on a dataset +that answered 200, or every read pays a full download to rediscover it. Note the +probe is expensive exactly where it fails (a 200 carries the whole body — 2.4 MB +for the lazy dataset above), which is why phase 5 matters. + +### Phase 3 — A supported seam in blosc2, then `C2Array` block support + +The prototype showed the machinery is already generic; what it lacks is a +sanctioned way in. Two options, in increasing order of tidiness: + +- **(a) A `fs=` argument on `FsspecNDSource`.** Two lines: when given, skip + `url_to_fs` and use the object as is. Anything with `isdir`, `ukey` and + `cat_file(path, start=, end=)` then works, `C2Array` included. Minimal, and + slightly dishonest — the class is named for fsspec. +- **(b) Split the class.** Lift the frame parsing and block planning + (`_read_frame_index`, `_chunk_extents`, `_block_extents`, `chunk_layout`, + `block_plan`, `wants_blocks`, `get_chunk`, `_special_chunk`) into a + `ByteRangeNDSource` base whose only abstract method is `read_range(offset, + size)`. `FsspecNDSource` becomes that base plus an fsspec transport; + `C2NDSource` becomes the same base plus an `httpx` transport that carries the + Caterva2 cookie. **Recommended**: there are now two implementations, which is + when a base class earns its place, and it puts the format knowledge in one + place where it has already needed four corrections (dictionaries, + variable-length blocks, non-extended headers, repeated-value chunks). + +On the Caterva2 side, `C2Array` then either implements the base itself or exposes +`read_range` and lets `blosc2.Proxy` do the rest. The URL is +`{urlbase}/api/fetch/{path}` with the auth cookie; `blocks_per_chunk` comes from +`self.blocks`, which `C2Array` already has. + +Note what does *not* change: the thresholds, the coalescing, the splicing into a +partially-filled cache chunk, the persistent cache and its bitmap, `max_concurrency` +— all of it is in `Proxy` and the base, and all of it is already tested. + +### Phase 4 — Multipart ranges (blosc2, optional) + +`block_plan` returns a coalesced list of ranges per chunk. Caterva2 answers +`multipart/byteranges`, so those could go out as **one request** instead of one +per run — a strictly better deal than any object store offers, and the thing that +would make Caterva2 the best backend for block reads rather than merely an equal +one. Needs a multipart parser on the client (~40 lines) and only pays where a +slice touches several disjoint runs of a chunk. Measure before building: with the +two-wave design a chunk's runs already go out in parallel, so this converts +parallel requests into one, which matters for per-request cost, not latency. + +Do not build this unless phase 5 is done: batching into a request shape the +server may stop honouring is a bad trade. + +### Phase 5 — Make the streaming paths honest (caterva2) + +Small, and it protects every future client: + +- set `Accept-Ranges: none` on the `StreamingResponse` returns in `fetch_data` + and `download_data`; +- answer a `Range` header on those paths with **416** (or 400) rather than + ignoring it. + +Then a client learns the answer in one cheap exchange instead of a full download, +phase 2's probe becomes nearly free, and the silent N-full-downloads failure mode +stops existing. If the range support is to be a documented feature rather than an +accident, this is the change that makes it one. + +## Risks and open questions + +- **The `FileResponse` branch is load-bearing but incidental.** Nothing in + Caterva2's tests asserts that a stored dataset is served by `FileResponse`, so + a refactor could turn it into a `StreamingResponse` and silently halve every + block client's performance. If phase 3 ships, Caterva2 wants a test that asserts + 206 and `Content-Range` on a stored `.b2nd`. +- **HDF5 leaves and `.b2z` members are unverified.** The code puts them on the + streaming path (`inner_key is not None`), but the local server returned 404 for + the HDF5 leaf paths I tried, so I could not confirm what they report in + `api/info`. If either reports `chunks`/`blocks` while being streamed, the + info-based discriminator alone would be wrong — which is why the 206 check is + the authority. +- **Peer/provider datasets** go through `provider.fetch` and a + `StreamingResponse`; same treatment, worth a look when peers matter. +- **Quota and accounting.** Many small ranged GETs replace one chunk GET. If + Caterva2 ever meters per request rather than per byte, block clients look + expensive while transferring far less. Worth deciding before it surprises + someone. +- **The gain depends on chunk size, as it does for S3.** blosc2 declines to take + a chunk apart below 1 MB compressed, so small-chunked datasets keep today's + behaviour exactly. `cube-1k-1k-1k.b2nd` (~102 KB per chunk) would never use + blocks; `kevlar-tomo.b2nd` (1.44 MB) always would. + +## Rough sizing + +| | where | size | +|---|---|---| +| 1. pooled client | caterva2 | ~10 lines | +| 2. capability check | caterva2 | ~30 lines + tests | +| 3a. `fs=` seam | blosc2 | ~5 lines | +| 3b. `ByteRangeNDSource` split | blosc2 | ~80 lines moved, ~20 new | +| 3. `C2Array` block support | caterva2 | ~60 lines + tests | +| 4. multipart | blosc2 | ~60 lines, only if measured | +| 5. honest streaming paths | caterva2 | ~10 lines + a test | + +## Reproducing the measurements + +A local server, which is what phases 2 and 5 need to be tested against: + +```sh +cd ~/ironArray/caterva2 +CATERVA2_SECRET=c2sikrit cat2-server & # serves _caterva2/state/public/ +CATERVA2_SECRET=c2sikrit cat2-admin adduser probe@example.com foobar11 +``` + +Then, for the range behaviour of any dataset: + +```python +import httpx + +r = httpx.get( + "http://localhost:8000/api/fetch/@public/kevlar-tomo.b2nd", + headers={"Range": "bytes=0-31"}, +) +print(r.status_code, r.headers.get("content-range"), len(r.content)) +# 206 'bytes 0-31/14435027' 32 -> served by FileResponse, blocks will work +# 200 None 14435027 -> served by a body builder, blocks must not be used +``` + +`bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any +local array, which is what decides whether a given dataset would benefit at all. From 7bb0eff1782b54a9788d1314cf9674ff4278f815 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:22:42 +0200 Subject: [PATCH 02/32] Pool the HTTP client C2Array requests go through httpx.get() builds a client, opens a connection and negotiates TLS per call. Against cat2.cloud that is 0.162 s per request against 0.046 s through a pooled client, on the existing chunk path -- 3.5x, and it grows in weight as reads get smaller, which block-granular reads will make them. The client is shared and thread-safe, which is what lets Proxy fan its fetches out, but it never keeps a cookie: auth belongs to the C2Array being read, and arrays with different tokens (or none) share the client. Login keeps a client of its own for the same reason. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 66 ++++++++++++++++++++++++++++- tests/ndarray/test_c2array_async.py | 6 ++- 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index ba85c2897..8d9874b86 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -7,7 +7,9 @@ from __future__ import annotations +import atexit import os +import threading from contextlib import contextmanager from typing import TYPE_CHECKING @@ -38,6 +40,64 @@ def _httpx(): return httpx +_client = None +_client_lock = threading.Lock() + + +def _forgetful_cookies(httpx): + """A cookie jar that never keeps anything, for the shared client. + + A client of its own per request could not carry a cookie from one request to + the next; a shared one can, and must not: the token belongs to the C2Array + being read, and arrays with different tokens (or none) share this client. A + `Set-Cookie` from any response would otherwise start authorizing requests + that asked for none. + """ + + class _NoCookies(httpx.Cookies): + def extract_cookies(self, response): + pass + + return _NoCookies() + + +def _sync_client(): + """The process-wide HTTP client every synchronous request goes through. + + `httpx.get()` builds a client, opens a connection and negotiates TLS for + each call and throws all of it away afterwards, which over a WAN link + measured ~0.37 s per request -- most of what a small read costs, and paid + once per chunk. A pooled client keeps the connection alive between + requests; it is thread-safe, which is what lets `Proxy` fan its fetches out. + + Auth stays per call rather than on the client: the cookie belongs to the + C2Array being read, and several of them (with different tokens, or none) + share this one client. + """ + global _client + if _client is None: + with _client_lock: + if _client is None: + httpx = _httpx() + # More connections than `Proxy`'s default concurrency, so that a + # caller raising it does not queue on the pool; keepalive covers + # the fan-out of one fetch, which is what there is to reuse + _client = httpx.Client( + timeout=TIMEOUT, + limits=httpx.Limits(max_connections=64, max_keepalive_connections=32), + cookies=_forgetful_cookies(httpx), + ) + return _client + + +@atexit.register +def _close_sync_client(): + global _client + if _client is not None: + _client.close() + _client = None + + @contextmanager def c2context( *, @@ -121,7 +181,7 @@ def _auth_headers(auth_token, headers=None): def _xget(url, params=None, headers=None, auth_token=None, timeout=TIMEOUT): headers = _auth_headers(auth_token, headers) - response = _httpx().get(url, params=params, headers=headers, timeout=timeout) + response = _sync_client().get(url, params=params, headers=headers, timeout=timeout) response.raise_for_status() return response @@ -129,7 +189,7 @@ def _xget(url, params=None, headers=None, auth_token=None, timeout=TIMEOUT): def _xpost(url, json=None, auth_token=None, timeout=TIMEOUT): auth_token = auth_token or _subscriber_data["auth_token"] headers = {"Cookie": auth_token} if auth_token else None - response = _httpx().post(url, json=json, headers=headers, timeout=timeout) + response = _sync_client().post(url, json=json, headers=headers, timeout=timeout) response.raise_for_status() return response.json() @@ -144,6 +204,8 @@ def _sub_url(urlbase, path): def login(username, password, urlbase): url = _sub_url(urlbase, "auth/jwt/login") creds = {"username": username, "password": password} + # Not the pooled client: this is the one request whose Set-Cookie matters, + # and it belongs to the caller rather than to every later request resp = _httpx().post(url, data=creds, timeout=TIMEOUT) resp.raise_for_status() return "=".join(list(resp.cookies.items())[0]) diff --git a/tests/ndarray/test_c2array_async.py b/tests/ndarray/test_c2array_async.py index af4978520..4556eaa5f 100644 --- a/tests/ndarray/test_c2array_async.py +++ b/tests/ndarray/test_c2array_async.py @@ -27,6 +27,8 @@ def json(self): class _FakeHttpx: + """Stands in for both the httpx module and the pooled sync client.""" + HTTPStatusError = Exception def __init__(self, meta): @@ -47,7 +49,9 @@ def fake_c2array(monkeypatch): "dtype": str(array.dtype), "schunk": {"cparams": {"typesize": array.dtype.itemsize}}, } - monkeypatch.setattr(c2array_mod, "_httpx", lambda: _FakeHttpx(meta)) + fake = _FakeHttpx(meta) + monkeypatch.setattr(c2array_mod, "_httpx", lambda: fake) + monkeypatch.setattr(c2array_mod, "_sync_client", lambda: fake) c2 = blosc2.C2Array("@public/fake.b2nd", urlbase="http://fake-server/") c2._chunks_source = array # stash the real array to serve chunk bytes from return c2 From 422ba4b49d513754d66d7d0696a6d567b1a914fe Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:26:33 +0200 Subject: [PATCH 03/32] Lift the frame reading out of FsspecNDSource into ByteRangeNDSource Everything FsspecNDSource knows about taking a Blosc2 frame apart -- the header, the chunk offsets, which chunks are worth reading block by block, where those blocks are and how few requests they fit in -- is the frame format and nothing else. It now lives in a base class whose only abstract method is read_range(offset, size); FsspecNDSource is that base plus four lines of fsspec, and C2Array will be the same base plus HTTP ranges. The frame index is read through read_range too, so a subclass has one seam to implement rather than two. That is what the test churn is: reads made while opening, and the read behind a whole-chunk fetch, now show up in the traffic the tests count. Co-Authored-By: Claude Opus 5 --- doc/reference/byterangendsource.rst | 19 +++++ doc/reference/classes.rst | 1 + doc/reference/fsspecndsource.rst | 9 +-- src/blosc2/__init__.py | 2 + src/blosc2/proxy.py | 105 +++++++++++++++++++--------- tests/test_fsspec.py | 29 +++++--- 6 files changed, 117 insertions(+), 48 deletions(-) create mode 100644 doc/reference/byterangendsource.rst diff --git a/doc/reference/byterangendsource.rst b/doc/reference/byterangendsource.rst new file mode 100644 index 000000000..af57c43f2 --- /dev/null +++ b/doc/reference/byterangendsource.rst @@ -0,0 +1,19 @@ +.. _ByteRangeNDSource: + +ByteRangeNDSource +================= + +A :ref:`ProxyNDSource` that serves the chunks -- and the single blocks -- of a +Blosc2 frame it can read byte ranges of, instead of transferring the whole +container. It knows the frame format and nothing about where the frame lives: +subclasses supply ``read_range(offset, size)`` and nothing else. +:ref:`FsspecNDSource` reads through fsspec, and :ref:`C2Array` reads over HTTP +ranges from a Caterva2 subscriber. For other sources, see :ref:`ProxyNDSource` +and :ref:`ProxySource`. + +.. currentmodule:: blosc2 + +.. autoclass:: ByteRangeNDSource + :members: + :exclude-members: all, any, max, mean, min, prod, std, sum, var + :member-order: groupwise diff --git a/doc/reference/classes.rst b/doc/reference/classes.rst index 1a33777fd..4efd1a25a 100644 --- a/doc/reference/classes.rst +++ b/doc/reference/classes.rst @@ -139,6 +139,7 @@ container APIs above. proxy proxysource proxyndsource + byterangendsource fsspecndsource simpleproxy embed_store diff --git a/doc/reference/fsspecndsource.rst b/doc/reference/fsspecndsource.rst index 399e0eb83..deb2a6fb6 100644 --- a/doc/reference/fsspecndsource.rst +++ b/doc/reference/fsspecndsource.rst @@ -3,10 +3,11 @@ FsspecNDSource ============== -A :ref:`ProxyNDSource` that serves the chunks of a Blosc2 frame living behind an -fsspec URL, reading each one with a range request instead of transferring the -whole container. For other sources, see :ref:`ProxyNDSource` and -:ref:`ProxySource`. +A :ref:`ByteRangeNDSource` that serves the chunks of a Blosc2 frame living +behind an fsspec URL, reading each one with a range request instead of +transferring the whole container. Everything about the frame format, block +granularity included, lives in the base class; this adds the fsspec transport. +For other sources, see :ref:`ProxyNDSource` and :ref:`ProxySource`. ``examples/ndarray/rw-fsspec.py`` is a runnable walkthrough of this and the other two ways to read an fsspec URL, and of writing one back. diff --git a/src/blosc2/__init__.py b/src/blosc2/__init__.py index 648014b58..835b1ac0d 100644 --- a/src/blosc2/__init__.py +++ b/src/blosc2/__init__.py @@ -593,6 +593,7 @@ def _raise(exc): ProxySource, ProxyNDSource, ProxyNDField, + ByteRangeNDSource, FsspecNDSource, SimpleProxy, jit, @@ -879,6 +880,7 @@ def _raise(exc): "NDArray", "NDField", "Operand", + "ByteRangeNDSource", "FsspecNDSource", "Proxy", "ProxyNDField", diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index feaa35491..3448fcb3e 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -1116,14 +1116,14 @@ def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: class _RangeReader: """The seek/read pair `_read_frame_index` needs, served by exact range requests.""" - def __init__(self, fs, path: str): - self._fs, self._path, self._pos = fs, path, 0 + def __init__(self, read_range): + self._read_range, self._pos = read_range, 0 def seek(self, pos: int) -> None: self._pos = pos def read(self, size: int) -> bytes: - data = self._fs.cat_file(self._path, start=self._pos, end=self._pos + size) + data = self._read_range(self._pos, size) self._pos += len(data) return data @@ -1152,17 +1152,11 @@ def _chunk_extents(offsets: np.ndarray, header: list) -> np.ndarray: return np.minimum(extents, header[8] + blosc2.MAX_OVERHEAD) -class FsspecNDSource(ProxyNDSource): +class ByteRangeNDSource(ProxyNDSource): """A :ref:`Proxy` source that serves parts of a remote Blosc2 frame. The frame stays where it is: only its header, its chunk offsets, and what a - slice actually touches ever cross the network. This is what - ``blosc2.open(url, lazy=True)`` builds; wrap it in a :ref:`Proxy` by hand - when the cache belongs at a path of your choosing rather than inside - ``cache_storage``:: - - src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") - a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") + slice actually touches ever cross the network. A chunk large enough to be worth taking apart is read block by block -- :meth:`chunk_layout` fetches the offsets of its blocks, :meth:`block_plan` @@ -1172,14 +1166,25 @@ class FsspecNDSource(ProxyNDSource): to save there; :meth:`wants_blocks` decides which is which without reading anything. + Everything above is the Blosc2 frame format and nothing else, so a subclass + only has to say how to read bytes: :meth:`read_range` is the one abstract + method, and the transport behind it decides nothing about the rest. + :ref:`FsspecNDSource` reads them with fsspec, and :ref:`C2Array` reads them + over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. + + A subclass sets its transport up first and then calls this constructor, + which reads the frame's header and chunk offsets through it (three small + reads). It may also set a ``stamp``, anything that names the exact bytes it + reads, so that :ref:`Proxy` can tell a cache built from other bytes. + Contiguous frames carrying a ``b2nd`` metalayer only, which is what :func:`blosc2.asarray` and friends write to a single file. Sparse frames and - ``.b2d`` stores are directories; open those with ``cache_storage=``. + ``.b2d`` stores are directories, and cannot be read this way. Parameters ---------- urlpath: str - The fsspec URL of the frame. + Where the frame is, for error messages and for the caller to read back. max_concurrency: int, optional How many fetches the enclosing :ref:`Proxy` may run at once. Every chunk or block costs one range request, so against an object store a slice is @@ -1189,28 +1194,15 @@ class FsspecNDSource(ProxyNDSource): the thread pool costs about 10 microseconds per chunk and saves nothing. """ - def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): - from blosc2.core import _import_fsspec + stamp = None + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.max_concurrency = max_concurrency - fsspec = _import_fsspec(urlpath) - fs, path = fsspec.url_to_fs(urlpath) - if fs.isdir(path): - raise NotImplementedError( - f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " - "chunk by chunk; open it with cache_storage= instead" - ) self.urlpath = urlpath - self._fs, self._path = fs, path - # Identifies the remote bytes, so a cache built against them can tell it - # has gone stale -- and chunk offsets from a replaced frame are garbage. - # fsspec's own token, rather than a tuple of the metadata fields we guess - # a backend exposes: memory:// has no mtime, which left it size-only. - self.stamp = fs.ukey(path) - # Exact ranges, not fs.open(): a buffered handle reads a whole block per + # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(_RangeReader(fs, path)) + raw, header, self._offsets = _read_frame_index(_RangeReader(self.read_range)) self._chunksize = header[8] self._extents = _chunk_extents(self._offsets, header) try: @@ -1264,12 +1256,19 @@ def get_chunk(self, nchunk: int) -> bytes: offset = int(self._offsets[nchunk]) if offset < 0: return self._special_chunk(offset) - data = self._fs.cat_file(self._path, start=offset, end=offset + int(self._extents[nchunk])) + data = self.read_range(offset, int(self._extents[nchunk])) return data[: struct.unpack(" bytes: - """The bytes at [*offset*, *offset* + *size*) of the frame.""" - return self._fs.cat_file(self._path, start=offset, end=offset + size) + """The bytes at [*offset*, *offset* + *size*) of the frame. + + The whole of the transport: everything else here is the frame format. + Fewer bytes may come back only at the end of the frame; anything else is + an error, since the caller has no way to ask for the rest. Must be safe + to call from several threads at once, which is what lets :ref:`Proxy` + overlap the fetches of one slice. + """ def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. @@ -1398,6 +1397,46 @@ def _special_chunk(self, offset: int) -> bytes: ) +class FsspecNDSource(ByteRangeNDSource): + """A :ref:`ByteRangeNDSource` reading its frame through fsspec. + + This is what ``blosc2.open(url, lazy=True)`` builds; wrap it in a + :ref:`Proxy` by hand when the cache belongs at a path of your choosing + rather than inside ``cache_storage``:: + + src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") + a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") + + Parameters + ---------- + urlpath: str + The fsspec URL of the frame. + max_concurrency: int, optional + As in :ref:`ByteRangeNDSource`. + """ + + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + from blosc2.core import _import_fsspec + + fsspec = _import_fsspec(urlpath) + fs, path = fsspec.url_to_fs(urlpath) + if fs.isdir(path): + raise NotImplementedError( + f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " + "chunk by chunk; open it with cache_storage= instead" + ) + self._fs, self._path = fs, path + # Identifies the remote bytes, so a cache built against them can tell it + # has gone stale -- and chunk offsets from a replaced frame are garbage. + # fsspec's own token, rather than a tuple of the metadata fields we guess + # a backend exposes: memory:// has no mtime, which left it size-only. + self.stamp = fs.ukey(path) + super().__init__(urlpath, max_concurrency) + + def read_range(self, offset: int, size: int) -> bytes: + return self._fs.cat_file(self._path, start=offset, end=offset + size) + + class ProxyNDField(blosc2.Operand): def __init__(self, proxy: Proxy, field: str): self.proxy = proxy diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index c7103f97c..b988f0dcb 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -733,7 +733,13 @@ def any_chunk_wants_blocks(monkeypatch): def _traffic(monkeypatch): - """Record every range read and whole-chunk read a source makes.""" + """Record every range read and whole-chunk read a source makes. + + Everything the source reads goes through `read_range`, opening the frame and + fetching a whole chunk included, so `reads` counts requests and a + whole-chunk fetch appears in both lists. Install it after the open where + the frame index would otherwise be counted in. + """ reads, chunks = [], [] for name, log in (("read_range", reads), ("get_chunk", chunks)): orig = getattr(blosc2.FsspecNDSource, name) @@ -758,9 +764,9 @@ def test_lazy_fetches_only_touched_blocks(monkeypatch): data, a = _incompressible((600, 600), (300, 600), (30, 600)) cbytes = a.schunk.cbytes // a.schunk.nchunks assert cbytes > blosc2.proxy.BLOCK_MIN_CBYTES - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("blocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) # after the open, which reads the frame index assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) # One read for the block offsets, one for the single block the slice lands in assert len(reads) == 2 @@ -772,23 +778,23 @@ def test_lazy_small_chunks_are_fetched_whole(monkeypatch): # Below the threshold a chunk is one cheap request, so blocks would only add # a round trip; nothing must go looking for block offsets a = blosc2.arange(0, 1000, dtype="i4", chunks=(100,)) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("smallblocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[150:250], a[150:250]) - assert not reads assert len(chunks) == 2 + assert len(reads) == len(chunks) # one request each, none of them for block offsets def test_lazy_whole_array_skips_the_block_path(monkeypatch, any_chunk_wants_blocks): # Wanting every block of a chunk is what fetching the chunk already does data, a = _incompressible((200, 200), (100, 200), (10, 200)) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("wholeblocks.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[:], data) - assert not reads assert len(chunks) == 2 + assert len(reads) == len(chunks) # one request each, none of them for block offsets @pytest.mark.parametrize( @@ -835,16 +841,16 @@ def test_lazy_block_cache_survives_reopen(tmp_path, monkeypatch, any_chunk_wants p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - fetched = len(reads) del p # A partly filled chunk survives, so the blocks in it do not travel again p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + reopened = len(reads) # every open reads the frame index afresh assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - assert len(reads) == fetched + assert len(reads) == reopened # ... and the ones missing from it still do assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) - assert len(reads) > fetched + assert len(reads) > reopened assert np.array_equal(p[...], data) @@ -863,12 +869,13 @@ def test_lazy_blocks_fall_back_for_memcpyed_chunks(monkeypatch, any_chunk_wants_ # A memcpyed chunk stores its blocks raw and has no offsets section to read data = np.random.default_rng(0).integers(0, 256, (300, 300), dtype="u1") a = blosc2.asarray(data, chunks=(150, 300), blocks=(15, 300), cparams={"clevel": 0}) - reads, chunks = _traffic(monkeypatch) p = blosc2.open(_put("memcpyed.b2nd", a), lazy=True) + reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - assert len(reads) == 1 # the offsets are read, and say there is nothing to skip assert len(chunks) == 1 + # The offsets are read, say there is nothing to skip, and the chunk follows + assert len(reads) == 2 def test_lazy_blocks_with_run_length_chunks(monkeypatch, any_chunk_wants_blocks): From d541eb3a170c617d8287cfeff78165d6cfa97c43 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:34:11 +0200 Subject: [PATCH 04/32] Read a C2Array's blocks over HTTP ranges Caterva2 serves a stored dataset with a FileResponse, which implements RFC 7233 on its own: a ranged request is answered 206 with the bytes asked for, seeked to in the file, and the auth cookie composes with it. So no new endpoint is needed for a Proxy over a C2Array to fetch the blocks a slice touches rather than the chunks they live in. On cat2.cloud's kevlar-tomo.b2nd a corner slice costs 0.031 MB instead of 2.723 MB -- 88x fewer bytes, and the same wall time on a fast link, where a round trip and a megabyte cost about the same. C2NDSource is ByteRangeNDSource plus that transport; C2Array grows the five members Proxy looks for and delegates them to it, so every existing Proxy(C2Array(...)) gets this without asking. What it costs when it does not apply is the point of the two gates before it: api/info rules out a dataset the subscriber computes (it reports an expression where a stored one reports a geometry) and one whose chunks are too small to be worth taking apart, both without a request, so those datasets behave exactly as they did. Everything else is decided by the status code of the first range read, once: a streamed dataset answers 200 with the whole body, which read_range refuses without reading it off the socket, and the array keeps to api/chunk for good. Co-Authored-By: Claude Opus 5 --- doc/reference/c2array.rst | 18 ++ src/blosc2/__init__.py | 3 +- src/blosc2/c2array.py | 126 +++++++++++ tests/ndarray/test_c2array_blocks.py | 321 +++++++++++++++++++++++++++ 4 files changed, 467 insertions(+), 1 deletion(-) create mode 100644 tests/ndarray/test_c2array_blocks.py diff --git a/doc/reference/c2array.rst b/doc/reference/c2array.rst index cfa6e7cee..da165991a 100644 --- a/doc/reference/c2array.rst +++ b/doc/reference/c2array.rst @@ -5,6 +5,15 @@ C2Array This is a class for remote arrays. This kind of array can also work as operand on a LazyExpr, LazyUDF or reduction. +Wrapped in a :ref:`Proxy`, a stored remote array is read at block granularity: +the proxy asks for the blocks a slice touches rather than the chunks they live +in, which for a multi-megabyte chunk is a small fraction of the bytes. That +rests on the subscriber serving the dataset from a file, ``Range`` header and +auth cookie both honoured; a dataset it computes instead (a lazy expression, an +HDF5 leaf) is fetched a whole chunk at a time, as everything was before. Which +one this is takes at most one request to find out, and is decided once -- +:meth:`C2Array.block_source` is what answers it. + .. currentmodule:: blosc2 @@ -28,6 +37,15 @@ This is a class for remote arrays. This kind of array can also work as operand o .. automethod:: __getitem__ +.. _C2NDSource: + +C2NDSource class +---------------- +.. autoclass:: C2NDSource + :members: + :member-order: groupwise + + .. _URLPath: URLPath class diff --git a/src/blosc2/__init__.py b/src/blosc2/__init__.py index 835b1ac0d..d6e965463 100644 --- a/src/blosc2/__init__.py +++ b/src/blosc2/__init__.py @@ -572,7 +572,7 @@ def _raise(exc): from .ref import Ref from .b2objects import open_b2object -from .c2array import c2context, C2Array, URLPath +from .c2array import c2context, C2Array, C2NDSource, URLPath from .dsl_kernel import DSLSyntaxError, DSLKernel, dsl_kernel, validate_dsl, validate_dsl_jit from .lazyexpr import ( @@ -853,6 +853,7 @@ def _raise(exc): "group_reduce", # Classes "C2Array", + "C2NDSource", "Column", "CParams", "CTable", diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 8d9874b86..fd7bdbf3c 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -8,6 +8,7 @@ from __future__ import annotations import atexit +import math import os import threading from contextlib import contextmanager @@ -22,6 +23,10 @@ from blosc2.b2objects import encode_b2object_payload, make_b2object_carrier, write_b2object_payload from blosc2.info import InfoReporter, format_nbytes_info +# blosc2/__init__ imports this module before blosc2.proxy, so this pulls proxy in +# early; it is safe because proxy only reaches into the package at call time +from blosc2.proxy import REMOTE_MAX_CONCURRENCY, ByteRangeNDSource + _subscriber_data = { "urlbase": os.environ.get("BLOSC_C2URLBASE"), "auth_token": "", @@ -255,6 +260,49 @@ def slice_to_string(slice_): return ", ".join(slice_parts) +_UNTRIED = object() +"""A block source that has not been asked for yet, as against one that failed.""" + + +class _NotRanged(Exception): + """The subscriber answered a range request with something other than a 206.""" + + +class C2NDSource(ByteRangeNDSource): + """The frame behind a :ref:`C2Array`, read over HTTP byte ranges. + + Caterva2 serves a *stored* dataset with a Starlette ``FileResponse``, which + implements RFC 7233 by itself: a ranged request comes back 206 with only the + bytes asked for, seeked to in the file rather than materialized, and the auth + cookie composes with it. That is everything :ref:`ByteRangeNDSource` needs, + so a slice costs the blocks it touches instead of the chunks they live in. + + A dataset the subscriber *builds* -- a lazy expression, an HDF5 leaf, a + ``.b2z`` member -- is streamed instead, and a streamed response ignores the + ``Range`` header and answers with the whole body. :meth:`read_range` refuses + such an answer without reading it off the socket, and :ref:`C2Array` then + keeps to whole chunks for good. Which is why this is built through + :meth:`C2Array.block_source` rather than directly: the fallback belongs with + the array, whose ``api/chunk`` path works for every dataset there is. + """ + + def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") + self._auth_token = array.auth_token + super().__init__(self._url, max_concurrency) + + def read_range(self, offset: int, size: int) -> bytes: + headers = _auth_headers(self._auth_token, {"Range": f"bytes={offset}-{offset + size - 1}"}) + with _sync_client().stream("GET", self._url, headers=headers) as response: + if response.status_code != 206: + # Whatever this is, it is not the bytes that were asked for: a 200 + # carries the whole dataset, which is the download this exists to + # avoid, so leave the body unread on the socket + raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") + response.read() + return response.content + + class C2Array(blosc2.Operand): """Remote compressed NDArray accessed from a Caterva2 server.""" @@ -305,6 +353,10 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N self.auth_token = auth_token self._aclient = None # lazy async client, shared across aget_chunk calls + # The block-reading source, built on first use: _UNTRIED, None (this + # dataset cannot be read in ranges) or a C2NDSource + self._block_source = _UNTRIED + self._block_lock = threading.Lock() # Try to 'open' the remote path try: @@ -504,6 +556,80 @@ async def aclose(self) -> None: await self._aclient.aclose() self._aclient = None + # -- Block-granular reads. A :ref:`Proxy` uses these to fetch the blocks a + # slice touches instead of whole chunks, wherever that is the cheaper way + # round; every one of them falls back to `get_chunk` when it is not. + + @property + def blocks_per_chunk(self) -> int: + """How many blocks a chunk of the remote array holds. + + Geometry, and `api/info` already carries it, so this costs no request: + chunks are padded to whole blocks, so every chunk holds the same number + of them, edge chunks included. + """ + blocks = self.blocks + if not all(blocks): # an empty array partitions into nothing + return 1 + return math.prod(math.ceil(c / b) for c, b in zip(self.chunks, blocks, strict=True)) + + def block_source(self) -> C2NDSource | None: + """The frame reader behind the block methods, or None if there is none. + + Built on the first request for it and never rebuilt. The fallback has to + be permanent: a subscriber that streams this dataset answers a range + request with the whole body, so retrying would pay a full download to + rediscover the same answer. + """ + if self._block_source is _UNTRIED: + with self._block_lock: + if self._block_source is _UNTRIED: + self._block_source = self._open_block_source() + return self._block_source + + def _open_block_source(self) -> C2NDSource | None: + """Decide, at whatever cost it takes, whether this dataset serves ranges.""" + # `api/info` rules out a dataset the subscriber computes for nothing: a + # stored one reports its geometry where a lazy expression reports + # `expression` and `operands` + if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): + return None + # Nor is a frame of small chunks worth an index read: blosc2 declines to + # take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would ever + # use a block, and the dataset keeps exactly the behaviour it had before + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) + if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: + return None + # Whether a dataset that reports a geometry is *served* from a file is + # something only the answer to a range request can say: an HDF5 leaf or a + # `.b2z` member reports one and is streamed all the same + try: + return C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): + # Not ranged, not a contiguous frame, not an NDArray, or not + # reachable: whole chunks work for all of those + return None + + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: + """Whether fetching *nwanted* blocks of a chunk beats fetching all of it.""" + source = self.block_source() + return source is not None and source.wants_blocks(nchunk, nwanted) + + def chunk_layout(self, nchunk: int): + """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" + return self.block_source().chunk_layout(nchunk) + + def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: + """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" + return self.block_source().block_plan(nchunk, nblocks) + + def read_range(self, offset: int, size: int) -> bytes: + """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" + source = self.block_source() + if source is None: + raise ValueError(f"{self.path} is not served in byte ranges by {self.urlbase}") + return source.read_range(offset, size) + @property def shape(self) -> tuple[int]: """The shape of the remote array""" diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py new file mode 100644 index 000000000..378a049c2 --- /dev/null +++ b/tests/ndarray/test_c2array_blocks.py @@ -0,0 +1,321 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Block-granular reads of a C2Array, against a stand-in for a subscriber. + +The server here answers the two endpoints the block path uses -- `api/info` for +the geometry and `api/fetch` for the bytes -- the way Caterva2 does: a stored +dataset comes back through a file response that honours `Range`, and one the +subscriber would compute comes back as a stream that ignores it. Which is the +distinction the whole arrangement rests on, and the one thing a test against a +live subscriber could not switch off at will. +""" + +import contextlib +import json +import math +import pathlib +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import numpy as np +import pytest + +import blosc2 + + +class _Subscriber: + """A Caterva2-shaped server over one .b2nd file.""" + + def __init__(self, path, ranges=True, cookie=None): + self.path = str(path) + self.frame = pathlib.Path(self.path).read_bytes() + self.array = blosc2.open(self.path) + self.ranges = ranges # False: stream the body and ignore Range, as a + self.cookie = cookie # computed dataset does + self.log = [] # (endpoint, status, bytes served) + + @property + def meta(self): + schunk = self.array.schunk + return { + "shape": list(self.array.shape), + "chunks": list(self.array.chunks), + "blocks": list(self.array.blocks), + "dtype": str(self.array.dtype), + "mtime": None, + "schunk": { + "cparams": {"typesize": self.array.dtype.itemsize}, + "nbytes": schunk.nbytes, + "cbytes": schunk.cbytes, + "cratio": schunk.cratio, + "blocksize": schunk.blocksize, + "vlmeta": {}, + }, + } + + +class _Handler(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *args): + pass # no stderr noise per request + + def handle(self): + # A client that hangs up on a body it refused, which is the point of the + # probe, leaves the write half of this raising + with contextlib.suppress(ConnectionResetError, BrokenPipeError): + super().handle() + + def _send(self, status, body, headers=(), endpoint=""): + self.server.subscriber.log.append((endpoint, status, len(body))) + self.send_response(status) + for name, value in headers: + self.send_header(name, value) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): + sub = self.server.subscriber + if sub.cookie and self.headers.get("Cookie") != sub.cookie: + self._send(401, b"unauthorized", endpoint="auth") + return + endpoint = self.path.split("/")[2] + if endpoint == "info": + self._send(200, json.dumps(sub.meta).encode(), endpoint="info") + elif endpoint == "chunk": + nchunk = int(self.path.split("nchunk=")[1]) + self._send(200, sub.array.schunk.get_chunk(nchunk), endpoint="chunk") + elif endpoint == "fetch": + self._fetch(sub) + else: + self._send(404, b"", endpoint=endpoint) + + def _fetch(self, sub): + wanted = self.headers.get("Range") + if not wanted or not sub.ranges: + # What a StreamingResponse does with a Range header: nothing at all + self._send(200, sub.frame, endpoint="fetch") + return + start, end = (int(n) for n in wanted.removeprefix("bytes=").split("-")) + end = min(end, len(sub.frame) - 1) + self._send( + 206, + sub.frame[start : end + 1], + [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + endpoint="fetch", + ) + + +def _serve(tmp_path, data, chunks, blocks, ranges=True, cookie=None, name="ds.b2nd"): + """A C2Array over *data*, served by a subscriber stand-in on localhost.""" + urlpath = str(tmp_path / name) + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + server.subscriber = _Subscriber(urlpath, ranges=ranges, cookie=cookie) + threading.Thread(target=server.serve_forever, daemon=True).start() + urlbase = f"http://127.0.0.1:{server.server_address[1]}/" + array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=cookie) + return array, server.subscriber, server + + +@pytest.fixture +def subscriber(tmp_path): + """Serve one array; the test parametrizes with `_serve`'s arguments.""" + servers = [] + + def build(*args, **kwargs): + array, sub, server = _serve(tmp_path, *args, **kwargs) + servers.append(server) + return array, sub + + yield build + for server in servers: + server.shutdown() + server.server_close() + + +@pytest.fixture +def any_chunk_wants_blocks(monkeypatch): + """Take the size threshold out of the way, so small test arrays use blocks.""" + monkeypatch.setattr(blosc2.proxy, "BLOCK_MIN_CBYTES", 0) + + +def _incompressible(shape, seed=0): + return np.random.default_rng(seed).random(shape) + + +def _bytes(sub, endpoint): + return sum(n for kind, _, n in sub.log if kind == endpoint) + + +def test_blocks_are_read_over_ranges(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + # The frame index, then one read for the block offsets and one for the block + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 6 + assert {status for _, status, _ in sub.log} == {206} + assert not _bytes(sub, "chunk") + # A block of a chunk, not the chunk: an eighth of it here, and never the frame + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 8 + # ... and the rest of the array still arrives correctly afterwards + assert np.array_equal(p[...], data) + + +@pytest.mark.parametrize( + ("shape", "chunks", "blocks", "item"), + [ + ((1000,), (500,), (50,), slice(120, 140)), + ((200, 200), (100, 200), (10, 20), (slice(5, 7), slice(30, 90))), + ((20, 60, 60), (10, 30, 60), (5, 10, 20), (3, slice(10, 20), slice(10, 20))), + ((200, 200), (100, 200), (10, 20), (5, 5)), + ], + ids=["1d", "2d", "3d", "point"], +) +def test_block_reads_are_correct(subscriber, any_chunk_wants_blocks, shape, chunks, blocks, item): + data = _incompressible(shape) + array, _ = subscriber(data, chunks=chunks, blocks=blocks) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[item], data[item]) + assert np.array_equal(p[...], data) + + +def test_blocks_carry_the_auth_cookie(subscriber, any_chunk_wants_blocks): + # fsspec's HTTP filesystem cannot carry this, which is why the block reads of + # a C2Array are its own rather than an fsspec URL pointed at the subscriber + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), cookie="token=sikrit") + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is not None + assert not any(status == 401 for _, status, _ in sub.log) + + +def test_a_streamed_dataset_falls_back_to_chunks(subscriber, any_chunk_wants_blocks): + # A lazy expression, an HDF5 leaf or a .b2z member is built rather than + # stored, and the response that carries it ignores Range + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), ranges=False) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is None + assert [kind for kind, _, _ in sub.log] == ["fetch", "chunk"] + # The probe must not read the body it refused: the whole dataset is what it + # would have downloaded to find out that ranges are not served + assert _bytes(sub, "fetch") == len(sub.frame) # served, but never read + + # And it is never probed again, whatever else is asked for + assert np.array_equal(p[...], data) + assert sum(1 for kind, _, _ in sub.log if kind == "fetch") == 1 + + +def test_a_computed_dataset_is_ruled_out_without_a_request(subscriber, any_chunk_wants_blocks): + # api/info tells a stored dataset from one the subscriber computes: the + # latter reports `expression` and `operands` where this reports a geometry + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + del array.meta["chunks"] + sub.log.clear() + + assert array.block_source() is None + assert not sub.log + + +def test_small_chunks_are_fetched_whole(subscriber): + # Below the threshold a chunk is one cheap request, so blocks would only add + # a round trip: nothing goes looking for the frame index, let alone a block + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is None + assert [kind for kind, _, _ in sub.log] == ["chunk"] + + +def test_blocks_accumulate_in_a_chunk(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + served = len(sub.log) + # A different block of the same chunk: what is already cached stays cached + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) > served + served = len(sub.log) + # Both are now in the same cached chunk, and both are still right + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) == served + + +def test_blocks_survive_a_reopened_cache(tmp_path, subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "c2-cache.b2nd") + + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + # A partly filled chunk survives, so the blocks in it do not travel again + p = blosc2.Proxy(array, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served + # ... and the ones missing from it still do + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert len(sub.log) > served + assert np.array_equal(p[...], data) + + +def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): + # A cache left by a run that fetched whole chunks (which is every run before + # this existed) holds complete chunks, so nothing in it is fetched again + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "chunkwise.b2nd") + + array._block_source = None # as if the subscriber served no ranges + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p, array + + array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served + assert np.array_equal(p[...], data) + + +def test_blocks_per_chunk_costs_no_request(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + sub.log.clear() + + assert array.blocks_per_chunk == math.prod((100 // 10, 200 // 20)) + assert not sub.log + + +def test_read_range_says_so_when_there_are_no_ranges(subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20), ranges=False) + + with pytest.raises(ValueError, match="not served in byte ranges"): + array.read_range(0, 32) From b021104b870e94888d4fcbc02f2324e7c14b35a2 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:53:36 +0200 Subject: [PATCH 05/32] Ask a Caterva2 subscriber for a whole wave of ranges at once RFC 7233 lets a Range header name several spans, and the answer is a multipart/byteranges body carrying each with its own Content-Range. Starlette builds that, so the blocks a slice wants -- across chunks, since they are all the same file -- can travel in one request instead of one each. No object store offers this; it is what makes a subscriber the better backend for block reads rather than merely an equal one. Measured against cat2.cloud on kevlar-tomo.b2nd, a slice touching ten chunks: 20 requests in 0.334 s becomes 2 in 0.204 s, and 1.008 s becomes 0.141 s against a subscriber reached one request at a time. Both waves collapse: the chunk headers that say where the blocks are, then the blocks. The seam is one method. ByteRangeNDSource grows read_ranges(spans), which by default is one read_range each, and max_ranges to say how many a transport will take; C2NDSource overrides the first and raises the second. A source with neither -- every fsspec backend -- is batched one range at a time, which is the reads it always made. The answer is taken apart by what each part says it holds rather than by trusting the order: Starlette sorts the spans and merges the ones that touch, and answers a plain 206 when they all merge into one. A subscriber that answers with less than was asked for is noticed once and never batched again. Also give C2Array a max_concurrency: Proxy.afetch already used 8 for one, and fetch was serial only for want of somewhere to read the figure from. That alone is what takes the unbatched block path from 1.008 s to 0.334 s. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 118 +++++++++++++++++++++++++- src/blosc2/proxy.py | 120 ++++++++++++++++++++------- tests/ndarray/test_c2array_blocks.py | 103 +++++++++++++++++++++-- 3 files changed, 299 insertions(+), 42 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index fd7bdbf3c..d631e24ed 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -263,11 +263,63 @@ def slice_to_string(slice_): _UNTRIED = object() """A block source that has not been asked for yet, as against one that failed.""" +MAX_RANGES_PER_REQUEST = 64 +"""How many byte ranges one request to a subscriber may ask for. + +There is no limit in the protocol, and the saving grows with the count -- but a +`Range` header is a header, which servers and proxies cap the length of (8 KB is +the usual figure, and 64 spans of a large frame are about a kilobyte), and a +failed request costs a round trip and every span in it. +""" + class _NotRanged(Exception): """The subscriber answered a range request with something other than a 206.""" +class _PartsMissing(Exception): + """A multi-range answer did not carry all the bytes that were asked for.""" + + +def _content_range(value: str) -> int: + """Where a `Content-Range: bytes start-end/total` header says its part starts.""" + return int(value.split()[1].split("-")[0]) + + +def _byteranges(response) -> list[tuple[int, bytes]]: + """The parts of a 206, as (where each starts in the frame, its bytes). + + One part for an ordinary 206, several for a `multipart/byteranges` body: a + boundary line, the part's own headers, a blank line and its bytes, over and + over, ending in the boundary followed by two dashes. `email` would parse it, + at the price of decoding a megabyte of compressed data as text. + """ + content_type = response.headers.get("content-type", "") + if "multipart/byteranges" not in content_type: + return [(_content_range(response.headers["content-range"]), response.content)] + boundary = content_type.split("boundary=")[1].strip().strip('"').encode() + parts = [] + for chunk in response.content.split(b"--" + boundary): + head, sep, body = chunk.partition(b"\r\n\r\n") + if not sep: # the preamble before the first boundary, and the closing -- + continue + for line in head.split(b"\r\n"): + name, _, value = line.partition(b":") + if name.strip().lower() == b"content-range": + # The body ends with the CRLF that belongs to the next boundary + parts.append((_content_range(value.decode()), body[: body.rfind(b"\r\n")])) + break + return parts + + +def _span_of(parts: list[tuple[int, bytes]], offset: int, size: int, url: str) -> bytes: + """The bytes of one requested span, out of whichever part covers it.""" + for start, data in parts: + if start <= offset < start + len(data): + return data[offset - start : offset - start + size] + raise _PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") + + class C2NDSource(ByteRangeNDSource): """The frame behind a :ref:`C2Array`, read over HTTP byte ranges. @@ -291,8 +343,40 @@ def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) + max_ranges = MAX_RANGES_PER_REQUEST + def read_range(self, offset: int, size: int) -> bytes: - headers = _auth_headers(self._auth_token, {"Range": f"bytes={offset}-{offset + size - 1}"}) + return self._get([(offset, size)])[0] + + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """Every span in one request, which HTTP has a shape for and S3 has not. + + RFC 7233 lets a `Range` header name several spans, and the answer is a + `multipart/byteranges` body carrying each with its own `Content-Range`. + Starlette builds that, so a whole wave of block reads -- across chunks, + since they are all the same file -- costs one round trip instead of one + each. Measured against cat2.cloud: 32 spans in 0.136 s against 0.208 s + for 32 requests eight at a time, and 1.530 s for them one at a time. + + The server may serve fewer parts than were asked for: Starlette sorts the + spans and merges the ones that touch, and answers a single 206 when they + all merge into one. So the answer is taken apart by what each part says + it holds, and each span read out of the part that covers it, rather than + by trusting the order. A server that answers a multi-range request with + bytes that do not cover the whole of it is not one to ask again: this + keeps to a range per request from there on. + """ + spans = list(spans) + if len(spans) > 1 and self.max_ranges > 1: + try: + return self._get(spans) + except _PartsMissing: + self.max_ranges = 1 + return [self.read_range(*span) for span in spans] + + def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: + wanted = ", ".join(f"{offset}-{offset + size - 1}" for offset, size in spans) + headers = _auth_headers(self._auth_token, {"Range": f"bytes={wanted}"}) with _sync_client().stream("GET", self._url, headers=headers) as response: if response.status_code != 206: # Whatever this is, it is not the bytes that were asked for: a 200 @@ -300,12 +384,23 @@ def read_range(self, offset: int, size: int) -> bytes: # avoid, so leave the body unread on the socket raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") response.read() - return response.content + parts = _byteranges(response) + return [_span_of(parts, offset, size, self._url) for offset, size in spans] class C2Array(blosc2.Operand): """Remote compressed NDArray accessed from a Caterva2 server.""" + max_concurrency = REMOTE_MAX_CONCURRENCY + """How many fetches a :ref:`Proxy` over this array may run at once. + + Every chunk or block is a request whose cost is mostly the round trip, so + overlapping them is what a remote source has to gain; :meth:`Proxy.afetch` + already used this figure for a `C2Array`, and `fetch` was serial only for + want of somewhere to read it from. `get_chunk` and the range reads are + thread-safe: they share one pooled HTTP client and hold no state of their own. + """ + def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | None = None): """Create an instance of a remote NDArray. @@ -615,20 +710,37 @@ def wants_blocks(self, nchunk: int, nwanted: int) -> bool: source = self.block_source() return source is not None and source.wants_blocks(nchunk, nwanted) + @property + def max_ranges(self) -> int: + """How many ranges one request to this subscriber may carry.""" + source = self.block_source() + return 1 if source is None else source.max_ranges + def chunk_layout(self, nchunk: int): """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" return self.block_source().chunk_layout(nchunk) + def chunk_layouts(self, nchunks: Sequence[int]) -> list: + """The same for several chunks; see :meth:`ByteRangeNDSource.chunk_layouts`.""" + return self.block_source().chunk_layouts(nchunks) + def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" return self.block_source().block_plan(nchunk, nblocks) def read_range(self, offset: int, size: int) -> bytes: """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" + return self._ranged().read_range(offset, size) + + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """The bytes of every span, in one request where the subscriber allows it.""" + return self._ranged().read_ranges(spans) + + def _ranged(self) -> C2NDSource: source = self.block_source() if source is None: raise ValueError(f"{self.path} is not served in byte ranges by {self.urlbase}") - return source.read_range(offset, size) + return source @property def shape(self) -> tuple[int]: diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 3448fcb3e..4a2027138 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -83,6 +83,12 @@ class ProxyNDSource(ABC): ``chunk_layout(nchunk)``, ``block_plan(nchunk, nblocks)`` and ``read_range(offset, size)``; :ref:`FsspecNDSource` implements them over byte ranges and is the worked example. + + A source whose transport can ask for several ranges at once says so with + ``max_ranges`` and serves ``read_ranges(spans)`` and + ``chunk_layouts(nchunks)`` as well; :ref:`Proxy` then sends a whole wave of + reads as one request. Both are optional, and a source without them is asked + one range at a time exactly as before. """ @property @@ -679,6 +685,10 @@ def _fetch_by_block(self, item, max_concurrency: int | None): whatever chunks are cheaper to take whole. Which is which is decided before any of it, so the chunks that do not want blocks cost exactly one request each, as they did before this existed. + + A transport that carries several ranges per request (`max_ranges`) + collapses each wave further, into as few requests as its limit allows; + one that does not sees exactly the reads it always saw. """ missing = self._missing_blocks(item) if not missing: @@ -686,39 +696,56 @@ def _fetch_by_block(self, item, max_concurrency: int | None): wanted = {n: bs for n, bs in missing.items() if self.src.wants_blocks(n, len(bs))} whole = [n for n in missing if n not in wanted] - layouts = dict( - zip(wanted, self._run(self.src.chunk_layout, list(wanted), max_concurrency), strict=True) - ) + layouts = dict(zip(wanted, self._chunk_layouts(list(wanted), max_concurrency), strict=True)) # A chunk with nothing to take apart (memcpyed, or a single block) says so # only once its header is read whole += [n for n, layout in layouts.items() if layout is None] wanted = {n: bs for n, bs in wanted.items() if layouts[n] is not None} - tasks = [(n, None) for n in whole] - tasks += [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] + # Each task is what one request will carry: a whole chunk on its own, or + # a batch of range reads (of one, for a transport that takes one) + runs = [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] + batch = max(getattr(self.src, "max_ranges", 1), 1) + tasks = [((n, None),) for n in whole] + list(itertools.batched(runs, batch)) - def fetch_one(task): - nchunk, run = task - return self.src.get_chunk(nchunk) if run is None else self.src.read_range(run[0], run[1]) + # `read_ranges` is the optional half of the protocol: a source that only + # has `read_range` is asked one range at a time, as `batch` is 1 for it + read_ranges = getattr(self.src, "read_ranges", None) + + def fetch(task): + if task[0][1] is None: + return [self.src.get_chunk(task[0][0])] + spans = [(run[0], run[1]) for _, run in task] + if read_ranges is None: + return [self.src.read_range(*span) for span in spans] + return read_ranges(spans) pending = {} try: - for (nchunk, run), data in zip(tasks, self._run(fetch_one, tasks, max_concurrency), strict=True): - if run is None: - self._store_chunk(nchunk, data) - continue - payloads = pending.setdefault(nchunk, {}) - for nblock, offset, size in run[2]: - payloads[nblock] = data[offset : offset + size] - # Write the chunk once its last outstanding block has landed, so - # nothing is held longer than it takes to splice it in - if len(payloads) == len(wanted[nchunk]): - self._write_blocks(nchunk, pending.pop(nchunk), layouts[nchunk][0]) + for task, answers in zip(tasks, self._run(fetch, tasks, max_concurrency), strict=True): + for (nchunk, run), data in zip(task, answers, strict=True): + if run is None: + self._store_chunk(nchunk, data) + continue + payloads = pending.setdefault(nchunk, {}) + for nblock, offset, size in run[2]: + payloads[nblock] = data[offset : offset + size] + # Write the chunk once its last outstanding block has landed, so + # nothing is held longer than it takes to splice it in + if len(payloads) == len(wanted[nchunk]): + self._write_blocks(nchunk, pending.pop(nchunk), layouts[nchunk][0]) finally: self._save_fetched() return self._cache + def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None): + """Where the blocks of every one of *nchunks* are, in as few requests as fit.""" + if max(getattr(self.src, "max_ranges", 1), 1) > 1: + # The source batches the reads itself, so there is nothing to overlap + return self.src.chunk_layouts(nchunks) + return self._run(self.src.chunk_layout, nchunks, max_concurrency) + def _write_blocks(self, nchunk: int, payloads: dict[int, bytes], header: bytes) -> None: """Put the blocks just fetched into the cache, keeping those already there. @@ -1196,6 +1223,15 @@ class ByteRangeNDSource(ProxyNDSource): stamp = None + max_ranges = 1 + """How many ranges one request of this transport may carry. + + One means one request each, which is all any object store offers. A + subscriber answering ``multipart/byteranges`` takes more -- see + :meth:`read_ranges` -- and then a slice costs a couple of requests rather + than a couple per chunk it touches. + """ + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.max_concurrency = max_concurrency self.urlpath = urlpath @@ -1270,6 +1306,16 @@ def read_range(self, offset: int, size: int) -> bytes: overlap the fetches of one slice. """ + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """The bytes of every ``(offset, size)`` in *spans*, in that order. + + One request each, unless a transport that can carry several ranges in one + overrides this and raises :attr:`max_ranges` to say how many. Nothing + else has to change for it: this is the only method a batching transport + needs, and everything that reads bytes goes through it. + """ + return [self.read_range(offset, size) for offset, size in spans] + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. @@ -1307,13 +1353,28 @@ def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | No Each of those is then fetched whole, at the cost of the one header read that found out. """ - cached = self._layouts.get(nchunk) - if cached is not None: - return cached + return self.chunk_layouts((nchunk,))[0] + + def chunk_layouts(self, nchunks: Sequence[int]) -> list: + """:meth:`chunk_layout` for several chunks, in as few requests as they fit. + + One request each unless the transport takes several ranges at once, and + none at all for a chunk already read: a fetch asks for layouts only where + blocks are missing, but the same chunk comes up again as a slice fills it + in. + """ + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] + for batch in itertools.batched(todo, max(self.max_ranges, 1)): + spans = [(int(self._offsets[n]), section) for n in batch] + heads = self.read_ranges(spans) + for nchunk, head in zip(batch, heads, strict=True): + self._layouts[nchunk] = self._parse_layout(head, section) + return [self._layouts[n] for n in nchunks] + + def _parse_layout(self, head: bytes, section: int): + """The layout a chunk's header section says it has, or None for no layout.""" nblocks = self.blocks_per_chunk - offset = int(self._offsets[nchunk]) - section = _CHUNK_HEADER_LEN + 4 * nblocks - head = self.read_range(offset, section) cbytes = struct.unpack(" tuple[bytes, np.ndarray, np.ndarray] | No or cbytes < section or len(head) < section ): - layout = None - else: - bstarts = np.frombuffer(head[_CHUNK_HEADER_LEN:], dtype=" list[tuple[int, int, tuple]]: """The range reads that cover *nblocks*, near-adjacent ones merged. diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 378a049c2..46c2d384b 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -31,12 +31,14 @@ class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" - def __init__(self, path, ranges=True, cookie=None): + def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) self.frame = pathlib.Path(self.path).read_bytes() self.array = blosc2.open(self.path) self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does + self.multipart = multipart # False: answer only the first range asked for + self.merge_ranges = merge_ranges # as Starlette does with ranges that touch self.log = [] # (endpoint, status, bytes served) @property @@ -102,25 +104,57 @@ def _fetch(self, sub): # What a StreamingResponse does with a Range header: nothing at all self._send(200, sub.frame, endpoint="fetch") return - start, end = (int(n) for n in wanted.removeprefix("bytes=").split("-")) - end = min(end, len(sub.frame) - 1) + spans = [] + for span in wanted.removeprefix("bytes=").split(","): + start, end = (int(n) for n in span.split("-")) + spans.append((start, min(end, len(sub.frame) - 1))) + # Starlette sorts the spans and merges the ones that touch, so a client + # cannot count on getting a part per span, nor on the order it asked in + spans.sort() + merged = [spans[0]] + for start, end in spans[1:]: + if start <= merged[-1][1] + 1 and sub.merge_ranges: + merged[-1] = (merged[-1][0], max(merged[-1][1], end)) + else: + merged.append((start, end)) + if len(merged) == 1 or not sub.multipart: + start, end = merged[0] # ... and answers a plain 206 when one is left + self._send( + 206, + sub.frame[start : end + 1], + [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + endpoint="fetch", + ) + return + boundary = "c2boundary" + body = b"" + for start, end in merged: + body += ( + f"--{boundary}\r\nContent-Type: application/octet-stream\r\n" + f"Content-Range: bytes {start}-{end}/{len(sub.frame)}\r\n\r\n" + ).encode() + body += sub.frame[start : end + 1] + b"\r\n" + body += f"--{boundary}--\r\n".encode() self._send( 206, - sub.frame[start : end + 1], - [("Content-Range", f"bytes {start}-{end}/{len(sub.frame)}"), ("Accept-Ranges", "bytes")], + body, + [ + ("Content-Type", f"multipart/byteranges; boundary={boundary}"), + ("Accept-Ranges", "bytes"), + ], endpoint="fetch", ) -def _serve(tmp_path, data, chunks, blocks, ranges=True, cookie=None, name="ds.b2nd"): +def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", **kwargs): """A C2Array over *data*, served by a subscriber stand-in on localhost.""" urlpath = str(tmp_path / name) blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) - server.subscriber = _Subscriber(urlpath, ranges=ranges, cookie=cookie) + server.subscriber = _Subscriber(urlpath, **kwargs) threading.Thread(target=server.serve_forever, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" - array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=cookie) + array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=kwargs.get("cookie")) return array, server.subscriber, server @@ -319,3 +353,56 @@ def test_read_range_says_so_when_there_are_no_ranges(subscriber, any_chunk_wants with pytest.raises(ValueError, match="not served in byte ranges"): array.read_range(0, 32) + + +def test_a_whole_wave_travels_in_one_request(subscriber, any_chunk_wants_blocks): + # A column through every chunk: each one wants a handful of blocks that lie + # apart in the file, which without batching is a request each + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + assert array.block_source() is not None # the frame index, read once per array + sub.log.clear() + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + # One request for the layouts of all four chunks, one for all their blocks + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] + assert {status for _, status, _ in sub.log} == {206} + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 4 + + # Which is the whole of the difference: one request per range otherwise + other, sub2 = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="unbatched.b2nd") + other.block_source().max_ranges = 1 + q = blosc2.Proxy(other, mode="w") + sub2.log.clear() + assert np.array_equal(q[:, 0:10], data[:, 0:10]) + assert len(sub2.log) > 4 * len(sub.log) + + +def test_merged_and_reordered_parts_are_read_correctly(subscriber, any_chunk_wants_blocks): + # The server sorts the spans and merges the ones that touch, so the answer + # carries fewer parts than were asked for and in an order of its own + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + assert np.array_equal(p[...], data) + assert array.max_ranges > 1 # ... and it never had to stop batching + + +def test_a_server_that_answers_one_range_stops_being_batched(subscriber, any_chunk_wants_blocks): + # A subscriber that takes the first span of a multi-range request and ignores + # the rest: the answer does not cover what was asked for, which is noticed + # and never repeated + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), multipart=False) + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[:, 0:10], data[:, 0:10]) + assert array.max_ranges == 1 + served = len(sub.log) + assert np.array_equal(p[:, 100:110], data[:, 100:110]) + # One request per range from here on, and no second attempt at batching + assert len(sub.log) > served + 2 + assert np.array_equal(p[...], data) From 1b2dd44769947a84bcca15cc8a00bbba8d1d1ad8 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:54:36 +0200 Subject: [PATCH 06/32] Record what the block-granularity work landed as, and what it found The plan said an HDF5 leaf or a .b2z member might report a geometry while being streamed, and that a C2Array over a computed dataset would be caught by the api/info check. Both turned out slightly otherwise, and the plan now says so rather than reading as if it had been right. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 15 ++++++ plans/cat2-block-granularity.md | 92 +++++++++++++++++++++++++++++++-- 2 files changed, 104 insertions(+), 3 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 165923a66..eea232aec 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -42,6 +42,21 @@ XXX version-specific blurb XXX with the traffic, since a fetch in flight is now a block rather than a chunk. `bench/ndarray/fsspec-block-granularity.py` measures both on any array. +* A `Proxy` over a `C2Array` reads **blocks** too, straight out of the stored + frame over HTTP byte ranges. Caterva2 serves a stored dataset from a file, so + the `Range` header is honoured and composes with the auth cookie; no new + endpoint is involved. On cat2.cloud's `kevlar-tomo.b2nd` a corner slice costs + 0.031 MB instead of 2.723 MB, and a slice touching ten chunks takes 0.14 s + against 1.01 s. Three things add up to it: one pooled HTTP client instead of a + connection per request (0.162 s → 0.046 s each), fetches that overlap by + default as `afetch()` already did, and one request carrying the whole wave of + ranges (`multipart/byteranges`, which no object store offers). A dataset the + subscriber *computes* — a lazy expression, an HDF5 leaf, a `.b2z` member — is + fetched a whole chunk at a time as before; which it is costs at most one small + request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is + the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it + with a `read_range(offset, size)` to give any transport the same treatment. + * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 4e6addb51..e0d038fea 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -1,8 +1,12 @@ # Block-Granular Reads For Caterva2 And `C2Array` -Analysis and plan — nothing implemented. Written 2026-08-17, after -[plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs -(merged as PR #701). +Written 2026-08-17, after [plans/fsspec-blocks.md](fsspec-blocks.md) landed block +fetching for fsspec URLs (merged as PR #701). + +**All five phases are implemented** (2026-08-17). Phases 1-4 are in blosc2 on +`cat2-block-granularity`; phase 5 is in Caterva2 on `range-honesty`. See +[what landed](#what-landed) at the end for the results and the two things the +work found out. ## The question @@ -312,3 +316,85 @@ print(r.status_code, r.headers.get("content-range"), len(r.content)) `bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any local array, which is what decides whether a given dataset would benefit at all. + +## What landed + +Every phase, in the order the plan gives them. Where the plan guessed and the +work found out otherwise, that is said below rather than quietly fixed. + +| phase | where | commit | +|---|---|---| +| 1. pooled client | blosc2 `c2array.py` | *Pool the HTTP client C2Array requests go through* | +| 2. capability check | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | +| 3b. `ByteRangeNDSource` | blosc2 `proxy.py` | *Lift the frame reading out of FsspecNDSource* | +| 3. `C2Array` blocks | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | +| 4. multipart | blosc2 both | *Ask a subscriber for a whole wave of ranges at once* | +| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges* | + +Option **(b)** was taken for phase 3, as recommended: `ByteRangeNDSource` holds +the frame format and one abstract `read_range`, `FsspecNDSource` is that plus +four lines of fsspec, and `C2NDSource` is that plus HTTP ranges with the auth +cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, +so every existing `Proxy(C2Array(...))` gets blocks without being asked. + +### The numbers, end to end + +Against `cat2.cloud/demo` on `kevlar-tomo.b2nd` (1.44 MB chunks, 47 blocks each): + +| | requests | bytes | time | +|---|---|---|---| +| chunks (before) | 2 | 2.723 MB | 0.28 s | +| blocks | 8 | 0.031 MB | 0.34 s | + +88x fewer bytes for a corner slice, and the same wall time on a link where a +round trip and a megabyte cost about the same. For a slice touching ten chunks, +where the request count is what decides: + +| | requests | time | +|---|---|---| +| blocks, one request at a time | 20 | 1.008 s | +| blocks, fetches overlapped | 20 | 0.334 s | +| blocks, multipart | 2 | 0.141 s | + +Phase 1 on its own: 0.162 s per request against 0.046 s pooled, on the existing +chunk path. + +### Two things the plan had wrong + +- **`C2Array` cannot be built over a computed dataset at all.** `api/info` for a + lazy expression carries no `schunk`, and `C2Array.__init__` reads + `meta["schunk"]["cparams"]`, so it raises long before any of this. The + info-based discriminator of phase 2 is still there and still right, but it + earns its place on the *other* case: +- **A `.b2z` member reports a full geometry and is streamed.** Confirmed against + a local server: `api/info` on `@public/tree-store.b2z/level1/leaf6` answers + with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the + status code of the first range read is the authority, exactly as phase 2 + argued — with phase 5 in place that costs 169 bytes and one round trip. + +Phase 4 was built because the measurement said to: 32 spans cost 0.136 s in one +multipart request against 0.208 s as 32 requests eight at a time, and 1.530 s +one at a time. Starlette *sorts and merges* the spans it is given and answers a +plain 206 when they all merge into one, so the client maps parts back by what +each says it holds rather than by order; a server that answers with less than +was asked for is noticed once and never batched again. + +The plan's guess that `_run` would already overlap a C2Array's fetches was wrong +in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, and +`C2Array` had none, so the sync path was serial. It has one now, the same 8 +`afetch` already used. + +### Left undone + +- **`api/chunk` does not serve container members.** A `Proxy` over a `.b2z` leaf + falls back to whole chunks correctly and then 404s, because `get_chunk` in the + server resolves the path without an inner key. It has never worked; nothing + here changed it, and nothing here depends on it. +- **The four requests to open a frame** (prefix, header, offsets header, offsets) + could be two: the header could be read optimistically with the prefix. It is in + `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would + make it one. +- **`C2Array` still has no `stamp`.** A cache is checked against the source's + geometry only, so a dataset replaced underneath while keeping its shape is not + noticed. `mtime` is in `api/info` and would do it, but adding one invalidates + every cache built before it, which wants its own decision. From b868c53f0e1685acb85f282cd03d61b3eb387fd6 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Mon, 17 Aug 2026 23:56:58 +0200 Subject: [PATCH 07/32] Keep C2NDSource's max_ranges with the rest of its class body --- src/blosc2/c2array.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index d631e24ed..d2c1649dc 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -338,13 +338,13 @@ class C2NDSource(ByteRangeNDSource): the array, whose ``api/chunk`` path works for every dataset there is. """ + max_ranges = MAX_RANGES_PER_REQUEST + def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) - max_ranges = MAX_RANGES_PER_REQUEST - def read_range(self, offset: int, size: int) -> bytes: return self._get([(offset, size)])[0] From e8d9ba1a4caab3647d6f47b174469c20fc6bc685 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:15:11 +0200 Subject: [PATCH 08/32] Measure the Caterva2 block path the way the fsspec one is measured bench/ndarray/fsspec-block-granularity.py answers "would blocks beat chunks for this array, and by how much" for an fsspec URL. This is the same question for a Caterva2 dataset, where it has a different shape: whether the dataset serves ranges at all is the first thing to know, and three separate changes -- a pooled client, overlapped fetches, one request carrying many ranges -- decide the rest. It prints, for a dataset of the caller's choosing: whether ranges are served and what finding that out cost, what the frame index costs, the request plan of each mode for six slice shapes, what a pooled connection is worth against a client per request, and the timed slices themselves. The plans come from Proxy._wanted_blocks and the source's own block_plan rather than from a reimplementation, and the chunk sizes are read out of the chunk headers rather than guessed from the gaps between them, so the counts in the plan table are the ones the timed run then produces. No service is needed: a stand-in subscriber serves api/info, api/fetch and api/chunk over loopback from any local .b2nd, with ranges, multipart, and the sort-and-merge Starlette does to the spans it is given. --streamed makes it answer the way a computed dataset is served, which is what the fallback costs. --latency-ms and --bandwidth-mbs put a network back in front of loopback. Against cat2.cloud/demo on kevlar-tomo.b2nd, a line through all ten chunks: chunk mode 10 req 14.43 MB 1.008 s blocks 20 req 0.20 MB 0.202 s multipart 2 req 0.20 MB 0.107 s 9.4x and the two slabs that want every block of their chunk come out identical in all three modes, which is the threshold declining to take a chunk apart. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 2 + bench/ndarray/cat2-block-granularity.py | 550 ++++++++++++++++++++++++ plans/cat2-block-granularity.md | 14 +- 3 files changed, 564 insertions(+), 2 deletions(-) create mode 100644 bench/ndarray/cat2-block-granularity.py diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index eea232aec..d153146c2 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -56,6 +56,8 @@ XXX version-specific blurb XXX request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. + `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, + against a real subscriber or a stand-in it starts itself. * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can diff --git a/bench/ndarray/cat2-block-granularity.py b/bench/ndarray/cat2-block-granularity.py new file mode 100644 index 000000000..e1c9e0f92 --- /dev/null +++ b/bench/ndarray/cat2-block-granularity.py @@ -0,0 +1,550 @@ +#!/usr/bin/env python + +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""Would block-granular reads beat whole-chunk ones for a given Caterva2 dataset? + +A ``Proxy`` over a ``C2Array`` used to fetch one whole compressed chunk per +request, through ``api/chunk``. A chunk is made of blocks, which blosc2 +compresses and decompresses independently, and Caterva2 serves a *stored* +dataset straight from its file, so ``api/fetch`` honours a ``Range`` header and +a slice can fetch only the blocks it touches. Whether that pays depends on +three things this script measures against a subscriber of your choosing: + +- whether the dataset **serves ranges at all**. One stored from a file does; + one the subscriber computes -- a lazy expression, an HDF5 leaf, a ``.b2z`` + member -- is streamed, cannot honour a range, and keeps to whole chunks; +- the **request plan**: how many requests each mode issues and how many bytes + they carry, read from the frame's own chunk headers for a few hundred bytes; +- the **wall time** of each pattern, which is the shipped code fetching real + slices from a real server. + +Usage +----- + # a local array, served by a stand-in subscriber over loopback + python cat2-block-granularity.py mydata.b2nd + + # ... with a network put back in front of every request + python cat2-block-granularity.py mydata.b2nd --latency-ms 45 --bandwidth-mbs 10 + + # ... served the way a computed dataset is, which is what the fallback costs + python cat2-block-granularity.py mydata.b2nd --streamed + + # against a real subscriber + python cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd \\ + --urlbase https://cat2.cloud/demo + + # ... an authenticated dataset + python cat2-block-granularity.py @personal/mine.b2nd --urlbase http://localhost:8000 \\ + --username me@example.com --password foobar11 + +Three modes are timed, each of them the shipped code with one thing changed: + +- ``chunks``: one ``api/chunk`` request per touched chunk. What a proxy over a + C2Array did before blocks existed, and what it still does for a dataset that + cannot serve ranges or whose chunks are too small to be worth taking apart. +- ``blocks``: one request for the block offsets of each chunk worth taking + apart, then one per coalesced run of the blocks wanted. Two dependent waves. +- ``multipart``: the same two waves, each collapsed into a single request -- + RFC 7233 lets one ``Range`` header name many spans, and Caterva2 answers + ``multipart/byteranges``. No object store offers this. + +The stand-in subscriber answers ``api/info``, ``api/fetch`` and ``api/chunk`` +the way Caterva2 does, ranges and multipart included (it sorts and merges the +spans it is given, as Starlette does, which is what the client has to survive). +Its request and byte counts are exact. Its *times* are not a subscriber's: +loopback answers in a fraction of a millisecond, where a subscriber over a WAN +takes tens of milliseconds, which is the regime the whole trade lives in. +``--latency-ms`` and ``--bandwidth-mbs`` put a stated network back in front of +each request; cat2.cloud from Europe measures about ``--latency-ms 45 +--bandwidth-mbs 10``. The simulated bandwidth is *per request*, so eight +parallel ones get eight times as much of it -- which is about right for an +object store and about wrong for one subscriber, and is why ``multipart`` can +come out behind ``blocks`` there while it wins against the real thing. + +Bytes counted are payload: the multipart envelope (about a hundred bytes per +part) and the HTTP headers of every request are not in them. +""" + +import argparse +import http.server +import json +import math +import pathlib +import statistics +import struct +import threading +import time + +import blosc2 +from blosc2 import c2array + +CHUNK_HEADER = blosc2.proxy._CHUNK_HEADER_LEN + + +# +# A stand-in subscriber, so this runs with no service to point at +# + + +class Subscriber: + """Caterva2's three read endpoints over one local .b2nd file.""" + + def __init__(self, urlpath, streamed=False): + self.path = pathlib.Path(urlpath) + self.size = self.path.stat().st_size + self.array = blosc2.open(str(self.path)) + # A dataset the subscriber would compute rather than store: served by a + # body builder, which has no way to honour a Range + self.streamed = streamed + + def meta(self): + schunk = self.array.schunk + return { + "shape": list(self.array.shape), + "chunks": list(self.array.chunks), + "blocks": list(self.array.blocks), + "dtype": str(self.array.dtype), + "mtime": None, + "schunk": { + "cparams": {"typesize": self.array.dtype.itemsize}, + "nbytes": schunk.nbytes, + "cbytes": schunk.cbytes, + "cratio": schunk.cratio, + "blocksize": schunk.blocksize, + "vlmeta": {}, + }, + } + + def read(self, start, end): + """The bytes at [start, end], seeked to rather than materialized.""" + with self.path.open("rb") as frame: + frame.seek(start) + return frame.read(end - start + 1) + + +class Handler(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + BOUNDARY = "c2boundary" + + def log_message(self, *args): + pass + + def _send(self, status, body, headers=()): + self.send_response(status) + for name, value in headers: + self.send_header(name, value) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def do_GET(self): # noqa: N802 (BaseHTTPRequestHandler's own spelling) + sub = self.server.subscriber + endpoint = self.path.split("/")[2] + if endpoint == "info": + self._send(200, json.dumps(sub.meta()).encode()) + elif endpoint == "chunk": + nchunk = int(self.path.split("nchunk=")[1]) + self._send(200, sub.array.schunk.get_chunk(nchunk)) + elif endpoint == "fetch": + self._fetch(sub) + else: + self._send(404, b"") + + def _fetch(self, sub): + wanted = self.headers.get("Range") + if sub.streamed: + # What the streaming paths answer since they were made honest: a 416 + # instead of the whole body with a 200 that no client could notice + if wanted: + self._send(416, b"", [("Accept-Ranges", "none")]) + else: + self._send(200, sub.read(0, sub.size - 1), [("Accept-Ranges", "none")]) + return + if not wanted: + self._send(200, sub.read(0, sub.size - 1), [("Accept-Ranges", "bytes")]) + return + spans = [] + for span in wanted.removeprefix("bytes=").split(","): + start, end = (int(n) for n in span.split("-")) + spans.append((start, min(end, sub.size - 1))) + # Starlette sorts the spans and merges the ones that touch, and answers a + # plain 206 when only one is left, so a client cannot count on a part per + # span nor on the order it asked in + spans.sort() + merged = [spans[0]] + for start, end in spans[1:]: + if start <= merged[-1][1] + 1: + merged[-1] = (merged[-1][0], max(merged[-1][1], end)) + else: + merged.append((start, end)) + if len(merged) == 1: + start, end = merged[0] + self._send( + 206, + sub.read(start, end), + [("Content-Range", f"bytes {start}-{end}/{sub.size}"), ("Accept-Ranges", "bytes")], + ) + return + body = b"" + for start, end in merged: + body += ( + f"--{self.BOUNDARY}\r\nContent-Type: application/octet-stream\r\n" + f"Content-Range: bytes {start}-{end}/{sub.size}\r\n\r\n" + ).encode() + body += sub.read(start, end) + b"\r\n" + body += f"--{self.BOUNDARY}--\r\n".encode() + self._send( + 206, + body, + [ + ("Content-Type", f"multipart/byteranges; boundary={self.BOUNDARY}"), + ("Accept-Ranges", "bytes"), + ], + ) + + +def stand_in(urlpath, streamed=False): + """Serve *urlpath* as ``@public/``, and return (server, urlbase, path).""" + server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Handler) + server.subscriber = Subscriber(urlpath, streamed) + threading.Thread(target=server.serve_forever, daemon=True).start() + urlbase = f"http://127.0.0.1:{server.server_address[1]}/" + return server, urlbase, f"@public/{pathlib.Path(urlpath).name}" + + +# +# What each mode would ask for +# + + +def default_patterns(shape): + """Slices worth asking about, for an array of any shape (as in the fsspec bench).""" + mid = [s // 2 for s in shape] + return [ + ("point", tuple(mid)), + ("line, last dim", (*mid[:-1], slice(None))), + ("line, first dim", (slice(None), *mid[1:])), + ( + "window (1/64 per dim)", + tuple(slice(m, m + max(1, s // 64)) for m, s in zip(mid, shape, strict=True)), + ), + ( + "slab (1% of dim 0)", + (slice(mid[0], mid[0] + max(1, shape[0] // 100)), *[slice(None)] * (len(shape) - 1)), + ), + ( + "slab (10% of dim 0)", + (slice(mid[0], mid[0] + max(1, shape[0] // 10)), *[slice(None)] * (len(shape) - 1)), + ), + ] + + +def chunk_cbytes(source, nchunks): + """The compressed size of each chunk, from 16 bytes of its header. + + Read rather than guessed at: the distance to the next chunk is an upper + bound only, and a frame with a hole in it would make the chunk mode look + dearer than it is. One request for the lot where the subscriber takes + several ranges, which is the same trick the fetch path uses. + """ + live = [n for n in nchunks if int(source._offsets[n]) >= 0] + heads = source.read_ranges([(int(source._offsets[n]), 16) for n in live]) if live else [] + sizes = dict.fromkeys(nchunks, 0) # a run-length chunk has no bytes in the file + for nchunk, head in zip(live, heads, strict=True): + sizes[nchunk] = struct.unpack(" 1 else 'no'}" + f" ({source.max_ranges} spans per request)\n" + f" frame index: {opening['requests']} requests, {opening['bytes']} bytes, once per C2Array" + ) + + proxy = blosc2.Proxy(array, mode="w") + print( + f"\n {'pattern':22s} {'chunks':>6s} {'blocks':>13s} {'chunk mode':>20s} " + f"{'block mode':>20s} {'multipart':>9s} ratio" + ) + plans = {} + for name, item in default_patterns(array.shape): + plan = plans[name] = request_plan(proxy, array, item) + print( + f" {name:22s} {plan['chunks touched']:6d} " + f"{plan['blocks wanted']:6d}/{plan['blocks total']:<6d} " + f"{plan['chunk requests']:5d} req {plan['chunk bytes'] / 1e6:7.2f} MB " + f"{plan['block requests']:5d} req {plan['block bytes'] / 1e6:7.2f} MB " + f"{plan['multipart requests']:5d} req {plan['ratio'] * 100:6.1f}%" + ) + + pooled, fresh = connection_setup(urlbase, path, token, args.reps) + print( + f"\n connection setup (api/info): {pooled * 1e3:6.1f} ms pooled, {fresh * 1e3:6.1f} ms with " + f"a client per request ({fresh / pooled:.1f}x)" + ) + _time_patterns(args, open_array, array, ["chunks", "blocks", "multipart"], latency, bandwidth, plans) + + +def _opened(array): + """What reading the frame index cost, which is paid once per C2Array.""" + fresh = c2array.C2Array(array.path, urlbase=array.urlbase, auth_token=array.auth_token) + tally = {"requests": 0, "bytes": 0} + original = c2array.C2NDSource.read_range + + def counted(self, offset, size): + tally["requests"] += 1 # before the read: a refused probe is a request too + data = original(self, offset, size) + tally["bytes"] += len(data) + return data + + c2array.C2NDSource.read_range = counted + try: + fresh.block_source() + finally: + c2array.C2NDSource.read_range = original + return tally + + +def _time_patterns(args, open_array, array, modes, latency, bandwidth, plans=None): + if latency or bandwidth: + print( + f"\n simulating a network: {latency * 1e3:.0f} ms per request" + + (f", {bandwidth / 1e6:.1f} MB/s across it" if bandwidth else "") + ) + header = "".join(f"{mode:>28s}" for mode in modes) + print(f"\n {'pattern':22s}{header}" + (" vs chunks" if len(modes) > 1 else "")) + for name, item in default_patterns(array.shape): + if plans and plans[name]["chunk bytes"] > args.max_mb * 1e6: + print(f" {name:22s} skipped ({plans[name]['chunk bytes'] / 1e6:.0f} MB > --max-mb)") + continue + results = {} + for mode in modes: + runs = [ + timed_slice(open_array, item, mode, args.concurrency, latency, bandwidth) + for _ in range(args.reps) + ] + results[mode] = (statistics.median(r[0] for r in runs), runs[0][1], runs[0][2]) + line = "".join( + f"{results[mode][1]:5d} req {results[mode][2] / 1e6:7.2f} MB {results[mode][0]:6.3f}s" + for mode in modes + ) + speedup = "" + if len(modes) > 1: # what the last mode, which is the shipped one, is worth + speedup = f" {results[modes[0]][0] / results[modes[-1]][0]:8.1f}x" + print(f" {name:22s}{line}{speedup}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index e0d038fea..9753a4acd 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -314,8 +314,18 @@ print(r.status_code, r.headers.get("content-range"), len(r.content)) # 200 None 14435027 -> served by a body builder, blocks must not be used ``` -`bench/ndarray/fsspec-block-granularity.py` measures the touch ratios of any -local array, which is what decides whether a given dataset would benefit at all. +`bench/ndarray/cat2-block-granularity.py` answers all of this for a dataset of +your choosing, and needs no server to point at (it stands one in over loopback): + +```sh +python bench/ndarray/cat2-block-granularity.py mydata.b2nd +python bench/ndarray/cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd \ + --urlbase https://cat2.cloud/demo +``` + +It says whether the dataset serves ranges at all, what each mode would ask for, +and what each one costs. `bench/ndarray/fsspec-block-granularity.py` is the same +question for an fsspec URL. ## What landed From f9f755cc063a8350050ee8308a48b8ce62f005d5 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:20:37 +0200 Subject: [PATCH 09/32] Say when a dataset has no chunks to fetch, rather than tracebacking Run against a .b2z member, the bench died in httpx: api/chunk resolves a path without its inner key and 404s, so a proxy over a container leaf cannot fetch chunks from it at all. That has never worked and is nothing this branch changed, but a tool whose job is to answer "what would this dataset cost" should answer it. Co-Authored-By: Claude Opus 5 --- bench/ndarray/cat2-block-granularity.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/bench/ndarray/cat2-block-granularity.py b/bench/ndarray/cat2-block-granularity.py index e1c9e0f92..7181cc49e 100644 --- a/bench/ndarray/cat2-block-granularity.py +++ b/bench/ndarray/cat2-block-granularity.py @@ -497,6 +497,17 @@ def open_array(): _time_patterns(args, open_array, array, ["chunks", "blocks", "multipart"], latency, bandwidth, plans) +def _no_chunks(array): + """Why api/chunk cannot serve this dataset, if it cannot.""" + import httpx + + try: + array.get_chunk(0) + except httpx.HTTPStatusError as exc: + return f"api/chunk answers {exc.response.status_code} for {array.path}" + return None + + def _opened(array): """What reading the frame index cost, which is paid once per C2Array.""" fresh = c2array.C2Array(array.path, urlbase=array.urlbase, auth_token=array.auth_token) @@ -518,6 +529,13 @@ def counted(self, offset, size): def _time_patterns(args, open_array, array, modes, latency, bandwidth, plans=None): + unavailable = _no_chunks(array) + if unavailable: + # A container member (a .b2z or .h5 leaf) is fetchable but not chunk-wise: + # api/chunk resolves a path without its inner key, so it 404s. A proxy + # over one cannot read it at all, whatever mode it would have used + print(f"\n nothing to time: {unavailable}") + return if latency or bandwidth: print( f"\n simulating a network: {latency * 1e3:.0f} ms per request" From 4fe79af4e4a14140922d5166b4b41cc8f61a713b Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:39:49 +0200 Subject: [PATCH 10/32] The api/chunk gap is fixed; note what a container leaf still lacks Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 9753a4acd..7a939fc7b 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -396,10 +396,19 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an ### Left undone -- **`api/chunk` does not serve container members.** A `Proxy` over a `.b2z` leaf - falls back to whole chunks correctly and then 404s, because `get_chunk` in the - server resolves the path without an inner key. It has never worked; nothing - here changed it, and nothing here depends on it. +- ~~**`api/chunk` does not serve container members.**~~ Fixed in Caterva2 on + `range-honesty` (*Serve chunks of a container leaf*): the endpoint resolves the + way `api/fetch` does, so a TreeStore leaf hands over its stored chunk, while + HDF5 leaves and CTables are refused with a 400 naming `slice_` rather than + being recompressed per request. A `.b2z` leaf still gets whole chunks only: + giving one the block path needs the offset of its frame inside the container, + which is the next item. +- **A container leaf could serve ranges too.** A TreeStore keeps its leaves as + ordinary frames inside the `.b2z`, so the bytes a block reader wants are in + the file at a fixed offset -- what is missing is a way for the server to say + where a leaf's frame starts, and for the client to add that base to every + range. Worth its own plan; it would give `.b2z` members everything a plain + `.b2nd` has. - **The four requests to open a frame** (prefix, header, offsets header, offsets) could be two: the header could be read optimistically with the prefix. It is in `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would From 944ca8004eb672c34364cc8289321e7a0694224d Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 07:48:55 +0200 Subject: [PATCH 11/32] Open a frame in two requests instead of four The format asks to be read in four steps -- how long is the header, the header, how long is the offsets chunk, the offsets chunk -- and each of the two questions costs a round trip to learn a number smaller than the answer it asks for. Over a network that is half the cost of opening an array. Both are guessed at instead: 8 KB of the head, which holds any ordinary header (165-320 bytes in practice), and the tail that the frame's own length bounds, capped at 64 KB so that a large trailer is not dragged along with the offsets (compressed offsets are small -- 4 KB for a frame of 100_000 chunks). A guess that falls short is followed by the exact read that would have happened anyway, so the worst case is the cost of today. A frame small enough to arrive whole in the first read costs one request: its offsets chunk is in those bytes already. Measured against cat2.cloud on kevlar-tomo.b2nd, opening a C2Array: as the format asks 0.237 s 4 requests, 303 bytes optimistic 0.138 s 2 requests, 8_306 bytes More bytes, fewer round trips, which is the trade worth making: 8 KB is under a millisecond on any link where a round trip is 45. It is in ByteRangeNDSource, so blosc2.open(url, lazy=True) gets it too. The seek/read shim the reader needed goes away with it -- it reads exact ranges now, which is what the transport underneath was doing all along. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 5 ++ plans/cat2-block-granularity.md | 10 ++-- src/blosc2/proxy.py | 79 ++++++++++++++++------------ tests/ndarray/test_c2array_blocks.py | 5 +- tests/test_fsspec.py | 57 ++++++++++++++++++++ 5 files changed, 116 insertions(+), 40 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index d153146c2..79419be93 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -56,6 +56,11 @@ XXX version-specific blurb XXX request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. + Opening a remote frame through either of them now costs two requests instead + of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small + enough to arrive whole in the first read: the two reads that only measured the + next one are guessed at generously instead, since over a network a few hundred + bytes and a few kilobytes cost the same. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 7a939fc7b..1c79d2261 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -409,10 +409,12 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an where a leaf's frame starts, and for the client to add that base to every range. Worth its own plan; it would give `.b2z` members everything a plain `.b2nd` has. -- **The four requests to open a frame** (prefix, header, offsets header, offsets) - could be two: the header could be read optimistically with the prefix. It is in - `ByteRangeNDSource`, so it would pay for fsspec as well, and multipart would - make it one. +- ~~**The four requests to open a frame**~~ are two, and one for a frame that + arrives whole in the first read (*Open a frame in two requests*): both reads + that only measured the next one are guessed at instead. 0.237 s → 0.138 s + against cat2.cloud. Getting to one for a large frame needs a suffix range + (`Range: bytes=-65536`) batched with the head read, which no fsspec backend + exposes and `read_ranges` has no way to express. - **`C2Array` still has no `stamp`.** A cache is checked against the source's geometry only, so a dataset replaced underneath while keeping its shape is not noticed. `mtime` is in `api/info` and would do it, but adding one invalidates diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 4a2027138..a323acd8e 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -52,6 +52,16 @@ # few unwanted bytes for one less request, which at object-store latencies is # always the right way round. BLOCK_GAP = 4096 +# Opening a frame asks two questions whose answers are shorter than the questions +# are dear: how long is the header, and how long is the offsets chunk. Both are +# guessed at generously instead, since over a network a read of a few hundred +# bytes and one of a few kilobytes cost the same. A guess that falls short costs +# the exact read that would have happened anyway, so these are ceilings, not +# promises: an ordinary header is 165-320 bytes, and a frame of 100_000 chunks +# has 4 KB of compressed offsets. +_FRAME_PREFETCH = 8192 +_INDEX_PREFETCH = 1 << 16 + # How many partly filled chunks keep their blocks in memory as well as in the # cache. Adding a block to a chunk rewrites that chunk, and the blocks already # in it have to come from somewhere: from here, or read back out of the cache and @@ -1101,25 +1111,33 @@ def _chunk_payloads(chunk: bytes, nblocks: int, wanted) -> dict[int, bytes]: return {int(n): chunk[bstarts[n] : bstarts[n] + extents[n]] for n in wanted} -def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: - """Read the header and the chunk offsets of a contiguous frame from *f*. +def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: + """Read the header and the chunk offsets of a contiguous frame. + + *read_range* is ``(offset, size) -> bytes``, so what this costs is round + trips. Returns the raw header bytes, the header decoded as the msgpack + array it is, and the absolute position of every chunk. A negative position + is not a position at all: it encodes a run-length chunk that was never + written to the file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the + layout. - Returns the raw header bytes, the header decoded as the msgpack array it is, - and the absolute position of every chunk. A negative position is not a - position at all: it encodes a run-length chunk that was never written to the - file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the layout. + The format asks to be read in four steps -- how long is the header, the + header, how long is the offsets chunk, the offsets chunk -- and each of the + two questions costs as much as the answer it asks for. So both are guessed + at instead: enough of the head to hold any ordinary header, and the tail + that the frame's own length bounds. A guess that falls short is followed by + the exact read that would have happened anyway, and a frame small enough to + arrive whole in the first read costs one. """ import msgpack - f.seek(0) - prefix = f.read(24) - if prefix[2:10] != _FRAME_MAGIC: + head = read_range(0, _FRAME_PREFETCH) + if head[2:10] != _FRAME_MAGIC: raise ValueError("not a Blosc2 contiguous frame") # header_len is the one field that must be located by hand; everything after # it comes out of unpacking the header, which is plain msgpack - header_len = struct.unpack(">i", prefix[11:15])[0] - f.seek(0) - raw = f.read(header_len) + header_len = struct.unpack(">i", head[11:15])[0] + raw = head[:header_len] if header_len <= len(head) else read_range(0, header_len) # raw=True because the flags field is a msgpack *string* holding four raw # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up # that byte is not valid UTF-8 and decoding the header blows up @@ -1130,31 +1148,24 @@ def _read_frame_index(f) -> tuple[bytes, list, np.ndarray]: if header[8] == 0: # chunksize return raw, header, np.empty(0, dtype=np.int64) - # The offsets live in a Blosc2 chunk of their own, right after the data ones - index_pos = header[1] + header[5] - f.seek(index_pos) - index_cbytes = struct.unpack("= frame_len: + index = head[index_pos:] # the whole frame arrived in the first read + else: + index = read_range(index_pos, min(frame_len - index_pos, _INDEX_PREFETCH)) + index_cbytes = struct.unpack(" len(index): + index = read_range(index_pos, index_cbytes) + offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) # Offsets are relative to the end of the header return raw, header, np.where(offsets >= 0, offsets + header_len, offsets) -class _RangeReader: - """The seek/read pair `_read_frame_index` needs, served by exact range requests.""" - - def __init__(self, read_range): - self._read_range, self._pos = read_range, 0 - - def seek(self, pos: int) -> None: - self._pos = pos - - def read(self, size: int) -> bytes: - data = self._read_range(self._pos, size) - self._pos += len(data) - return data - - def _frame_metalayer(raw: bytes, header: list, name: str): """Decode the *name* metalayer out of an already-read frame header.""" offset = header[13][1][name.encode()] # KeyError if there is no such metalayer @@ -1238,7 +1249,7 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(_RangeReader(self.read_range)) + raw, header, self._offsets = _read_frame_index(self.read_range) self._chunksize = header[8] self._extents = _chunk_extents(self._offsets, header) try: diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 46c2d384b..181f93cf1 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -195,8 +195,9 @@ def test_blocks_are_read_over_ranges(subscriber, any_chunk_wants_blocks): sub.log.clear() assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - # The frame index, then one read for the block offsets and one for the block - assert [kind for kind, _, _ in sub.log] == ["fetch"] * 6 + # The frame index (header, then offsets), one read for the chunk's block + # offsets, and one for the block the slice lands in + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 assert {status for _, status, _ in sub.log} == {206} assert not _bytes(sub, "chunk") # A block of a chunk, not the chunk: an eighth of it here, and never the frame diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index b988f0dcb..3d4f2b625 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -759,6 +759,63 @@ def _incompressible(shape, chunks, blocks, seed=0): return data, blosc2.asarray(data, chunks=chunks, blocks=blocks) +def test_lazy_open_costs_two_reads(monkeypatch): + # The format asks four questions to find the chunk offsets; both of the two + # that only measure the next read are guessed at instead + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("openreads.b2nd", a) + reads, chunks = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 2 + assert not chunks + assert src.shape == (600, 600) + assert len(src._offsets) == 2 # ... and it read the offsets it came for + + +def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): + # A frame that fits in the first read is wholly in hand: the offsets chunk + # is in those bytes too, so there is nothing left to ask for + a = blosc2.arange(0, 100, dtype="i4", chunks=(10,)) + assert a.schunk.cbytes < blosc2.proxy._FRAME_PREFETCH + url = _put("smallframe.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 1 + assert len(src._offsets) == 10 + assert np.array_equal(blosc2.Proxy(src)[:], a[:]) + + +def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): + # A metalayer big enough to push the header past the guess: the exact read + # the format asks for happens after all, and nothing is misread + monkeypatch.setattr(blosc2.proxy, "_FRAME_PREFETCH", 256) + data = np.arange(1000, dtype="i4") + a = blosc2.asarray(data, chunks=(100,), meta={"big": {"pad": "x" * 4096}}) + url = _put("bigheader.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 3 # the guess, the header, the offsets + assert reads[1] > 4096 + assert np.array_equal(blosc2.Proxy(src)[:], data) + + +def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): + # The same for the offsets chunk, which is bounded by the frame's own length + # but capped in case a large trailer sits behind it + monkeypatch.setattr(blosc2.proxy, "_INDEX_PREFETCH", 16) + data, a = _incompressible((600, 600), (30, 600), (30, 600)) + url = _put("bigindex.b2nd", a) + reads, _ = _traffic(monkeypatch) + + src = blosc2.FsspecNDSource(url) + assert len(reads) == 3 # the head, the capped tail, the offsets in full + assert reads[1] == 16 + assert np.array_equal(blosc2.Proxy(src)[:], data) + + def test_lazy_fetches_only_touched_blocks(monkeypatch): # Chunks big enough to be worth taking apart, at the real threshold data, a = _incompressible((600, 600), (300, 600), (30, 600)) From b2918c896ca9db52ece1f8d4de149ef4815c0bcf Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:00:26 +0200 Subject: [PATCH 12/32] Notice a remote array that was replaced under a proxy's cache A cache was checked against the source's geometry alone, so a dataset rewritten with the same shape and partitioning was adopted and went on being served from whatever the earlier run had fetched. In block mode that is worse than stale data: the cached chunks were spliced at offsets read from the frame the cache was built against, and the new frame's offsets are somewhere else. C2Array now has a stamp, like FsspecNDSource: api/info's mtime and the compressed size, both already fetched at construction, so it costs no request. The size is in there because a rewrite inside one clock tick is what an mtime cannot see. A subscriber reporting no mtime leaves the array unstamped and the cache checked on geometry alone, as before. The vlmeta entry is renamed fsspec-stamp -> proxy-stamp, since it is no longer only fsspec's. Caches from earlier builds of this development cycle are not adopted; mode="w" once is the way through, and nothing released is affected. Also: do not stamp a cache opened read-only. blosc2.open(path, mode="r") over a persisted proxy hands the Proxy a cache it may not write to, and recording the stamp there raised "Cannot do this action with reading mode" instead of opening it -- reachable only once a C2Array had a stamp to record, which is why the network tests caught it and the local ones did not. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 7 +- plans/cat2-block-granularity.md | 9 ++- src/blosc2/c2array.py | 19 +++++ src/blosc2/proxy.py | 16 ++-- src/blosc2/schunk.py | 2 +- tests/ndarray/test_c2array_blocks.py | 106 +++++++++++++++++++++++++-- tests/ndarray/test_proxy.py | 2 +- 7 files changed, 143 insertions(+), 18 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 79419be93..5bd236902 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -68,7 +68,12 @@ XXX version-specific blurb XXX earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same shape and dtype; anything else at that path raises. A cache that holds only - some blocks of a chunk keeps them across runs too. + some blocks of a chunk keeps them across runs too. Sources that can name the + bytes they read are held to that as well, so a remote array *replaced* while + keeping its shape is noticed rather than served stale: `FsspecNDSource` uses + fsspec's token and `C2Array` the subscriber's mtime, both free with metadata + they already fetch. Caches from earlier 4.11.1 development builds are not + adopted (the stamp moved to a `proxy-stamp` entry); pass `mode="w"` once. * Querying a `utf8()` column through its FULL index no longer materializes the index vocabulary. The query literal is turned into an alphabetical rank by diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 1c79d2261..90c5bab38 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -415,7 +415,8 @@ in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, an against cat2.cloud. Getting to one for a large frame needs a suffix range (`Range: bytes=-65536`) batched with the head read, which no fsspec backend exposes and `read_ranges` has no way to express. -- **`C2Array` still has no `stamp`.** A cache is checked against the source's - geometry only, so a dataset replaced underneath while keeping its shape is not - noticed. `mtime` is in `api/info` and would do it, but adding one invalidates - every cache built before it, which wants its own decision. +- ~~**`C2Array` still has no `stamp`.**~~ It has one now (*Notice a remote array + that was replaced*): `api/info`'s mtime and the compressed size, so a cache + built from other bytes raises instead of being served stale. The vlmeta entry + is `proxy-stamp` rather than `fsspec-stamp`, since it is no longer only + fsspec's; caches from earlier builds of this cycle are not adopted. diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index d2c1649dc..71a76777d 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -655,6 +655,25 @@ async def aclose(self) -> None: # slice touches instead of whole chunks, wherever that is the cheaper way # round; every one of them falls back to `get_chunk` when it is not. + @property + def stamp(self) -> str | None: + """What names the exact remote bytes, for a :ref:`Proxy` to check a cache by. + + Geometry cannot tell a dataset that was replaced from the one a cache was + filled from: a shape and a partitioning survive a rewrite, while every + cached chunk -- and, in block mode, every offset they were fetched by -- + goes stale. The subscriber's own mtime does tell, and `api/info` carries + it, so this costs no request; the compressed size goes in with it, since + a rewrite within the same clock tick is what an mtime cannot see. + + None when the subscriber reports no mtime, which leaves the cache checked + on its geometry alone, as every source without a stamp is. + """ + mtime = self.meta.get("mtime") + if mtime is None: + return None + return f"{mtime}:{self.meta['schunk'].get('cbytes', '')}" + @property def blocks_per_chunk(self) -> int: """How many blocks a chunk of the remote array holds. diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index a323acd8e..ce75dd869 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -72,7 +72,7 @@ # vlmeta entries the proxy keeps its own state in: what it has fetched, and which # remote bytes the cache was filled from. A caller cannot write these. -_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "fsspec-stamp"}) +_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp"}) # `jit` kwargs that tune *how* an expression is evaluated, not what container the # result is stored in. Unlike storage kwargs (`cparams`, `chunks`, `urlpath`, ...), @@ -294,7 +294,8 @@ def __init__( else raises rather than being silently reused or overwritten. A source that can name the exact bytes it reads, as - :ref:`FsspecNDSource` does with its ``stamp``, is checked against that + :ref:`FsspecNDSource` does with its ``stamp`` (fsspec's token) and + :ref:`C2Array` with the subscriber's mtime, is checked against that too: a cache built from different bytes raises, even when the geometry still fits. For every other source geometry is all there is to check, so a source whose contents changed underneath while its geometry did @@ -396,14 +397,17 @@ def __init__( # Geometry alone cannot tell a replaced source from the one the cache was # filled from, so record whatever identity the source can name itself by stamp = getattr(self.src, "stamp", None) - if stamp is not None: - self._schunk_cache.vlmeta["fsspec-stamp"] = stamp + if stamp is not None and getattr(self._schunk_cache, "mode", None) != "r": + # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` + # hands over for a persisted proxy: nothing may be written there, and a + # proxy over one stays observational anyway (see `__getitem__`) + self._schunk_cache.vlmeta["proxy-stamp"] = stamp if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) if reserved: # Writing these would hand the proxy a bitmap or an identity it # never earned: a caller's `proxy-fetched` makes it skip chunks it - # has not fetched, and a caller's `fsspec-stamp` makes a good cache + # has not fetched, and a caller's `proxy-stamp` makes a good cache # fail its identity check (or a stale one pass it) raise ValueError( f"{', '.join(reserved)} {'is' if len(reserved) == 1 else 'are'} reserved " @@ -594,7 +598,7 @@ def _reopen_cache(self, urlpath: str): # goes stale. Only for sources that can name themselves; the rest are # adopted on geometry alone, as documented. stamp = getattr(self.src, "stamp", None) - if stamp is not None and schunk.vlmeta.get("fsspec-stamp") != stamp: + if stamp is not None and schunk.vlmeta.get("proxy-stamp") != stamp: raise ValueError( f"the cache at {urlpath} was built against different remote bytes; " f"pass mode='w' to fetch them anew" diff --git a/src/blosc2/schunk.py b/src/blosc2/schunk.py index f6e32e90a..c06ba2b9f 100644 --- a/src/blosc2/schunk.py +++ b/src/blosc2/schunk.py @@ -2011,7 +2011,7 @@ def _cache_stamp(path: str): cache = blosc2_ext.open(path, "r", 0, **kwargs) except RuntimeError: return None - return getattr(cache, "schunk", cache).vlmeta.get("fsspec-stamp") + return getattr(cache, "schunk", cache).vlmeta.get("proxy-stamp") def _open_fsspec_url(urlpath: str, mode: str, offset: int, kwargs: dict): diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 181f93cf1..9376144d3 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -18,6 +18,7 @@ import contextlib import json import math +import os import pathlib import threading from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer @@ -33,13 +34,18 @@ class _Subscriber: def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) - self.frame = pathlib.Path(self.path).read_bytes() - self.array = blosc2.open(self.path) self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does self.multipart = multipart # False: answer only the first range asked for self.merge_ranges = merge_ranges # as Starlette does with ranges that touch self.log = [] # (endpoint, status, bytes served) + self.reload() + + def reload(self): + """Pick up the file as it is now, as a subscriber would on the next request.""" + self.frame = pathlib.Path(self.path).read_bytes() + self.array = blosc2.open(self.path) + self.mtime = pathlib.Path(self.path).stat().st_mtime @property def meta(self): @@ -49,7 +55,7 @@ def meta(self): "chunks": list(self.array.chunks), "blocks": list(self.array.blocks), "dtype": str(self.array.dtype), - "mtime": None, + "mtime": self.mtime, "schunk": { "cparams": {"typesize": self.array.dtype.itemsize}, "nbytes": schunk.nbytes, @@ -329,9 +335,10 @@ def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_bl array._block_source = None # as if the subscriber served no ranges p = blosc2.Proxy(array, urlpath=cache, mode="a") assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) - del p, array + del p - array, _ = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + # The same dataset, opened afresh: this one takes the blocks path + array = blosc2.C2Array(array.path, urlbase=array.urlbase) p = blosc2.Proxy(array, urlpath=cache, mode="a") served = len(sub.log) assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) @@ -407,3 +414,92 @@ def test_a_server_that_answers_one_range_stops_being_batched(subscriber, any_chu # One request per range from here on, and no second attempt at batching assert len(sub.log) > served + 2 assert np.array_equal(p[...], data) + + +# --- a cache is checked against the bytes it was filled from ---------------- + + +def _replace(sub, data, chunks, blocks): + """Rewrite the served dataset, as an upload of new data would.""" + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=sub.path, mode="w") + stat = pathlib.Path(sub.path).stat() + os.utime(sub.path, (stat.st_atime, stat.st_mtime + 10)) # a tick the clock cannot swallow + sub.reload() + + +def test_a_cache_is_stamped_with_the_remote_mtime(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + + assert array.stamp == f"{sub.mtime}:{sub.array.schunk.cbytes}" + assert p.schunk.vlmeta["proxy-stamp"] == array.stamp + + +def test_a_cache_from_other_bytes_is_refused(tmp_path, subscriber, any_chunk_wants_blocks): + # Same shape, same partitioning, different data: geometry cannot tell, and + # every cached chunk (and the offsets it was fetched by) is stale + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "stamped.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + other = _incompressible((200, 200), seed=1) + _replace(sub, other, chunks=(100, 200), blocks=(10, 20)) + replaced = blosc2.C2Array(array.path, urlbase=array.urlbase) + assert replaced.stamp != array.stamp + + with pytest.raises(ValueError, match="different remote bytes"): + blosc2.Proxy(replaced, urlpath=cache, mode="a") + + # ... and mode="w" is the way through, with the new data behind it + p = blosc2.Proxy(replaced, urlpath=cache, mode="w") + assert np.array_equal(p[0:5, 0:10], other[0:5, 0:10]) + assert np.array_equal(p[...], other) + + +def test_a_cache_from_the_same_bytes_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "unchanged.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + assert again.stamp == array.stamp + p = blosc2.Proxy(again, urlpath=cache, mode="a") + served = len(sub.log) + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert len(sub.log) == served # what it holds was not fetched again + + +def test_no_stamp_when_the_subscriber_reports_no_mtime(tmp_path, subscriber): + # Then the cache is checked on geometry alone, as every unstamped source is + data = _incompressible((200, 200)) + array, _sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + del array.meta["mtime"] + assert array.stamp is None + + cache = str(tmp_path / "unstamped.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="a") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert "proxy-stamp" not in p.schunk.vlmeta.getall() + del p + assert blosc2.Proxy(array, urlpath=cache, mode="a") is not None + + +def test_a_read_only_cache_is_not_stamped(tmp_path, subscriber): + # `blosc2.open(path, mode="r")` rebuilds the proxy over a cache that may not + # be written to; recording the stamp there raised instead of opening it + data = _incompressible((200, 200)) + array, _sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "readonly.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + reopened = blosc2.open(cache, mode="r") + assert np.array_equal(reopened[0:5, 0:10], data[0:5, 0:10]) diff --git a/tests/ndarray/test_proxy.py b/tests/ndarray/test_proxy.py index 1fffce5aa..3b611a84d 100644 --- a/tests/ndarray/test_proxy.py +++ b/tests/ndarray/test_proxy.py @@ -369,7 +369,7 @@ def test_evicted_chunk_is_fetched_again(): def test_vlmeta_cannot_overwrite_proxy_state(): # A caller-supplied bitmap would make the proxy skip chunks it never fetched source = blosc2.asarray(np.arange(20).reshape(4, 5), chunks=(2, 5), blocks=(1, 5)) - for name in ("proxy-fetched", "proxy-fetched-blocks", "fsspec-stamp"): + for name in ("proxy-fetched", "proxy-fetched-blocks", "proxy-stamp"): with pytest.raises(ValueError, match="reserved"): blosc2.Proxy(source, vlmeta={name: b"nonsense"}) # Anything else still goes through From 7b9aa6230160c2dbe69503fc252dc98fb17671ac Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:03:25 +0200 Subject: [PATCH 13/32] Record the whole of the block-granularity work in the plan The four follow-ons (the bench, the container-leaf chunks, the two-request open, the stamp) were only visible as strikethroughs in a list of things left undone, and the headline numbers predated the last two commits. Both are now one run of the bench against cat2.cloud, which is where those numbers should come from anyway. Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 153 ++++++++++++++++++++------------ 1 file changed, 95 insertions(+), 58 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 90c5bab38..13d62df53 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -3,10 +3,12 @@ Written 2026-08-17, after [plans/fsspec-blocks.md](fsspec-blocks.md) landed block fetching for fsspec URLs (merged as PR #701). -**All five phases are implemented** (2026-08-17). Phases 1-4 are in blosc2 on -`cat2-block-granularity`; phase 5 is in Caterva2 on `range-honesty`. See -[what landed](#what-landed) at the end for the results and the two things the -work found out. +**All five phases are implemented**, and four more things that came out of them +(2026-08-17/18). The blosc2 side is on `cat2-block-granularity`, the Caterva2 +side on `range-honesty`. Everything from *What was verified* to *Reproducing the +measurements* is the analysis as it stood before any of it was written; see +[what landed](#what-landed) at the end for what was built, what it measures at, +and where the analysis turned out to be wrong. ## The question @@ -329,8 +331,11 @@ question for an fsspec URL. ## What landed -Every phase, in the order the plan gives them. Where the plan guessed and the -work found out otherwise, that is said below rather than quietly fixed. +All five phases, and four more things the work turned up on the way. Where the +plan guessed and the work found out otherwise, that is said below rather than +quietly fixed. + +### The five phases | phase | where | commit | |---|---|---| @@ -338,85 +343,117 @@ work found out otherwise, that is said below rather than quietly fixed. | 2. capability check | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | | 3b. `ByteRangeNDSource` | blosc2 `proxy.py` | *Lift the frame reading out of FsspecNDSource* | | 3. `C2Array` blocks | blosc2 `c2array.py` | *Read a C2Array's blocks over HTTP ranges* | -| 4. multipart | blosc2 both | *Ask a subscriber for a whole wave of ranges at once* | -| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges* | +| 4. multipart | blosc2 `proxy.py`, `c2array.py` | *Ask a Caterva2 subscriber for a whole wave of ranges at once* | +| 5. honest streaming | caterva2 `server.py` | *Say which responses serve byte ranges, and refuse the rest* | Option **(b)** was taken for phase 3, as recommended: `ByteRangeNDSource` holds the frame format and one abstract `read_range`, `FsspecNDSource` is that plus four lines of fsspec, and `C2NDSource` is that plus HTTP ranges with the auth -cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, +cookie. `C2Array` keeps the five members `Proxy` looks for and delegates them, so every existing `Proxy(C2Array(...))` gets blocks without being asked. +### And four things that came out of it + +| | where | commit | +|---|---|---| +| a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | +| chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four* | +| a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | + +- **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question + asked of a subscriber, and it needs no service to point at: a stand-in serves + `api/info`, `api/fetch` and `api/chunk` over loopback from any local `.b2nd`, + ranges, multipart and Starlette's sort-and-merge included. It reports whether + a dataset serves ranges at all and what finding out cost, the request plan of + each mode, what a pooled connection is worth, and the timed slices. +- **`api/chunk` serves container leaves**, which it never did: it was the last + read endpoint resolving with `get_abspath` rather than `split_and_resolve`, so + a `Proxy` over a `.b2z` leaf could not fetch anything at all. HDF5 leaves and + CTables are refused with a 400 naming `slice_` instead of being recompressed + per request, and the opened leaf is cached (keyed on the container's mtime, as + `get_filtered_array` is) so a chunk-by-chunk read does not reopen the `.b2z` + every time. +- **Opening a frame costs two requests**, or one when the frame arrives whole in + the first read: the two reads that only measured the next one are guessed at + instead (8 KB of head, and a tail the frame's own length bounds). It is in + `ByteRangeNDSource`, so `blosc2.open(url, lazy=True)` gets it too. +- **`C2Array` has a stamp**: `api/info`'s mtime and the compressed size, so a + cache built against a dataset that was since replaced raises instead of being + served stale. In block mode that is worse than stale data -- the cached chunks + were spliced at offsets read from the frame the cache was built against. + ### The numbers, end to end -Against `cat2.cloud/demo` on `kevlar-tomo.b2nd` (1.44 MB chunks, 47 blocks each): +`bench/ndarray/cat2-block-granularity.py @public/examples/kevlar-tomo.b2nd +--urlbase https://cat2.cloud/demo` (1.44 MB chunks, 47 blocks each), median of 5: -| | requests | bytes | time | -|---|---|---|---| -| chunks (before) | 2 | 2.723 MB | 0.28 s | -| blocks | 8 | 0.031 MB | 0.34 s | +| pattern | chunk mode | blocks | multipart | vs chunks | +|---|---|---|---|---| +| point | 1 req, 1.38 MB, 0.169 s | 2 req, 0.02 MB, 0.097 s | 2 req, 0.02 MB, 0.088 s | 1.9x | +| line, last dim | 1 req, 1.38 MB, 0.140 s | 2 req, 0.02 MB, 0.087 s | 2 req, 0.02 MB, 0.090 s | 1.6x | +| line, first dim | 10 req, 14.43 MB, 0.956 s | 20 req, 0.20 MB, 0.198 s | **2 req, 0.20 MB, 0.111 s** | **8.6x** | +| window (1/64 per dim) | 1 req, 1.38 MB, 0.160 s | 2 req, 0.02 MB, 0.092 s | 2 req, 0.02 MB, 0.089 s | 1.8x | +| slab (1% of dim 0) | 1 req, 1.38 MB, 0.169 s | 1 req, 1.38 MB, 0.155 s | 1 req, 1.38 MB, 0.161 s | 1.0x | +| slab (10% of dim 0) | 1 req, 1.38 MB, 0.155 s | 1 req, 1.38 MB, 0.148 s | 1 req, 1.38 MB, 0.149 s | 1.0x | -88x fewer bytes for a corner slice, and the same wall time on a link where a -round trip and a megabyte cost about the same. For a slice touching ten chunks, -where the request count is what decides: +1.4% of the bytes for anything that lands in a corner of a chunk, and the two +slabs that want every block of theirs come out identical in all three modes, +which is the threshold declining to take a chunk apart. Around it: -| | requests | time | +| | before | after | |---|---|---| -| blocks, one request at a time | 20 | 1.008 s | -| blocks, fetches overlapped | 20 | 0.334 s | -| blocks, multipart | 2 | 0.141 s | +| one request (`api/info`) | 163.3 ms, a client per request | **42.3 ms**, pooled (3.9x) | +| opening a `C2Array` | 0.237 s, 4 requests, 303 bytes | **0.138 s**, 2 requests, 8306 bytes | +| a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | -Phase 1 on its own: 0.162 s per request against 0.046 s pooled, on the existing -chunk path. +Phase 4 was built because that last row said to. Starlette *sorts and merges* +the spans it is given and answers a plain 206 when they all merge into one, so +the client maps the parts back by what each says it holds rather than by order; +a server that answers with less than was asked for is noticed once and never +batched again. -### Two things the plan had wrong +### What the work found out - **`C2Array` cannot be built over a computed dataset at all.** `api/info` for a lazy expression carries no `schunk`, and `C2Array.__init__` reads - `meta["schunk"]["cparams"]`, so it raises long before any of this. The + `meta["schunk"]["cparams"]`, so it raises long before any of this. The info-based discriminator of phase 2 is still there and still right, but it earns its place on the *other* case: - **A `.b2z` member reports a full geometry and is streamed.** Confirmed against a local server: `api/info` on `@public/tree-store.b2z/level1/leaf6` answers - with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the + with `blocks`, `chunks` and `schunk`, and `api/fetch` streams it. So the status code of the first range read is the authority, exactly as phase 2 - argued — with phase 5 in place that costs 169 bytes and one round trip. - -Phase 4 was built because the measurement said to: 32 spans cost 0.136 s in one -multipart request against 0.208 s as 32 requests eight at a time, and 1.530 s -one at a time. Starlette *sorts and merges* the spans it is given and answers a -plain 206 when they all merge into one, so the client maps parts back by what -each says it holds rather than by order; a server that answers with less than -was asked for is noticed once and never batched again. - -The plan's guess that `_run` would already overlap a C2Array's fetches was wrong -in the other direction: `Proxy.fetch` reads `max_concurrency` off the source, and -`C2Array` had none, so the sync path was serial. It has one now, the same 8 -`afetch` already used. + argued -- with phase 5 in place that costs 169 bytes and one round trip. +- **A `Proxy` over a container leaf never worked**, which is how the bench found + the `api/chunk` gap: it died in httpx rather than reporting anything. Fixed + above; the bench now says so plainly when a dataset has no chunks to serve. +- **`Proxy.fetch` was serial over a C2Array.** It reads `max_concurrency` off the + source and `C2Array` had none, so the plan's assumption that a chunk's runs + already go out in parallel was wrong. It has one now, the same 8 `afetch` + already used, and that alone takes the unbatched block path from 1.008 s to + 0.334 s. +- **A read-only cache cannot be stamped.** `blosc2.open(path, mode="r")` over a + persisted proxy hands `Proxy` a cache it may not write to, and recording the + stamp there raised instead of opening it -- reachable only once a `C2Array` had + a stamp to record, and caught by the network tests rather than the local ones. ### Left undone -- ~~**`api/chunk` does not serve container members.**~~ Fixed in Caterva2 on - `range-honesty` (*Serve chunks of a container leaf*): the endpoint resolves the - way `api/fetch` does, so a TreeStore leaf hands over its stored chunk, while - HDF5 leaves and CTables are refused with a 400 naming `slice_` rather than - being recompressed per request. A `.b2z` leaf still gets whole chunks only: - giving one the block path needs the offset of its frame inside the container, - which is the next item. - **A container leaf could serve ranges too.** A TreeStore keeps its leaves as ordinary frames inside the `.b2z`, so the bytes a block reader wants are in the file at a fixed offset -- what is missing is a way for the server to say where a leaf's frame starts, and for the client to add that base to every - range. Worth its own plan; it would give `.b2z` members everything a plain + range. Worth its own plan; it would give `.b2z` members everything a plain `.b2nd` has. -- ~~**The four requests to open a frame**~~ are two, and one for a frame that - arrives whole in the first read (*Open a frame in two requests*): both reads - that only measured the next one are guessed at instead. 0.237 s → 0.138 s - against cat2.cloud. Getting to one for a large frame needs a suffix range - (`Range: bytes=-65536`) batched with the head read, which no fsspec backend - exposes and `read_ranges` has no way to express. -- ~~**`C2Array` still has no `stamp`.**~~ It has one now (*Notice a remote array - that was replaced*): `api/info`'s mtime and the compressed size, so a cache - built from other bytes raises instead of being served stale. The vlmeta entry - is `proxy-stamp` rather than `fsspec-stamp`, since it is no longer only - fsspec's; caches from earlier builds of this cycle are not adopted. +- **Opening a frame could be one request rather than two**, with a suffix range + (`Range: bytes=-65536`) batched with the head read. No fsspec backend exposes + suffix ranges and `read_ranges` has no way to express one, so it would be a + C2-only path through the shared base -- which is why it was not taken. +- **Peer datasets stay on the streaming side.** They are fetched from their owner + and re-serialized here, so there is no file to seek into; phase 5 makes them + say so, and that is all they can do without a design of their own. +- **Quota and accounting**, from the risks above, is still undecided: block reads + replace one chunk GET with a handful of small ranged ones, and multipart puts + a whole wave in one. A subscriber metering per request would see block clients + as expensive while they transfer far less. From b6b7303a26822c4857090aaa4e33fa6d9a775947 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:09:41 +0200 Subject: [PATCH 14/32] Plan byte ranges for container leaves A .b2z is a zip of stored frames, so a leaf's bytes are a contiguous frame at a fixed offset and blosc2 already computes where. If api/fetch answers a leaf's ranged request by seeking into the container, every client that reads a .b2nd over ranges reads a leaf the same way, knowing nothing about containers. Co-Authored-By: Claude Opus 5 --- plans/container-leaf-ranges.md | 171 +++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 plans/container-leaf-ranges.md diff --git a/plans/container-leaf-ranges.md b/plans/container-leaf-ranges.md new file mode 100644 index 000000000..8e1ab4d16 --- /dev/null +++ b/plans/container-leaf-ranges.md @@ -0,0 +1,171 @@ +# Byte Ranges For Container Leaves + +Analysis and plan, written 2026-08-18, after +[plans/cat2-block-granularity.md](cat2-block-granularity.md) gave `C2Array` the +block path and left this as the one thing a `.b2z` leaf still could not have. + +## The question + +A `Proxy` over `@public/tree.b2z/g/a` fetches whole chunks. A proxy over +`@public/a.b2nd` fetches the blocks a slice touches, because Caterva2 serves a +stored dataset from its file and `Range` works on it. A leaf is stored too -- +inside a container -- so what stands between it and the same treatment? + +**Verdict: one server change, and no client change at all.** A leaf's frame is a +contiguous, self-consistent Blosc2 frame at a fixed offset in the `.b2z`, and +blosc2 already knows where. If `api/fetch` answers a leaf's ranged request by +seeking into the container, every client that can read a `.b2nd` over ranges +reads a leaf the same way, knowing nothing about containers. + +## What was verified + +Against a local server from `~/ironArray/caterva2` and stores written by +`blosc2.TreeStore`. + +### A `.b2z` is a zip of *stored* frames + +``` +a.b2nd method=0 size=287667 header_offset=0 +g/b.b2nd method=0 size= 1835 header_offset=287703 +embed.b2e method=0 size= 245 header_offset=289576 +``` + +`method=0` is the whole point: every member is written uncompressed, so a +member's bytes in the file *are* the frame. Each one is self-consistent -- +`b2frame` magic at byte 2, and the `frame_len` in its header equals the member's +length -- so a reader handed the window sees an ordinary frame beginning at 0. + +blosc2 computes the windows already, for its own reading: `DictStore.offsets` +maps a zip member to `{offset, length}` (parsed off the local file headers) and +`DictStore.map_tree` maps a logical key to that member: + +``` +map_tree['/g/deep'] = 'g/deep.b2nd' +offsets['g/deep.b2nd'] = {'offset': 2901, 'length': 240} +``` + +Every leaf of a `TreeStore` is such a member, whether it was given an `NDArray`, +a plain NumPy array or an `SChunk` (`.b2nd`, `.b2f`). + +**The stored member and `leaf.to_cframe()` are not the same bytes.** Same +length, same data, same vlmeta, but not byte-identical -- what `api/fetch` +serves today for a whole member is a re-serialization, not the file. Nothing +depends on the two agreeing (a client reads geometry from `api/info` and the +frame index over ranges), but it decides one thing in the plan below: the +whole-member path is left alone. + +### An embedded leaf is not addressable, and that is right + +`EmbedStore` keeps its members inside a Blosc2 super-chunk of its own +(`self._store[offset:offset+len] = serialized`), which is compressed, so an +embedded leaf has no raw window in the file. A `C2Array` leaf is a reference +rather than bytes at all. Both must say "no window" rather than a wrong one -- +the same rule as before: **blocks where the bytes already exist as a frame**. + +### The server has no route to those bytes today + +``` +GET api/fetch/@public/inspect.b2z -> 500 (!) +GET api/fetch/@public/inspect.b2z/a Range: -> 416 (correct today: it is built, not stored) +GET api/download/@public/inspect.b2z Range: -> 416 (correct: download never serves ranges) +``` + +The 500 is a bug of its own: a `TreeStore` container falls through +`fetch_data`'s type ladder into the `SChunk` branch and dies on +`schunk.typesize`, where the docstring promises "its stored image is served". + +### What the client would need: nothing + +`C2Array` over a leaf already gets its geometry from `api/info` (a leaf reports +`chunks`, `blocks` and `schunk` like any array), already probes with a `Range`, +already falls back to `api/chunk` when the probe fails, and already reads a +frame that starts at 0. If the server maps a leaf's ranges onto the container, +all of that works unchanged. + +## What not to build + +- **A new endpoint.** `api/fetch` is where a dataset's bytes come from, leaf or + not; a second door would need its own auth, its own tests and its own client. +- **A client that parses the zip itself.** It would have to know the zip layout + *and* blosc2's member-naming convention, and there is no route to the + container's bytes anyway (`api/fetch` on a container 500s, `api/download` + refuses ranges) -- so it needs a server change regardless, and a bigger one. +- **Serving a whole member from its window.** Cheaper than rebuilding the + cframe, and tempting, but it would change the bytes clients get today for the + sake of an optimization nobody asked for. +- **HDF5 leaves.** An HDF5 dataset is not a Blosc2 frame; there is no window to + hand over. They keep the 400 that names `slice_`. + +## Plan + +### Phase 1 — `DictStore.member_window(key)` (blosc2) + +The format knowledge belongs where the format is: + +```python +def member_window(self, key: str) -> tuple[int, int] | None: + """Where the frame behind *key* lies in the ``.b2z``, as (offset, nbytes).""" +``` + +`None` for a directory-backed store, an embedded leaf, a `C2Array` reference, or +a key that names no leaf. Built out of `map_tree` and `offsets`, which are +already maintained; ~10 lines and a test that the window decodes to the leaf. + +### Phase 2 — Serve a leaf's ranges from the container (caterva2) + +In `fetch_data`, where a member currently refuses every range: + +- look the window up (through the cached container opened for `api/chunk`); +- with no `Range`, answer as today (`Accept-Ranges: bytes` instead of `none`); +- with one, answer 206 from the container file, offset by the window's start and + clamped to its length -- a range must never reach past the leaf it names, both + because the client's frame would make no sense and because the window is the + only part of the container this path is about. + +RFC 7233 by hand, which is ~60 lines: parse, sort, merge what touches, single +206 with `Content-Range` or `multipart/byteranges` for several, 416 with +`Content-Range: bytes */len` for the unsatisfiable, 400 for the malformed. The +client already survives merged and reordered parts, since Starlette does that to +ranges too, and falls back to a range per request against a server that answers +with less than it asked for -- so multipart is worth implementing rather than +leaving to that fallback. + +### Phase 3 — A container fetched whole stops being a 500 (caterva2) + +`api/fetch/{container}` should return the stored image, as its docstring says: +the `.b2z` itself, through the same `FileResponse` a `.b2nd` gets. Two lines and +a test. Independent of the rest, and worth doing on its own. + +### Phase 4 — Tests, and the bench + +Server: a leaf's ranged request returns exactly the container's bytes at the +window; a range past the leaf's end is clamped; a multi-range request comes back +multipart; an embedded leaf and an HDF5 leaf still refuse; the whole-member fetch +still returns what it always did. + +Client: nothing changes, which is the thing to assert -- a `Proxy` over a leaf +reads blocks, over the stand-in subscriber, and its traffic is a fraction of the +chunk path's. + +Then `bench/ndarray/cat2-block-granularity.py @public/tree.b2z/leaf` against a +local server, which should print `byte ranges: served` where it printed `not +served` and time the three modes as it does for a plain array. + +## Risks and open questions + +- **A hand-rolled RFC 7233.** Starlette's is a classmethod on `FileResponse` and + could be reused, at the price of depending on a private API; the alternative + is ~25 lines of parsing with tests of its own. Either way the client is the + same one that already reads Starlette's answers, so the two must agree on + merging and on what a 416 looks like. +- **The window is only as fresh as the store index.** The opened container is + cached keyed on its mtime (as `api/chunk` does), so a rewritten `.b2z` gets + new windows; a `.b2z` rewritten *in place* within one mtime tick would serve + stale offsets, which is the same exposure every cached read here has. +- **A leaf has no mtime of its own**, so a `C2Array` over one stamps its proxy + cache with the container's. Coarse but never wrong: rewriting any leaf + rewrites the container. +- **Compressed zip members would break this silently.** Nothing in blosc2 writes + them today (`method=0` throughout), but a `.b2z` produced by another tool + could, and its window would decode to nonsense. The member's own frame magic + is what catches that, on the server, before the window is offered. From e0cc7da670ebaaee8ed06e0029a0247ad6a8a1d0 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:11:36 +0200 Subject: [PATCH 15/32] Say where a leaf's frame lies inside a .b2z A zip store writes each external leaf uncompressed, so a member's bytes in the file are the Blosc2 frame that leaf would have been written as on its own: self-contained, beginning at a known offset, and readable by anything that reads a frame. The store computes those windows already, for its own reading -- map_tree names the member, offsets says where it is -- but nothing could ask. member_window(key) is that question, and it answers None where there is no window rather than a wrong one: a directory-backed store keeps leaves in files of their own, an embedded leaf lives inside the store's own compressed super-chunk, and a C2Array leaf is a reference rather than bytes. Wanted by a server that would hand a byte-range reader the window instead of rebuilding the leaf per request, which is plans/container-leaf-ranges.md. Co-Authored-By: Claude Opus 5 --- doc/reference/dict_store.rst | 6 ++++- src/blosc2/dict_store.py | 45 ++++++++++++++++++++++++++++++++++++ tests/test_dict_store.py | 42 +++++++++++++++++++++++++++++++++ 3 files changed, 92 insertions(+), 1 deletion(-) diff --git a/doc/reference/dict_store.rst b/doc/reference/dict_store.rst index 38bd00579..fd3078af6 100644 --- a/doc/reference/dict_store.rst +++ b/doc/reference/dict_store.rst @@ -90,7 +90,11 @@ Quick example ----------- Use :meth:`DictStore.to_b2z` to pack a directory-backed store into a ``.b2z`` archive, and :meth:`DictStore.to_b2d` to materialize a store as a - ``.b2d`` directory. + ``.b2d`` directory. :meth:`DictStore.member_window` says where a leaf's + frame lies inside a ``.b2z``, for a reader that can take a byte window + rather than the leaf itself. + + .. automethod:: member_window Public Members -------------- diff --git a/src/blosc2/dict_store.py b/src/blosc2/dict_store.py index 82ada0d3a..035be5971 100644 --- a/src/blosc2/dict_store.py +++ b/src/blosc2/dict_store.py @@ -439,6 +439,51 @@ def _update_map_tree_from_offsets(self): if os.path.splitext(filepath)[1] in external_exts or self._probe_external_leaf_offset(filepath): self.map_tree[self._logical_key_from_relpath(filepath)] = filepath + def member_window(self, key: str) -> tuple[int, int] | None: + """Where the frame behind *key* lies in the ``.b2z``, as ``(offset, nbytes)``. + + A zip store keeps each external leaf as a *stored* member, so those bytes + are the leaf's Blosc2 frame as it would have been written on its own: + self-contained, beginning at ``offset``, and readable by anything that + reads a frame -- ``blosc2.open(path, offset=offset)`` is what this store + does with it, and a server can hand the same window to a byte-range + reader instead of rebuilding the leaf per request. + + None when there is no such window, which is not an error but an answer: + a directory-backed store keeps its leaves in files of their own, an + embedded leaf lives inside the store's own compressed super-chunk, a + :ref:`C2Array` leaf is a reference rather than bytes, and a key that + names a group or nothing at all has none either. + + Parameters + ---------- + key: str + The logical key of the leaf, as :meth:`keys` reports it. + + Returns + ------- + out: tuple or None + ``(offset, nbytes)`` into the ``.b2z`` file, or None. + + Examples + -------- + >>> import numpy as np, blosc2 + >>> with blosc2.TreeStore("win.b2z", mode="w") as tstore: + ... tstore["/a"] = np.arange(1000, dtype="i4") + >>> tstore = blosc2.open("win.b2z") + >>> offset, nbytes = tstore.member_window("/a") + >>> frame = open("win.b2z", "rb").read()[offset : offset + nbytes] + >>> np.array_equal(blosc2.ndarray_from_cframe(frame)[:], np.arange(1000, dtype="i4")) + True + """ + if not self.is_zip_store: + return None + relpath = self.map_tree.get(key) + if relpath is None: + return None + window = self.offsets.get(relpath) + return (window["offset"], window["length"]) if window else None + def _annotate_external_value( self, key: str, diff --git a/tests/test_dict_store.py b/tests/test_dict_store.py index 92bd8bb15..7405c5dd5 100644 --- a/tests/test_dict_store.py +++ b/tests/test_dict_store.py @@ -6,6 +6,7 @@ ####################################################################### import os +import pathlib import shutil import subprocess import sys @@ -852,3 +853,44 @@ def test_dict_store_overwrite_key_across_tiers(tmp_path): dstore["/k"] = np.array([9], dtype=np.int8) # embedded overwrite assert len(dstore) == 1 assert dstore["/k"][:].tolist() == [9] + + +def test_member_window(populated_dict_store): + """Where a leaf's frame lies in a .b2z, for a reader that can take a window. + + A zip store writes each external leaf uncompressed, so those bytes are the + frame the leaf would have been written as on its own -- which is what lets a + server hand a byte-range reader the window instead of rebuilding the leaf. + """ + dstore, path = populated_dict_store + dstore.close() + reopened = blosc2.open(path) + + if path.endswith(".b2d"): # a directory store keeps each leaf in a file + assert all(reopened.member_window(key) is None for key in reopened) + return + + raw = pathlib.Path(path).read_bytes() + windows = {key: reopened.member_window(key) for key in reopened} + assert any(window is not None for window in windows.values()) + for key, window in windows.items(): + if window is None: + # An embedded leaf lives inside the store's own compressed + # super-chunk, so it has no window in the file to point at + assert key in reopened.estore + continue + offset, nbytes = window + frame = raw[offset : offset + nbytes] + assert frame[2:9] == b"b2frame" # a whole frame, starting where it says + assert np.array_equal(blosc2.from_cframe(frame)[:], reopened[key][:]) + # ... and nothing is claimed for what has no window of its own + assert reopened.member_window("/nope") is None + + +def test_member_window_of_a_tree_group(tmp_path): + path = str(tmp_path / "grouped.b2z") + with blosc2.TreeStore(path, mode="w") as tstore: + tstore["/g/leaf"] = np.arange(10, dtype="i4") + tstore = blosc2.open(path) + assert tstore.member_window("/g/leaf") is not None + assert tstore.member_window("/g") is None # a group is not a leaf From 460b060f2b3e3ee6031026c90b4659c90c7d621d Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:20:45 +0200 Subject: [PATCH 16/32] Pin that a container leaf reads its blocks like any other array The client side of container-leaf ranges is that there is no client side: a subscriber serves a leaf out of its window in the .b2z, so what arrives is a frame beginning at 0 and everything already written for a .b2nd applies. The stand-in subscriber can now be pointed at a leaf, which is the same six lines: it serves the window instead of the file, exactly as Caterva2 does. The tests then assert what matters -- a Proxy over a leaf fetches blocks rather than chunks, and the leaf is stamped with the container's mtime, since a leaf has none of its own. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 62 ++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 9 deletions(-) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 9376144d3..1d9b3c1bf 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -32,8 +32,9 @@ class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" - def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges=True): + def __init__(self, path, key=None, ranges=True, cookie=None, multipart=True, merge_ranges=True): self.path = str(path) + self.key = key # a leaf inside a .b2z container, rather than a file of its own self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does self.multipart = multipart # False: answer only the first range asked for @@ -42,10 +43,20 @@ def __init__(self, path, ranges=True, cookie=None, multipart=True, merge_ranges= self.reload() def reload(self): - """Pick up the file as it is now, as a subscriber would on the next request.""" - self.frame = pathlib.Path(self.path).read_bytes() - self.array = blosc2.open(self.path) + """Pick up the file as it is now, as a subscriber would on the next request. + + A leaf is served out of its window in the container, which is what makes + it look to a client exactly like a dataset of its own: byte 0 of what it + asks for is the frame's first byte. + """ + raw = pathlib.Path(self.path).read_bytes() self.mtime = pathlib.Path(self.path).stat().st_mtime + if self.key is None: + self.frame, self.array = raw, blosc2.open(self.path) + return + store = blosc2.open(self.path) + offset, nbytes = store.member_window(self.key) + self.frame, self.array = raw[offset : offset + nbytes], store[self.key] @property def meta(self): @@ -152,15 +163,25 @@ def _fetch(self, sub): ) -def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", **kwargs): - """A C2Array over *data*, served by a subscriber stand-in on localhost.""" +def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", key=None, **kwargs): + """A C2Array over *data*, served by a subscriber stand-in on localhost. + + With *key*, the array is a leaf of a TreeStore container instead of a file + of its own, and the subscriber serves it from its window -- which is what + Caterva2 does, and what the client is meant not to notice. + """ urlpath = str(tmp_path / name) - blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + if key is None: + blosc2.asarray(data, chunks=chunks, blocks=blocks, urlpath=urlpath, mode="w") + else: + with blosc2.TreeStore(urlpath, mode="w") as tstore: + tstore[key] = blosc2.asarray(data, chunks=chunks, blocks=blocks) server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) - server.subscriber = _Subscriber(urlpath, **kwargs) + server.subscriber = _Subscriber(urlpath, key=key, **kwargs) threading.Thread(target=server.serve_forever, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" - array = blosc2.C2Array(f"@public/{name}", urlbase=urlbase, auth_token=kwargs.get("cookie")) + path = f"@public/{name}{key or ''}" + array = blosc2.C2Array(path, urlbase=urlbase, auth_token=kwargs.get("cookie")) return array, server.subscriber, server @@ -503,3 +524,26 @@ def test_a_read_only_cache_is_not_stamped(tmp_path, subscriber): reopened = blosc2.open(cache, mode="r") assert np.array_equal(reopened[0:5, 0:10], data[0:5, 0:10]) + + +def test_blocks_of_a_container_leaf(subscriber, any_chunk_wants_blocks): + """A leaf of a .b2z is a whole frame inside the container, and a subscriber + serves it from that window -- so the client reads its blocks knowing nothing + about containers, which is the whole of what it takes.""" + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="tree.b2z", key="/g/leaf") + p = blosc2.Proxy(array, mode="w") + assert array.block_source() is not None + sub.log.clear() + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert not _bytes(sub, "chunk") + assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 8 + assert np.array_equal(p[...], data) + + +def test_a_container_leaf_is_stamped_like_any_other(subscriber): + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20), name="tree.b2z", key="/g/leaf") + # A leaf has no mtime of its own: the container's is what says it changed + assert array.stamp == f"{sub.mtime}:{sub.array.schunk.cbytes}" From 830f3704cba54a6d57f48a4a8986c4aefddd939c Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:24:10 +0200 Subject: [PATCH 17/32] Record what container-leaf ranges landed as Including the one deviation from the plan: the whole-leaf fetch is served from the window too, because the rebuild it replaced re-partitioned the array and so disagreed with the chunks api/info reports for the same leaf. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 9 +++++ plans/container-leaf-ranges.md | 74 ++++++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 5bd236902..13fa56cf0 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -64,6 +64,15 @@ XXX version-specific blurb XXX `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. +* `DictStore.member_window(key)` says where a leaf's frame lies inside a `.b2z`, + as `(offset, nbytes)`. A zip store keeps each external leaf uncompressed, so + those bytes are the frame that leaf would have been written as on its own -- + which lets a reader take the window instead of the leaf: Caterva2 now serves a + container leaf from it, so a `Proxy` over `@public/tree.b2z/leaf` reads blocks + exactly as it does over a `.b2nd` (2.8x on a point read, 20x fewer bytes). + None where there is no window: a directory-backed store, an embedded leaf, a + `C2Array` reference. + * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can outlive the process. The cache must come from a proxy over a source of the same diff --git a/plans/container-leaf-ranges.md b/plans/container-leaf-ranges.md index 8e1ab4d16..29f942f71 100644 --- a/plans/container-leaf-ranges.md +++ b/plans/container-leaf-ranges.md @@ -169,3 +169,77 @@ served` and time the three modes as it does for a plain array. them today (`method=0` throughout), but a `.b2z` produced by another tool could, and its window would decode to nonsense. The member's own frame magic is what catches that, on the server, before the window is offered. + +## What landed + +All four phases, 2026-08-18. blosc2 on `cat2-block-granularity`, Caterva2 on +`range-honesty`. + +| phase | where | commit | +|---|---|---| +| 1. `member_window` | blosc2 `dict_store.py` | *Say where a leaf's frame lies inside a .b2z* | +| 2. leaf ranges | caterva2 `server.py`, `srv_utils.py` | *Serve a container leaf from its window in the file* | +| 3. container fetched whole | caterva2 `server.py` | (same commit) | +| 4. tests | both | (same commits, plus *Pin that a container leaf reads its blocks like any other array*) | + +**The client did not change, at all.** That was the design's claim and it held: +a `Proxy` over `@public/leaves.b2z/big` probes with a `Range` as it does for any +dataset, gets a 206, reads the frame index, and fetches blocks. Nothing in +blosc2 knows that the frame it is reading lives inside a container. + +### One deviation, for a reason that only turned up in the measuring + +The plan said not to serve a *whole* member from its window, on the grounds that +it would change the bytes clients get. It changes them for the better, which the +first measurement showed: the rebuild it replaced went through +`array.slice(..., contiguous=True).to_cframe()`, which **re-partitions**. A leaf +stored with `chunks=(1, 1000, 500)` came back with `chunks=(4, 1000, 500)` while +`api/info` went on reporting the stored ones -- so the two disagreed about the +same leaf, and a client caching that got a partitioning the source never had. +Serving the window makes them agree, because they are the same bytes. + +That also settled what `Accept-Ranges` may say on that response: `bytes`, since +the ranged and whole views are now one representation. Serving two different +byte streams for one URL and advertising ranges over them would have been the +kind of thing `If-Range` exists to catch. + +### The numbers + +A leaf of 8 chunks (3.36 MB each, 20 blocks per chunk) against a local server, +with a network put in front of every request (`--latency-ms 45 +--bandwidth-mbs 10`, cat2.cloud's shape), median of 3: + +| pattern | before (chunks only) | after, blocks | after, multipart | +|---|---|---|---| +| point | 1 req, 3.36 MB, 0.410 s | 2 req, 0.17 MB, 0.145 s | 2 req, 0.17 MB, 0.151 s | +| line, last dim | 1 req, 3.36 MB, 0.408 s | 2 req, 0.17 MB, 0.140 s | 2 req, 0.17 MB, 0.142 s | +| line, first dim | 8 req, 26.92 MB, 0.454 s | 16 req, 1.35 MB, 0.184 s | 2 req, 1.35 MB, 0.271 s | +| window (1/64 per dim) | 1 req, 3.36 MB, 0.413 s | 2 req, 0.17 MB, 0.152 s | 2 req, 0.17 MB, 0.147 s | +| slab (1% of dim 0) | 1 req, 3.36 MB, 0.414 s | 1 req, 3.36 MB, 0.417 s | 1 req, 3.36 MB, 0.415 s | +| slab (10% of dim 0) | 1 req, 3.36 MB, 0.416 s | 1 req, 3.36 MB, 0.414 s | 1 req, 3.36 MB, 0.412 s | + +2.8x on a point or a window, 20x fewer bytes, and the slabs that want every +block of their chunk unchanged -- the same shape a plain `.b2nd` has, which is +the point. Run against the same data stored both ways, the two tables agree row +by row to within the noise. + +The whole-leaf fetch, which now reads the window instead of rebuilding it: + +| | time | bytes | chunks it returns | +|---|---|---|---| +| before | 0.034 s | 26.94 MB | (4, 1000, 500), where `api/info` said (1, 1000, 500) | +| after | 0.016 s | 26.92 MB | (1, 1000, 500), as `api/info` says | + +### Left undone + +- **Only `.b2z` leaves.** An HDF5 dataset is not a Blosc2 frame and a `.b2d` + store keeps its leaves in files of their own (which could be served directly, + and are not today: `api/fetch` on a `.b2d` member is untested ground). +- **An embedded leaf stays whole-chunked.** Its bytes live inside the store's own + compressed super-chunk, so there is no window; `member_window` says so and the + client falls back, which is the arrangement working as intended rather than a + gap to close. +- **`If-Range` / `ETag`.** A leaf rewritten between a client's frame-index read + and its block reads would be read at offsets that no longer mean anything. The + proxy's stamp catches that between *runs* (the container's mtime), not within + one, which is the same exposure a plain `.b2nd` has over any object store. From acf2d24af597a87b4253f70e282450bd80307854 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 08:24:28 +0200 Subject: [PATCH 18/32] Cross off container-leaf ranges in the block-granularity plan Co-Authored-By: Claude Opus 5 --- plans/cat2-block-granularity.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 13d62df53..e9d8a3214 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -440,12 +440,12 @@ batched again. ### Left undone -- **A container leaf could serve ranges too.** A TreeStore keeps its leaves as - ordinary frames inside the `.b2z`, so the bytes a block reader wants are in - the file at a fixed offset -- what is missing is a way for the server to say - where a leaf's frame starts, and for the client to add that base to every - range. Worth its own plan; it would give `.b2z` members everything a plain - `.b2nd` has. +- ~~**A container leaf could serve ranges too.**~~ It does now, and it took no + client change at all: see [plans/container-leaf-ranges.md](container-leaf-ranges.md). + The subscriber serves a leaf out of its window in the `.b2z`, so what arrives + is a frame beginning at 0 and everything written for a `.b2nd` applies. 2.8x + on a point read of a leaf, 20x fewer bytes, and the same table as a plain + array row for row. - **Opening a frame could be one request rather than two**, with a suffix range (`Range: bytes=-65536`) batched with the head read. No fsspec backend exposes suffix ranges and `read_ranges` has no way to express one, so it would be a From 438d2fc1b3449e3236a2afec9aba1904dc649edd Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Tue, 18 Aug 2026 19:24:51 +0200 Subject: [PATCH 19/32] Shim to reproduce itertools.batched for python 3.11 --- src/blosc2/proxy.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index ce75dd869..06ff731e2 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -17,6 +17,18 @@ from collections.abc import Sequence from concurrent.futures import ThreadPoolExecutor +try: + from itertools import batched +except ImportError: + # Python 3.11 has no itertools.batched + from itertools import islice + + def batched(iterable, n): + it = iter(iterable) + while batch := tuple(islice(it, n)): + yield batch + + try: from numpy.typing import DTypeLike except (ImportError, AttributeError): @@ -720,7 +732,7 @@ def _fetch_by_block(self, item, max_concurrency: int | None): # a batch of range reads (of one, for a transport that takes one) runs = [(n, run) for n in wanted for run in self.src.block_plan(n, wanted[n])] batch = max(getattr(self.src, "max_ranges", 1), 1) - tasks = [((n, None),) for n in whole] + list(itertools.batched(runs, batch)) + tasks = [((n, None),) for n in whole] + list(batched(runs, batch)) # `read_ranges` is the optional half of the protocol: a source that only # has `read_range` is asked one range at a time, as `batch` is 1 for it @@ -1380,7 +1392,7 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: """ section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] - for batch in itertools.batched(todo, max(self.max_ranges, 1)): + for batch in batched(todo, max(self.max_ranges, 1)): spans = [(int(self._offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): From 3c24a11e492d70ac22b2089b37bb7cab4ee2cac0 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 07:43:37 +0200 Subject: [PATCH 20/32] Skip the block tests on wasm, where a socket cannot listen The stand-in subscriber binds a real ThreadingHTTPServer, and Pyodide serves listen(2) out of node's `ws` module -- absent from the cibuildwheel xbuildenv, so the call takes the whole interpreter down as a fatal error rather than raising something a test could catch. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 1d9b3c1bf..bd85aca06 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -28,6 +28,11 @@ import blosc2 +# The stand-in subscriber binds a real socket, and Pyodide has no listen(2): +# node asks for the `ws` module that is not there, and takes the runtime down +# with it rather than raising +pytestmark = pytest.mark.skipif(blosc2.IS_WASM, reason="no listening sockets on wasm32") + class _Subscriber: """A Caterva2-shaped server over one .b2nd file.""" From aeb5a1b50641758666f7ed7aa92cb2c6888e3e82 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:23:47 +0200 Subject: [PATCH 21/32] Read a frame's chunk offsets when a chunk is first asked for Opening a frame read the header and the chunk offsets together, which is two requests for something an open does not always need: the header is what says a frame can be read this way at all -- the magic, and the b2nd metalayer whose geometry the source is built from -- while the offsets are only ever wanted once something asks where a chunk is. A Proxy over a cache that already holds the slice wanted asks that of nothing, and paid a round trip for the answer anyway. _read_frame_index splits into _read_frame_header and _read_frame_offsets, and ByteRangeNDSource calls the second on the first chunk touched. _offsets and _extents become properties over a memoized index, so every call site is as it was. Under a lock, since the fetches they serve run in a thread pool: two threads reading the same index is a wasted request rather than a wrong answer, but it is still a wasted request, and one per thread of the first wave. Measured on memory://, requests then bytes, before -> after: opening a 300 KB frame 2/8275 -> 1/8192; blosc2.open(url, lazy=True) the same; a later run over a cache_storage= that holds the slice 2/8275 -> 1/8192. A frame small enough to arrive whole in the first read was already one request and still is. Nothing changes once a chunk is actually fetched -- a corner slice is 4 requests and the same bytes either way -- since there the offsets are read regardless, only later. C2Array is unaffected: block_source() already put the whole open off until a fetch wanted a chunk, so a cache covering the slice cost nothing there before this and costs nothing after. Measured both ways to be sure. The test added for it therefore guards behaviour that already existed, and says so. Three fsspec tests that install their traffic counter after the open now see one read more, because the offsets read moved into the window they count; the wave test sees three waves rather than two, for the same reason. What they were written to prove -- that a wave is batched, not that it is two requests -- is unchanged: four chunks still cost three requests, not three per chunk. Also: the release note claimed a lazy open costs "one request", which is one *read*. FsspecNDSource asks the filesystem for the object's metadata too, which is where its stamp comes from, so its floor is that call plus the header read. C2Array's floor is the single api/info that carries geometry and stamp together. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 10 ++- plans/cat2-block-granularity.md | 10 ++- src/blosc2/proxy.py | 100 ++++++++++++++++++++------- tests/ndarray/test_c2array_blocks.py | 31 ++++++++- tests/test_fsspec.py | 70 ++++++++++++++----- 5 files changed, 176 insertions(+), 45 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 13fa56cf0..2f2d12942 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -60,7 +60,15 @@ XXX version-specific blurb XXX of four (0.237 s → 0.138 s against cat2.cloud), and one for a frame small enough to arrive whole in the first read: the two reads that only measured the next one are guessed at generously instead, since over a network a few hundred - bytes and a few kilobytes cost the same. + bytes and a few kilobytes cost the same. Of those two, only the header is read + when the frame is opened — it is what says the frame can be read this way at + all — and where the chunks are waits for the first chunk anything asks about. + So `blosc2.open(url, lazy=True)` reads once, and so does a whole run over a + `cache_storage=` that already holds the slice wanted: it fetches nothing, and + now asks for no index either. `FsspecNDSource` also asks the filesystem for + the object's metadata, which is where its `stamp` comes from, so its floor is + that call plus the header read; `C2Array` gets geometry and stamp together + from `api/info`, and its floor is that one request. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index e9d8a3214..08a4ece3b 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -358,7 +358,7 @@ so every existing `Proxy(C2Array(...))` gets blocks without being asked. |---|---|---| | a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | | chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | -| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* | | a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | - **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question @@ -405,6 +405,14 @@ which is the threshold declining to take a chunk apart. Around it: |---|---|---| | one request (`api/info`) | 163.3 ms, a client per request | **42.3 ms**, pooled (3.9x) | | opening a `C2Array` | 0.237 s, 4 requests, 303 bytes | **0.138 s**, 2 requests, 8306 bytes | + +Of those two requests only the first is an open: the header says whether the +frame can be read this way, and the offsets wait for the first chunk anything +asks about. That is worth a request on every `blosc2.open(url, lazy=True)`, and +a whole run's traffic where a `cache_storage=` already holds the slice wanted. +For a `C2Array` it moves a request rather than saving one -- `block_source()` +already put the whole open off until a fetch wanted a chunk, so a cache that +covers the slice cost nothing there before this and costs nothing after. | a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | Phase 4 was built because that last row said to. Starlette *sorts and merges* diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 06ff731e2..706c2032b 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -13,6 +13,7 @@ import os import struct import textwrap +import threading from abc import ABC, abstractmethod from collections.abc import Sequence from concurrent.futures import ThreadPoolExecutor @@ -1127,23 +1128,24 @@ def _chunk_payloads(chunk: bytes, nblocks: int, wanted) -> dict[int, bytes]: return {int(n): chunk[bstarts[n] : bstarts[n] + extents[n]] for n in wanted} -def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: - """Read the header and the chunk offsets of a contiguous frame. +def _read_frame_header(read_range) -> tuple[bytes, list, bytes]: + """Read the header of a contiguous frame. *read_range* is ``(offset, size) -> bytes``, so what this costs is round trips. Returns the raw header bytes, the header decoded as the msgpack - array it is, and the absolute position of every chunk. A negative position - is not a position at all: it encodes a run-length chunk that was never - written to the file. See ``README_CFRAME_FORMAT.rst`` in c-blosc2 for the - layout. - - The format asks to be read in four steps -- how long is the header, the - header, how long is the offsets chunk, the offsets chunk -- and each of the - two questions costs as much as the answer it asks for. So both are guessed - at instead: enough of the head to hold any ordinary header, and the tail - that the frame's own length bounds. A guess that falls short is followed by - the exact read that would have happened anyway, and a frame small enough to - arrive whole in the first read costs one. + array it is, and the prefetched head those came out of -- which + `_read_frame_offsets` needs, since a frame small enough to have arrived + whole carries its offsets in there too. See ``README_CFRAME_FORMAT.rst`` + in c-blosc2 for the layout. + + The format asks how long the header is before handing it over, and the + question costs as much as the answer. So it is guessed at instead: enough + of the head to hold any ordinary header, and a guess that falls short is + followed by the exact read that would have happened anyway. + + This is everything that says whether a frame can be read this way at all -- + the magic, and the metalayer a caller goes on to decode -- so it is what an + open must do eagerly, and the offsets can wait for the first chunk touched. """ import msgpack @@ -1158,11 +1160,24 @@ def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up # that byte is not valid UTF-8 and decoding the header blows up header = msgpack.unpackb(raw, raw=True, strict_map_key=False) + return raw, header, head + +def _read_frame_offsets(read_range, header: list, head: bytes, header_len: int) -> np.ndarray: + """The absolute position of every chunk of a frame whose header is in hand. + + A negative position is not a position at all: it encodes a run-length chunk + that was never written to the file. + + Like the header before it, the offsets chunk announces its length in an + answer that costs as much as the chunk does, so the tail the frame's own + length bounds is read instead and the announcement checked against it. A + frame that arrived whole in *head* is already past this and costs nothing. + """ # An empty frame has no chunks, so it has no offsets chunk either: what sits # at index_pos is the trailer, and reading it as one fails obscurely if header[8] == 0: # chunksize - return raw, header, np.empty(0, dtype=np.int64) + return np.empty(0, dtype=np.int64) # The offsets live in a Blosc2 chunk of their own, right after the data ones, # so all that follows them is the trailer: the frame's own length says how @@ -1179,7 +1194,7 @@ def _read_frame_index(read_range) -> tuple[bytes, list, np.ndarray]: index = read_range(index_pos, index_cbytes) offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) # Offsets are relative to the end of the header - return raw, header, np.where(offsets >= 0, offsets + header_len, offsets) + return np.where(offsets >= 0, offsets + header_len, offsets) def _frame_metalayer(raw: bytes, header: list, name: str): @@ -1227,9 +1242,13 @@ class ByteRangeNDSource(ProxyNDSource): over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. A subclass sets its transport up first and then calls this constructor, - which reads the frame's header and chunk offsets through it (three small - reads). It may also set a ``stamp``, anything that names the exact bytes it - reads, so that :ref:`Proxy` can tell a cache built from other bytes. + which reads the frame's header through it -- one small read, and everything + an open decides: that this is a frame at all, and one holding an NDArray. + Where the chunks are waits for the first one anything asks about, so a + :ref:`Proxy` over a cache that already holds the slice wanted opens the + source without ever fetching its index. It may also set a ``stamp``, + anything that names the exact bytes it reads, so that :ref:`Proxy` can tell + a cache built from other bytes. Contiguous frames carrying a ``b2nd`` metalayer only, which is what :func:`blosc2.asarray` and friends write to a single file. Sparse frames and @@ -1264,12 +1283,20 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self.urlpath = urlpath # Exact ranges, not a file handle: a buffered one reads a whole block per # seek (50 MiB on s3fs by default), which would undo the point of a lazy - # open. Chunk reads are stateless, so nothing here is shared between threads - raw, header, self._offsets = _read_frame_index(self.read_range) - self._chunksize = header[8] - self._extents = _chunk_extents(self._offsets, header) + # open. Chunk reads are stateless, so the index below is the only state a + # thread pool shares, and the only thing here that needs a lock + raw, self._header, self._head = _read_frame_header(self.read_range) + self._header_len = len(raw) + self._chunksize = self._header[8] + # Where the chunks are is read on the first one touched, not here: a + # `Proxy` over a cache that already holds the slice asked for fetches + # nothing, and then the offsets are a request spent on nothing at all. + # Everything an open has to decide -- that this is a frame, and one with + # a b2nd metalayer -- is in the header that was just read. + self._index = None + self._index_lock = threading.Lock() try: - _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, header, "b2nd") + _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, self._header, "b2nd") except KeyError: raise NotImplementedError( f"{urlpath} has no b2nd metalayer, so it is a plain SChunk rather than an " @@ -1299,6 +1326,31 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): # many short-lived processes ever share one cached array. self._layouts = {} + def _frame_index(self) -> tuple[np.ndarray, np.ndarray]: + """Where every chunk of the frame is, and how much to read at each. + + Read once, on the first chunk anything asks about. Under a lock because + the fetches this serves run in a thread pool: without one the first wave + of them would each read the index, which is a wasted request apiece and + no worse -- what they read is the same either way. + """ + with self._index_lock: + if self._index is None: + offsets = _read_frame_offsets(self.read_range, self._header, self._head, self._header_len) + self._index = (offsets, _chunk_extents(offsets, self._header)) + self._head = None # the prefetch has nothing left to answer + return self._index + + @property + def _offsets(self) -> np.ndarray: + """Where each chunk begins, negative for one that lives in its offset.""" + return self._frame_index()[0] + + @property + def _extents(self) -> np.ndarray: + """How many bytes to read at each chunk's offset to be sure of covering it.""" + return self._frame_index()[1] + @property def shape(self) -> tuple: return self._shape diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index bd85aca06..4467c48db 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -351,6 +351,30 @@ def test_blocks_survive_a_reopened_cache(tmp_path, subscriber, any_chunk_wants_b assert np.array_equal(p[...], data) +def test_a_cache_that_holds_the_slice_costs_no_request(tmp_path, subscriber, any_chunk_wants_blocks): + # Re-running a script over a cache that already covers the slice: `api/info` + # is all it takes. Nothing opens the frame, because opening it is what + # `block_source` puts off until a fetch actually wants a chunk -- and this + # fetch wants none. + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "held.b2nd") + item = (slice(0, 5), slice(0, 10)) + blosc2.Proxy(array, urlpath=cache, mode="a").fetch(item) + + # A later run: its own array over the same subscriber, its own source + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + p.fetch(item) + assert np.array_equal(p[item], data[item]) + assert not sub.log + + # ... and a slice the cache does not hold opens the frame then + assert np.array_equal(p[100:105, 0:10], data[100:105, 0:10]) + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 # head, offsets, layout, block + + def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): # A cache left by a run that fetched whole chunks (which is every run before # this existed) holds complete chunks, so nothing in it is fetched again @@ -395,12 +419,13 @@ def test_a_whole_wave_travels_in_one_request(subscriber, any_chunk_wants_blocks) data = _incompressible((400, 200)) array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) p = blosc2.Proxy(array, mode="w") - assert array.block_source() is not None # the frame index, read once per array + assert array.block_source() is not None # the header, read once per array sub.log.clear() assert np.array_equal(p[:, 0:10], data[:, 0:10]) - # One request for the layouts of all four chunks, one for all their blocks - assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] + # One request for where the four chunks are, one for the layouts of all of + # them, one for all their blocks -- three waves, not three per chunk + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch", "fetch"] assert {status for _, status, _ in sub.log} == {206} assert _bytes(sub, "fetch") < sub.array.schunk.cbytes / 4 diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index 3d4f2b625..fdcf19295 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -759,23 +759,29 @@ def _incompressible(shape, chunks, blocks, seed=0): return data, blosc2.asarray(data, chunks=chunks, blocks=blocks) -def test_lazy_open_costs_two_reads(monkeypatch): - # The format asks four questions to find the chunk offsets; both of the two - # that only measure the next read are guessed at instead +def test_lazy_open_costs_one_read(monkeypatch): + # The format asks how long the header is in an answer as dear as the header, + # so the head is guessed at generously instead -- and that one read is the + # whole of an open, since where the chunks are is nothing an open decides data, a = _incompressible((600, 600), (300, 600), (30, 600)) url = _put("openreads.b2nd", a) reads, chunks = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 2 + assert len(reads) == 1 assert not chunks assert src.shape == (600, 600) - assert len(src._offsets) == 2 # ... and it read the offsets it came for + + # ... and the offsets are read by the first thing that asks where a chunk is + assert len(src._offsets) == 2 + assert len(reads) == 2 + assert len(src._offsets) == 2 # read once and kept, not once per question + assert len(reads) == 2 -def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): +def test_lazy_open_of_a_small_frame_never_reads_twice(monkeypatch): # A frame that fits in the first read is wholly in hand: the offsets chunk - # is in those bytes too, so there is nothing left to ask for + # is in those bytes too, so there is nothing left to ask for, then or later a = blosc2.arange(0, 100, dtype="i4", chunks=(10,)) assert a.schunk.cbytes < blosc2.proxy._FRAME_PREFETCH url = _put("smallframe.b2nd", a) @@ -784,6 +790,7 @@ def test_lazy_open_of_a_small_frame_costs_one_read(monkeypatch): src = blosc2.FsspecNDSource(url) assert len(reads) == 1 assert len(src._offsets) == 10 + assert len(reads) == 1 assert np.array_equal(blosc2.Proxy(src)[:], a[:]) @@ -797,9 +804,34 @@ def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): reads, _ = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 3 # the guess, the header, the offsets + assert len(reads) == 2 # the guess, then the header assert reads[1] > 4096 assert np.array_equal(blosc2.Proxy(src)[:], data) + assert len(reads) > 2 # the offsets, and the chunks themselves + + +def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path): + # What the deferral is for: a later run over a cache that already covers the + # slice fetches nothing, and so has no use for where the chunks are either + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("cachedslice.b2nd", a) + cache = str(tmp_path / "cachedslice-cache.b2nd") + item = (slice(0, 30), slice(0, 600)) + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch(item) + + reads, _ = _traffic(monkeypatch) + src = blosc2.FsspecNDSource(url) + proxy = blosc2.Proxy(src, urlpath=cache, mode="a") + assert len(reads) == 1 # the header, which is what says the frame is readable + + proxy.fetch(item) + assert len(reads) == 1 + assert np.array_equal(proxy[0:30, 0:600], data[0:30, 0:600]) + assert len(reads) == 1 + + # A slice it does not hold pays for the offsets then, and reads right + assert np.array_equal(proxy[300:330, 0:600], data[300:330, 0:600]) + assert len(reads) > 1 def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): @@ -811,7 +843,9 @@ def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): reads, _ = _traffic(monkeypatch) src = blosc2.FsspecNDSource(url) - assert len(reads) == 3 # the head, the capped tail, the offsets in full + assert len(reads) == 1 # the head, and the offsets not asked for yet + assert len(src._offsets) == 20 + assert len(reads) == 3 # ... then the capped tail, and the offsets in full assert reads[1] == 16 assert np.array_equal(blosc2.Proxy(src)[:], data) @@ -823,10 +857,11 @@ def test_lazy_fetches_only_touched_blocks(monkeypatch): assert cbytes > blosc2.proxy.BLOCK_MIN_CBYTES p = blosc2.open(_put("blocks.b2nd", a), lazy=True) - reads, chunks = _traffic(monkeypatch) # after the open, which reads the frame index + reads, chunks = _traffic(monkeypatch) # after the open, which reads the header assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - # One read for the block offsets, one for the single block the slice lands in - assert len(reads) == 2 + # One read for where the chunks are, one for the block offsets inside the one + # wanted, one for the single block the slice lands in + assert len(reads) == 3 assert not chunks assert sum(reads) < cbytes / 5 @@ -851,7 +886,9 @@ def test_lazy_whole_array_skips_the_block_path(monkeypatch, any_chunk_wants_bloc reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[:], data) assert len(chunks) == 2 - assert len(reads) == len(chunks) # one request each, none of them for block offsets + # One request per chunk and none for block offsets, after the one that says + # where the chunks are + assert len(reads) == len(chunks) + 1 @pytest.mark.parametrize( @@ -902,7 +939,7 @@ def test_lazy_block_cache_survives_reopen(tmp_path, monkeypatch, any_chunk_wants # A partly filled chunk survives, so the blocks in it do not travel again p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") - reopened = len(reads) # every open reads the frame index afresh + reopened = len(reads) # every open reads the header afresh assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) assert len(reads) == reopened # ... and the ones missing from it still do @@ -931,8 +968,9 @@ def test_lazy_blocks_fall_back_for_memcpyed_chunks(monkeypatch, any_chunk_wants_ reads, chunks = _traffic(monkeypatch) assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) assert len(chunks) == 1 - # The offsets are read, say there is nothing to skip, and the chunk follows - assert len(reads) == 2 + # Where the chunks are, then the block offsets, which say there is nothing to + # skip, and the chunk follows + assert len(reads) == 3 def test_lazy_blocks_with_run_length_chunks(monkeypatch, any_chunk_wants_blocks): From 993718883d013c5e54aefae27964fa236e4baa2b Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:38:57 +0200 Subject: [PATCH 22/32] Keep where the chunks and blocks are in the cache A run learned two things about where bytes lie in a remote frame and threw both away when it ended: the chunk offsets, and the block offsets of the chunks it took apart. The next run over the same cache read them again -- the offsets to fetch anything at all, a layout for every chunk it wanted blocks of that a previous run had only half filled. Both now go in the cache's vlmeta, as the bytes they were read as, so what comes back goes through `_parse_layout` exactly as a fresh read does rather than through a second parser that could drift from it. Layouts are kept for the partly filled chunks alone: a fetch asks for a layout only where blocks are missing, so a complete chunk is never asked about again and an untouched one never was, which bounds the blob to what a later fetch could actually use. Only for a source that can name the bytes it read, and only against a cache whose recorded stamp matches. These are positions *in a file*: a frame replaced underneath keeps its geometry while every chunk and every block inside it moves, and blocks spliced at the previous frame's positions decode to nonsense rather than to stale data. The stamp is checked here rather than taken on trust from how the cache was come by, since a `_cache=` handed in never passed `_reopen_cache`. An index that does not fit the source as it stands -- wrong blocks per chunk, wrong number of offsets -- is dropped rather than used. Measured on a 4-chunk frame of 1.36 MB chunks, 100 blocks each, a later run fetching different blocks of chunks a previous run half filled: 4 requests to 2 against the subscriber stand-in (the header and the blocks, nothing between), 46 to 40 against fsspec, where a layout is a request per chunk rather than one for all of them. On a 200-chunk array `_save_fetched` takes 0.53 ms with the index in it, against a round trip of tens of milliseconds saved. `C2Array` takes the index but does not hand it on until there is a source to hand it to: passing it through `block_source()` would build the source to receive it, and building one reads the frame's header -- a request, at the very moment of a run that may go on to fetch nothing at all. A cache with no `proxy-index` entry reads as it did, so nothing already on disk needs rebuilding; `proxy-index` joins the vlmeta keys a caller may not set. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 10 ++- plans/cat2-block-granularity.md | 11 +++- src/blosc2/c2array.py | 25 ++++++- src/blosc2/proxy.py | 99 ++++++++++++++++++++++++++-- tests/ndarray/test_c2array_blocks.py | 42 +++++++++++- tests/test_fsspec.py | 70 ++++++++++++++++++++ 6 files changed, 245 insertions(+), 12 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 2f2d12942..573cdb9b9 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -68,7 +68,15 @@ XXX version-specific blurb XXX now asks for no index either. `FsspecNDSource` also asks the filesystem for the object's metadata, which is where its `stamp` comes from, so its floor is that call plus the header read; `C2Array` gets geometry and stamp together - from `api/info`, and its floor is that one request. + from `api/info`, and its floor is that one request. A persisted cache keeps + what the source read about *where* things are — the frame's chunk offsets, and + the block offsets of the chunks it holds only part of — so a later run over it + starts from those instead of reading them again. A warm fetch of blocks missing + from a chunk already half held goes from 4 requests to 2 against a subscriber, + and drops the offsets read and one layout read per chunk touched against an + object store. Only for a source that can name the bytes it read: positions in a + frame are worth nothing against a frame that was replaced, so an unstamped + source keeps none of this and reads as before. `bench/ndarray/cat2-block-granularity.py` measures all of it on any dataset, against a real subscriber or a stand-in it starts itself. diff --git a/plans/cat2-block-granularity.md b/plans/cat2-block-granularity.md index 08a4ece3b..2c6db7d65 100644 --- a/plans/cat2-block-granularity.md +++ b/plans/cat2-block-granularity.md @@ -358,7 +358,7 @@ so every existing `Proxy(C2Array(...))` gets blocks without being asked. |---|---|---| | a bench for all of it | blosc2 `bench/ndarray/` | *Measure the Caterva2 block path the way the fsspec one is measured* | | chunks of a container leaf | caterva2 `server.py` | *Serve chunks of a container leaf, and refuse the ones that are not stored* | -| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* | +| a two-request frame open | blosc2 `proxy.py` | *Open a frame in two requests instead of four*, then *Read a frame's chunk offsets when a chunk is first asked for* and *Keep where the chunks and blocks are in the cache* | | a stamp for `C2Array` | blosc2 `c2array.py` | *Notice a remote array that was replaced under a proxy's cache* | - **`bench/ndarray/cat2-block-granularity.py`** is the fsspec bench's question @@ -413,6 +413,15 @@ a whole run's traffic where a `cache_storage=` already holds the slice wanted. For a `C2Array` it moves a request rather than saving one -- `block_source()` already put the whole open off until a fetch wanted a chunk, so a cache that covers the slice cost nothing there before this and costs nothing after. + +The cache then keeps both indexes it read -- the frame's chunk offsets, and the +block offsets of the chunks it holds only part of -- under the same stamp that +guards the cached chunks themselves. A warm fetch of blocks missing from a +half-held chunk: 4 requests to 2 against the subscriber stand-in (header and +blocks, nothing between), 46 to 40 against fsspec, where the layout wave is a +request per chunk rather than one for all of them. Layouts are kept for the +partly filled chunks alone, which bounds the blob to what a later fetch could +ask about: a complete chunk is never asked again, an untouched one never was. | a wave of 32 ranges | 1.530 s one at a time, 0.208 s eight at a time | **0.136 s**, one multipart request | Phase 4 was built because that last row said to. Starlette *sorts and merges* diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 71a76777d..972607db5 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -452,6 +452,8 @@ def __init__(self, path: str, /, urlbase: str | None = None, auth_token: str | N # dataset cannot be read in ranges) or a C2NDSource self._block_source = _UNTRIED self._block_lock = threading.Lock() + # An index a `Proxy` handed over before the source existed; see adopt_index + self._pending_index = None # Try to 'open' the remote path try: @@ -718,12 +720,33 @@ def _open_block_source(self) -> C2NDSource | None: # something only the answer to a range request can say: an HDF5 leaf or a # `.b2z` member reports one and is streamed all the same try: - return C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) + source.adopt_index(self._pending_index) + return source except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): # Not ranged, not a contiguous frame, not an NDArray, or not # reachable: whole chunks work for all of those return None + def adopt_index(self, state) -> None: + """Keep an index a `Proxy` read out of its cache until there is a source. + + Handing it straight to :meth:`block_source` would build the source to + receive it, and building it reads the frame's header -- a request, at the + very moment of a run that may go on to fetch nothing at all. So it waits + here, and `_open_block_source` passes it on to the source it builds. + """ + self._pending_index = state + + def index_state(self, keep=()) -> dict | None: + """What a `Proxy` should keep of what was read; see :ref:`ByteRangeNDSource`.""" + source = self._block_source + if source is _UNTRIED or source is None: + # No source was ever built, so nothing was read through one: hand back + # whatever came out of the cache, rather than dropping it + return self._pending_index + return source.index_state(keep) + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: """Whether fetching *nwanted* blocks of a chunk beats fetching all of it.""" source = self.block_source() diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 706c2032b..2da6ad360 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -85,7 +85,9 @@ def batched(iterable, n): # vlmeta entries the proxy keeps its own state in: what it has fetched, and which # remote bytes the cache was filled from. A caller cannot write these. -_RESERVED_VLMETA = frozenset({"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp"}) +_RESERVED_VLMETA = frozenset( + {"proxy-fetched", "proxy-fetched-blocks", "proxy-fetched-bpc", "proxy-stamp", "proxy-index"} +) # `jit` kwargs that tune *how* an expression is evaluated, not what container the # result is stored in. Unlike storage kwargs (`cparams`, `chunks`, `urlpath`, ...), @@ -415,6 +417,18 @@ def __init__( # hands over for a persisted proxy: nothing may be written there, and a # proxy over one stays observational anyway (see `__getitem__`) self._schunk_cache.vlmeta["proxy-stamp"] = stamp + # Where the chunks are, and where the blocks of the partly filled ones + # are, as an earlier run read them. Only from a cache that names the very + # same remote bytes, checked here rather than taken on trust from how the + # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. + adopt = getattr(self.src, "adopt_index", None) + if ( + adopt is not None + and not fresh + and stamp is not None + and self._schunk_cache.vlmeta.get("proxy-stamp") == stamp + ): + adopt(self._schunk_cache.vlmeta.get("proxy-index")) if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) if reserved: @@ -563,6 +577,27 @@ def _save_fetched(self) -> None: self._schunk_cache.vlmeta[self._fetched_key] = bytes(self._fetched) if self._blocks_per_chunk > 1: self._schunk_cache.vlmeta["proxy-fetched-bpc"] = self._blocks_per_chunk + # Where the source read things to be, so the next run over this cache need + # not ask again. Only for a source that can name the bytes it read: an + # unstamped one cannot tell a replaced frame from the one these positions + # came from, and reusing them across a replacement is worse than serving + # stale data. Bounded by keeping layouts for the partly filled chunks + # alone, which are the only ones a later fetch would ask about. + state = getattr(self.src, "index_state", None) + if state is not None and getattr(self.src, "stamp", None) is not None: + self._schunk_cache.vlmeta["proxy-index"] = state(self._partly_filled()) + + def _partly_filled(self) -> list[int]: + """Chunks the cache holds some of the blocks of, but not all.""" + bpc = self._blocks_per_chunk + nchunks = self._schunk_cache.nchunks + if bpc == 1 or not nchunks: + return [] + # Counted over the whole bitmap at once: a loop over chunks x blocks is + # millions of bit tests on a large array, and this runs after every fetch + bits = np.unpackbits(np.frombuffer(bytes(self._fetched), dtype=np.uint8), bitorder="little") + counts = bits[: nchunks * bpc].reshape(nchunks, bpc).sum(axis=1) + return np.flatnonzero((counts > 0) & (counts < bpc)).tolist() def _reopen_cache(self, urlpath: str): """Adopt the cache container stored at *urlpath*, checking it fits the source.""" @@ -1318,13 +1353,62 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): if all(self._blocks) else 1 ) - # ponytail: layouts are memoized for the life of the source, so a second - # slice pays no header read. Persisting them in the cache would save one - # round trip more, but only for a new process reaching into a chunk it - # had partly explored before -- every other repeat already skips the read, - # since a fetch asks for layouts only where blocks are missing. Do it if - # many short-lived processes ever share one cached array. + # Layouts are memoized for the life of the source, and `_sections` keeps + # the bytes each was read as, so a `Proxy` can hand them to its cache and + # a later run start from them instead of reading them again. self._layouts = {} + self._sections = {} + + def index_state(self, keep: Sequence[int] = ()) -> dict: + """Where things are, as the bytes they were read as, for a cache to keep. + + The frame's chunk offsets, and the header sections of the chunks in + *keep*. A layout is worth keeping only for a chunk the cache holds some + but not all of: a fetch asks for layouts only where blocks are missing, + so a chunk that is complete is never asked about again, and one that is + empty was never read. + + Kept as read rather than as parsed, so that what comes back goes through + :meth:`_parse_layout` exactly as a fresh read does -- one parser, not a + second one that could disagree with it. + """ + offsets = self._index[0] if self._index is not None else None + return { + "bpc": self.blocks_per_chunk, + "offsets": b"" if offsets is None else offsets.tobytes(), + "layouts": [[n, self._sections[n]] for n in keep if n in self._sections], + } + + def adopt_index(self, state: dict | None) -> None: + """Take up what an earlier run left behind in :meth:`index_state`. + + Only ever called with a state saved against the very same remote bytes -- + :ref:`Proxy` checks the source's ``stamp`` against the one its cache + recorded first -- and that is what makes these safe to reuse. They are + positions *in a file*: a frame replaced underneath keeps its geometry + while every chunk and every block inside it moves, and blocks spliced at + the positions of the frame before it decode to nonsense rather than to + stale data. Anything that does not fit the source as it stands now is + dropped rather than trusted. + """ + if not state or self._index is not None: + return + if state.get("bpc") != self.blocks_per_chunk: + return + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + offsets = state.get("offsets") or b"" + if offsets: + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) + array = np.frombuffer(offsets, dtype=np.int64) + if len(array) != nchunks: + return + with self._index_lock: + self._index = (array, _chunk_extents(array, self._header)) + self._head = None # the prefetch has nothing left to answer + for nchunk, head in state.get("layouts") or (): + if len(head) <= section: # what a read of one asks for, or a short answer + self._sections[nchunk] = head + self._layouts[nchunk] = self._parse_layout(head, section) def _frame_index(self) -> tuple[np.ndarray, np.ndarray]: """Where every chunk of the frame is, and how much to read at each. @@ -1448,6 +1532,7 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: spans = [(int(self._offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): + self._sections[nchunk] = head self._layouts[nchunk] = self._parse_layout(head, section) return [self._layouts[n] for n in nchunks] diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 4467c48db..31a636bf1 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -370,9 +370,47 @@ def test_a_cache_that_holds_the_slice_costs_no_request(tmp_path, subscriber, any assert np.array_equal(p[item], data[item]) assert not sub.log - # ... and a slice the cache does not hold opens the frame then + # ... and a slice the cache does not hold opens the frame then: the header, + # the layout of the chunk it lands in, and the blocks. Not where the chunks + # are -- the earlier run left that in the cache assert np.array_equal(p[100:105, 0:10], data[100:105, 0:10]) - assert [kind for kind, _, _ in sub.log] == ["fetch"] * 4 # head, offsets, layout, block + assert [kind for kind, _, _ in sub.log] == ["fetch"] * 3 + + +def test_a_kept_index_halves_a_warm_fetch(tmp_path, subscriber, any_chunk_wants_blocks): + # A later run wanting different blocks of chunks a previous one half filled: + # where the chunks are and where those blocks are both came out of the cache, + # so what travels is the header and the blocks, and nothing between + data = _incompressible((400, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "kept.b2nd") + blosc2.Proxy(array, urlpath=cache, mode="a").fetch((slice(None), slice(0, 10))) + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + assert np.array_equal(p[:, 100:110], data[:, 100:110]) + assert [kind for kind, _, _ in sub.log] == ["fetch", "fetch"] # the header, the blocks + assert np.array_equal(p[...], data) # ... and the rest still reads right + + +def test_a_kept_index_does_not_open_the_frame_to_be_taken_up(tmp_path, subscriber, any_chunk_wants_blocks): + # Handing the index to the source would build the source to receive it, and + # building it reads the header -- a request, at the very moment of a run that + # may go on to fetch nothing. It waits with the array until there is a source + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "unopened.b2nd") + item = (slice(0, 5), slice(0, 10)) + blosc2.Proxy(array, urlpath=cache, mode="a").fetch(item) + + again = blosc2.C2Array(array.path, urlbase=array.urlbase) + sub.log.clear() + p = blosc2.Proxy(again, urlpath=cache, mode="a") + assert again._pending_index is not None # taken out of the cache, not yet used + assert not sub.log + p.fetch(item) + assert not sub.log def test_a_whole_chunk_cache_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index fdcf19295..7e36edaa2 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -810,6 +810,76 @@ def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): assert len(reads) > 2 # the offsets, and the chunks themselves +def test_a_cache_keeps_where_the_chunks_and_blocks_are(monkeypatch, tmp_path): + # A later run over a partly filled cache asks for blocks it has not got, and + # for nothing else: where the chunks are, and where the blocks inside the + # chunks it half holds are, both came out of the cache + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("keptindex.b2nd", a) + cache = str(tmp_path / "keptindex-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache) + state = holder.schunk.vlmeta["proxy-index"] + assert len(state["offsets"]) == 2 * 8 # one int64 per chunk of the frame + assert [n for n, _ in state["layouts"]] == [0] # the one chunk half held + del holder + + reads, _ = _traffic(monkeypatch) + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert len(reads) == 1 # the header, which an open reads whatever else it knows + + # Different blocks of the chunk already half held: only they travel + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) == 2 + assert np.array_equal(p[...], data) # ... and the rest still reads right + + +def test_a_cache_keeps_nothing_for_a_source_that_cannot_name_its_bytes(tmp_path, monkeypatch): + # Positions in a file are only reusable against the same file, and a source + # without a stamp cannot say it is the same one. Nothing is kept for it, + # rather than kept and hoped for + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("unstamped.b2nd", a) + cache = str(tmp_path / "unstamped-cache.b2nd") + src = blosc2.FsspecNDSource(url) + src.stamp = None # as a source that cannot name its bytes leaves it + blosc2.Proxy(src, urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache) + assert "proxy-index" not in holder.schunk.vlmeta + del holder + + reads, _ = _traffic(monkeypatch) + src = blosc2.FsspecNDSource(url) + src.stamp = None + p = blosc2.Proxy(src, urlpath=cache, mode="a") + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) > 2 # the header, and the offsets and layout it kept nothing of + assert np.array_equal(p[...], data) + + +def test_a_kept_index_of_the_wrong_shape_is_dropped(tmp_path, monkeypatch): + # Belt to the stamp's braces: whatever else went wrong, an index that does not + # fit the source as it stands is read afresh rather than used + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("wrongindex.b2nd", a) + cache = str(tmp_path / "wrongindex-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + holder = blosc2.open(cache, mode="a") + state = dict(holder.schunk.vlmeta["proxy-index"]) + state["offsets"] = state["offsets"][:8] # one chunk's worth, for a frame of two + holder.schunk.vlmeta["proxy-index"] = state + del holder + + reads, _ = _traffic(monkeypatch) + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) + assert len(reads) > 2 # the offsets were read again + assert np.array_equal(p[...], data) + + def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path): # What the deferral is for: a later run over a cache that already covers the # slice fetches nothing, and so has no use for where the chunks are either From 95850a8138d4fab3fc9ff048447675d0d2c16a75 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:51:46 +0200 Subject: [PATCH 23/32] Count the reads an S3 block fetch makes now, and pin the kept index `test_lazy_block_reads` installs its hook after the open, so the offsets read -- which used to happen at the open and now waits for the first chunk asked about -- landed inside the window it counts. Three reads rather than two: where the chunks are, where the blocks of the one wanted are, and the block. The bytes it bounds are unchanged, since the same reads happen either way. Only this file was left, because s3fs and moto are not in the default test environment; the same shift in the memory:// tests went with the commit that caused it. While here, two tests for the index the cache now keeps, in the file that exists for what memory:// cannot show. `ukey` on a real store is an object identity rather than a size, so this is where a replaced object can be checked honestly: one test replaces a frame with another of the same shape and partitioning and holds that the cache is refused, since every offset in it now points elsewhere in a frame of the same size -- a stale chunk serves old data, a stale offset serves nonsense. The other holds that a later run over a half-filled chunk reads one block and nothing else: no offsets, no layout, both out of the cache. Co-Authored-By: Claude Opus 5 --- tests/test_fsspec_s3.py | 59 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 2 deletions(-) diff --git a/tests/test_fsspec_s3.py b/tests/test_fsspec_s3.py index b6cf98522..001cef1d9 100644 --- a/tests/test_fsspec_s3.py +++ b/tests/test_fsspec_s3.py @@ -137,8 +137,63 @@ def test_lazy_block_reads(blocky, max_concurrency): p.src.read_range = lambda *args: (out := original(*args), traffic.append(len(out)))[0] assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) - # The offsets and one block, against 1.4 MB of chunk - assert len(traffic) == 2 + # Where the chunks are, where the blocks of the one wanted are, and one + # block, against 1.4 MB of chunk. The frame's header is not in here: the + # open read that, before the hook went on + assert len(traffic) == 3 assert sum(traffic) < 200_000 # And the array still reads back whole, over the blocks already cached assert np.array_equal(p[...], data) + + +def test_a_kept_index_is_refused_when_the_object_was_replaced(s3_endpoint, tmp_path): + """The cache keeps *positions* now, which a replaced object invalidates. + + A stale chunk cache serves old data; a stale index is worse, since blocks + fetched at the old frame's offsets and spliced into a chunk decode to + nonsense. s3fs is where this can be checked honestly: its `ukey` is a real + object identity, where `memory://` has almost no metadata to build one from. + """ + data = np.random.default_rng(0).random((600, 600)) + url = f"s3://{BUCKET}/replaced.b2nd" + blosc2.asarray(data, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + cache = str(tmp_path / "replaced-cache.b2nd") + + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert np.array_equal(p[10:12, 30:40], data[10:12, 30:40]) + del p + holder = blosc2.open(cache) + assert "proxy-index" in holder.schunk.vlmeta # where the chunks and blocks are + del holder + + # Same shape, same partitioning, other bytes: every offset in there now + # points somewhere else in a frame of the same size + other = np.random.default_rng(1).random((600, 600)) + blosc2.asarray(other, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + with pytest.raises(ValueError, match="different remote bytes"): + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + + # ... and starting afresh reads the new bytes, index and all + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w") + assert np.array_equal(p[10:12, 30:40], other[10:12, 30:40]) + assert np.array_equal(p[...], other) + + +def test_a_kept_index_spares_a_later_run_the_reads(s3_endpoint, tmp_path): + # The same slice geometry as test_lazy_block_reads, one run later: the + # offsets and the layout of the half-held chunk both come out of the cache, + # so only the blocks that are missing travel + data = np.random.default_rng(0).random((600, 600)) + url = f"s3://{BUCKET}/kept.b2nd" + blosc2.asarray(data, chunks=(300, 600), blocks=(30, 600)).save(url, mode="w") + cache = str(tmp_path / "kept-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a").fetch((slice(10, 12), slice(30, 40))) + + src = blosc2.FsspecNDSource(url) + traffic = [] + original = src.read_range + src.read_range = lambda *args: (out := original(*args), traffic.append(len(out)))[0] + p = blosc2.Proxy(src, urlpath=cache, mode="a") + assert np.array_equal(p[60:62, 30:40], data[60:62, 30:40]) + assert len(traffic) == 1 # one block, and nothing to say where it was + assert np.array_equal(p[...], data) From eb7fd15050dfd6cd5817693a464b8cd61a250f65 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 19 Aug 2026 23:57:40 +0200 Subject: [PATCH 24/32] Stop the subscriber stand-in idling half a second per test Each test in this file starts a server and shuts it down, and `shutdown()` waits for `serve_forever`'s poll interval to elapse before the loop notices it was asked to stop. At the default half second, and with 27 tests, that was 13.5 s of a 14.5 s file spent waiting for nothing -- enough to make it the slowest file in the suite, and under `--dist loadfile` the slowest file is the wall clock. A hundredth of a second instead: the file goes from 14.45 s to 0.98 s and out of the top ten, and the whole suite from 26 s to 21 s. Nothing about what is tested changes; the poll interval only decides how often an idle serve loop looks up. Co-Authored-By: Claude Opus 5 --- tests/ndarray/test_c2array_blocks.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 31a636bf1..55f09bf1c 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -183,7 +183,11 @@ def _serve(tmp_path, data, chunks, blocks, name="ds.b2nd", key=None, **kwargs): tstore[key] = blosc2.asarray(data, chunks=chunks, blocks=blocks) server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) server.subscriber = _Subscriber(urlpath, key=key, **kwargs) - threading.Thread(target=server.serve_forever, daemon=True).start() + # A short poll interval, because `shutdown()` waits for one to elapse before + # the serve loop notices: at the default 0.5 s that is half a second of doing + # nothing per test, and this file has enough of them for that to be most of + # what it costs + threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True).start() urlbase = f"http://127.0.0.1:{server.server_address[1]}/" path = f"@public/{name}{key or ''}" array = blosc2.C2Array(path, urlbase=urlbase, auth_token=kwargs.get("cookie")) From 31a84fc7cc37eb06098e23d9e6d7d49c93b067d1 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 00:32:09 +0200 Subject: [PATCH 25/32] Check the cache's stamp, and the bytes a range answer actually carries A review of the block-granularity branch turned up a handful of ways the block path takes an answer on trust: - The stale-index guard compared the stamp with the copy of itself the constructor had just written, so it passed for every writable cache. A `_cache=` handed in never passes `_reopen_cache`, which is where the other check lives, so a replaced remote frame had the old chunk offsets adopted and its blocks spliced at positions that mean nothing now. - `_span_of` checked only that a part *began* before the span it was asked for, so a part ending early sliced short and the payload went into a chunk whose `bstarts` promises the full length. - A 206 without a Content-Range, or a multipart answer without a boundary, raised KeyError/IndexError out of `Proxy.fetch` instead of being refused as an answer that cannot be placed. - A 503 to the one-time probe was written down as "this dataset is streamed" for the life of the array, though nothing was downloaded to find that out. Transient failures now leave the source untried, while a streamed 200 is still an answer for good. - The frame behind a C2Array was never checked against the geometry `api/info` reported for it, though a Proxy takes the block grid from one and the `bstarts` from the other. - `member_window` handed out the *uncompressed* size of a zip member and never checked the member was stored, so a repacked `.b2z` would have had deflated bytes served as a frame. The shared HTTP client's forgetful cookie jar was inert, since httpx re-wraps an `httpx.Cookies` subclass in a plain one and drops the override -- a `Set-Cookie` from any subscriber was kept and replayed to every other. A bare `CookieJar` is handed through instead. And the kept index gets the same treatment: the offsets are written little-endian rather than however this host stores them, a stored header section is adopted only at exactly the length a read of one asks for (a short one parsed to None and pinned that chunk to whole-chunk fetches for good, or tripped over `struct.unpack` if it was really short), and it is written back only when it says something new instead of after every fetch that moved a byte. Two reads that were paying for it: the frame index is taken once per call rather than once per element under its lock, and layout batches past the first are overlapped like any other request rather than run one after the next. Co-Authored-By: Claude Opus 5 --- src/blosc2/c2array.py | 133 ++++++++++++++++++++++----- src/blosc2/dict_store.py | 15 ++- src/blosc2/proxy.py | 63 +++++++++---- tests/ndarray/test_c2array_blocks.py | 68 +++++++++++++- tests/test_fsspec.py | 28 ++++++ 5 files changed, 260 insertions(+), 47 deletions(-) diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index 972607db5..e9260819f 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -10,6 +10,7 @@ import atexit import math import os +import struct import threading from contextlib import contextmanager from typing import TYPE_CHECKING @@ -49,7 +50,7 @@ def _httpx(): _client_lock = threading.Lock() -def _forgetful_cookies(httpx): +def _forgetful_cookies(): """A cookie jar that never keeps anything, for the shared client. A client of its own per request could not carry a cookie from one request to @@ -57,10 +58,15 @@ def _forgetful_cookies(httpx): being read, and arrays with different tokens (or none) share this client. A `Set-Cookie` from any response would otherwise start authorizing requests that asked for none. + + A jar rather than an `httpx.Cookies` subclass: the client re-wraps whatever + it is handed in a plain `httpx.Cookies`, which copies the cookies over and + drops the subclass, but hands a bare `CookieJar` straight through. """ + import http.cookiejar - class _NoCookies(httpx.Cookies): - def extract_cookies(self, response): + class _NoCookies(http.cookiejar.CookieJar): + def extract_cookies(self, response, request): pass return _NoCookies() @@ -90,7 +96,7 @@ def _sync_client(): _client = httpx.Client( timeout=TIMEOUT, limits=httpx.Limits(max_connections=64, max_keepalive_connections=32), - cookies=_forgetful_cookies(httpx), + cookies=_forgetful_cookies(), ) return _client @@ -276,14 +282,36 @@ def slice_to_string(slice_): class _NotRanged(Exception): """The subscriber answered a range request with something other than a 206.""" + def __init__(self, message: str, status: int | None = None): + super().__init__(message) + self.status = status + + @property + def transient(self) -> bool: + """Whether asking again could be answered differently. + + A 200 is the dataset itself, streamed, and no amount of asking again will + make it a file; a server that is busy or broken says nothing at all about + how the dataset is served, and costs no download to ask twice. + """ + return self.status is not None and (self.status >= 500 or self.status == 429) + class _PartsMissing(Exception): """A multi-range answer did not carry all the bytes that were asked for.""" def _content_range(value: str) -> int: - """Where a `Content-Range: bytes start-end/total` header says its part starts.""" - return int(value.split()[1].split("-")[0]) + """Where a `Content-Range: bytes start-end/total` header says its part starts. + + Malformed answers are refused rather than guessed at: `bytes */1234` for a + range that could not be satisfied has no start to read, and a part written to + no shape at all cannot be placed in the frame. + """ + try: + return int(value.split()[1].split("-")[0]) + except (IndexError, ValueError) as exc: + raise _NotRanged(f"a 206 carried an unreadable Content-Range: {value!r}") from exc def _byteranges(response) -> list[tuple[int, bytes]]: @@ -296,8 +324,17 @@ def _byteranges(response) -> list[tuple[int, bytes]]: """ content_type = response.headers.get("content-type", "") if "multipart/byteranges" not in content_type: - return [(_content_range(response.headers["content-range"]), response.content)] - boundary = content_type.split("boundary=")[1].strip().strip('"').encode() + # Where the part sits is the one thing the body cannot say, so an answer + # without it is refused: a caching proxy that strips the header would + # otherwise have its bytes placed wherever they were asked for + single = response.headers.get("content-range") + if single is None: + raise _NotRanged("a 206 arrived without a Content-Range to place it by") + return [(_content_range(single), response.content)] + _, sep, boundary = content_type.partition("boundary=") + if not sep: + raise _NotRanged(f"a multipart answer named no boundary: {content_type!r}") + boundary = boundary.strip().strip('"').encode() parts = [] for chunk in response.content.split(b"--" + boundary): head, sep, body = chunk.partition(b"\r\n\r\n") @@ -315,7 +352,10 @@ def _byteranges(response) -> list[tuple[int, bytes]]: def _span_of(parts: list[tuple[int, bytes]], offset: int, size: int, url: str) -> bytes: """The bytes of one requested span, out of whichever part covers it.""" for start, data in parts: - if start <= offset < start + len(data): + # The whole span, not just its first byte: a part that begins inside it + # and ends early would otherwise slice short, and a short block payload + # is spliced against a `bstarts` that promises the full length + if start <= offset and offset + size <= start + len(data): return data[offset - start : offset - start + size] raise _PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") @@ -344,6 +384,19 @@ def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") self._auth_token = array.auth_token super().__init__(self._url, max_concurrency) + # A `Proxy` mixes the two: the block grid and the fetched bitmap come from + # the array's `api/info`, while the header sections and `bstarts` come from + # this frame. They have to be the same dataset for that to mean anything, + # and the magic bytes alone do not say so -- a window off by a member of a + # `.b2z`, or a path serving a file other than the one described, reads a + # frame that parses and splices blocks into chunks of the wrong shape. + # Geometry alone, since that is what the block arithmetic on both sides is + # built out of, and a dtype `api/info` reports as a repr would fail to + # parse here for a dataset that reads perfectly well + described = (tuple(array.shape), tuple(array.chunks), tuple(array.blocks)) + found = (tuple(self._shape), tuple(self._chunks), tuple(self._blocks)) + if described != found: + raise ValueError(f"{self._url} serves {found}, where its dataset is {described}") def read_range(self, offset: int, size: int) -> bytes: return self._get([(offset, size)])[0] @@ -382,7 +435,10 @@ def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: # Whatever this is, it is not the bytes that were asked for: a 200 # carries the whole dataset, which is the download this exists to # avoid, so leave the body unread on the socket - raise _NotRanged(f"{self._url} answered {response.status_code} to a Range request") + raise _NotRanged( + f"{self._url} answered {response.status_code} to a Range request", + response.status_code, + ) response.read() parts = _byteranges(response) return [_span_of(parts, offset, size, self._url) for offset, size in spans] @@ -701,31 +757,58 @@ def block_source(self) -> C2NDSource | None: with self._block_lock: if self._block_source is _UNTRIED: self._block_source = self._open_block_source() - return self._block_source + # A failure that says nothing about the dataset leaves it _UNTRIED, so the + # next fetch asks again; this one keeps to whole chunks either way + return None if self._block_source is _UNTRIED else self._block_source - def _open_block_source(self) -> C2NDSource | None: - """Decide, at whatever cost it takes, whether this dataset serves ranges.""" + def _open_block_source(self): + """Decide, at whatever cost it takes, whether this dataset serves ranges. + + None for a dataset that does not serve ranges, which is an answer for + good; `_UNTRIED` for a subscriber that could not say, which is not. + """ + httpx = _httpx() # `api/info` rules out a dataset the subscriber computes for nothing: a # stored one reports its geometry where a lazy expression reports # `expression` and `operands` if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): return None - # Nor is a frame of small chunks worth an index read: blosc2 declines to - # take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would ever - # use a block, and the dataset keeps exactly the behaviour it had before - nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) - if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: - return None # Whether a dataset that reports a geometry is *served* from a file is # something only the answer to a range request can say: an HDF5 leaf or a # `.b2z` member reports one and is streamed all the same try: + # Nor is a frame of small chunks worth an index read: blosc2 declines + # to take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would + # ever use a block, and the dataset keeps the behaviour it had before. + # Inside the `try`, since `api/info` need not carry what these read. + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) + if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: + return None source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) source.adopt_index(self._pending_index) return source - except (_NotRanged, ValueError, NotImplementedError, RuntimeError, _httpx().HTTPError): - # Not ranged, not a contiguous frame, not an NDArray, or not - # reachable: whole chunks work for all of those + except _NotRanged as exc: + return _UNTRIED if exc.transient else None + except httpx.HTTPStatusError as exc: + # A busy or broken subscriber said nothing about how this is served + status = exc.response.status_code + return _UNTRIED if status >= 500 or status == 429 else None + except httpx.TransportError: + # Nothing was downloaded to find this out, so asking again is cheap + return _UNTRIED + except ( + _PartsMissing, + ValueError, + NotImplementedError, + RuntimeError, + KeyError, + IndexError, + struct.error, + httpx.HTTPError, + ): + # Not ranged, not a contiguous frame, not an NDArray, answered with + # something unreadable, or described by an `api/info` without the + # fields these read: whole chunks work for all of those return None def adopt_index(self, state) -> None: @@ -760,15 +843,15 @@ def max_ranges(self) -> int: def chunk_layout(self, nchunk: int): """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" - return self.block_source().chunk_layout(nchunk) + return self._ranged().chunk_layout(nchunk) def chunk_layouts(self, nchunks: Sequence[int]) -> list: """The same for several chunks; see :meth:`ByteRangeNDSource.chunk_layouts`.""" - return self.block_source().chunk_layouts(nchunks) + return self._ranged().chunk_layouts(nchunks) def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" - return self.block_source().block_plan(nchunk, nblocks) + return self._ranged().block_plan(nchunk, nblocks) def read_range(self, offset: int, size: int) -> bytes: """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" diff --git a/src/blosc2/dict_store.py b/src/blosc2/dict_store.py index 035be5971..4435de09a 100644 --- a/src/blosc2/dict_store.py +++ b/src/blosc2/dict_store.py @@ -482,7 +482,11 @@ def member_window(self, key: str) -> tuple[int, int] | None: if relpath is None: return None window = self.offsets.get(relpath) - return (window["offset"], window["length"]) if window else None + if not window or not window.get("stored"): + # A deflated member is not a frame where it lies, whatever its suffix + # says, so there is no window onto it to hand out + return None + return (window["offset"], window["length"]) def _annotate_external_value( self, @@ -1000,7 +1004,14 @@ def _get_zip_offsets(self) -> dict[str, dict[str, int]]: filename_len = int.from_bytes(local_header[26:28], "little") extra_len = int.from_bytes(local_header[28:30], "little") data_offset = info.header_offset + 30 + filename_len + extra_len - self.offsets[info.filename] = {"offset": data_offset, "length": info.file_size} + # The *stored* length, which is what lies at `data_offset`: it is + # `file_size` only for a member kept whole, and a `.b2z` repacked + # by any other tool may well have deflated its members instead + self.offsets[info.filename] = { + "offset": data_offset, + "length": info.compress_size, + "stored": info.compress_type == zipfile.ZIP_STORED, + } return self.offsets def close(self) -> None: diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 2da6ad360..4f3c7e2a3 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -402,6 +402,8 @@ def __init__( # Blocks of the last few partly filled chunks, so a rewrite need not read # the chunk back out of the cache to find out what is already in it self._hot_payloads = {} + # The index as last written to the cache, to write it again only if it moves + self._saved_index = None nchunks = self._schunk_cache.nchunks nbits = nchunks * self._blocks_per_chunk self._fetched = bytearray((nbits + 7) // 8) if fresh else self._load_fetched(nchunks) @@ -412,6 +414,9 @@ def __init__( # Geometry alone cannot tell a replaced source from the one the cache was # filled from, so record whatever identity the source can name itself by stamp = getattr(self.src, "stamp", None) + # Read before writing, or the check below would compare the stamp with + # the copy of itself just written and pass for every writable cache + stored = None if fresh else self._schunk_cache.vlmeta.get("proxy-stamp") if stamp is not None and getattr(self._schunk_cache, "mode", None) != "r": # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` # hands over for a persisted proxy: nothing may be written there, and a @@ -422,12 +427,7 @@ def __init__( # same remote bytes, checked here rather than taken on trust from how the # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. adopt = getattr(self.src, "adopt_index", None) - if ( - adopt is not None - and not fresh - and stamp is not None - and self._schunk_cache.vlmeta.get("proxy-stamp") == stamp - ): + if adopt is not None and not fresh and stamp is not None and stored == stamp: adopt(self._schunk_cache.vlmeta.get("proxy-index")) if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) @@ -585,7 +585,13 @@ def _save_fetched(self) -> None: # alone, which are the only ones a later fetch would ask about. state = getattr(self.src, "index_state", None) if state is not None and getattr(self.src, "stamp", None) is not None: - self._schunk_cache.vlmeta["proxy-index"] = state(self._partly_filled()) + index = state(self._partly_filled()) + # Only when it says something new: the offsets are the bulk of it and + # never change once read, so a slice-by-slice walk would otherwise + # rewrite eight bytes per chunk of the frame after every fetch + if index != self._saved_index: + self._schunk_cache.vlmeta["proxy-index"] = index + self._saved_index = index def _partly_filled(self) -> list[int]: """Chunks the cache holds some of the blocks of, but not all.""" @@ -803,9 +809,17 @@ def fetch(task): def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None): """Where the blocks of every one of *nchunks* are, in as few requests as fit.""" - if max(getattr(self.src, "max_ranges", 1), 1) > 1: - # The source batches the reads itself, so there is nothing to overlap - return self.src.chunk_layouts(nchunks) + batch = max(getattr(self.src, "max_ranges", 1), 1) + if batch > 1: + # The source reads a batch of them in one request, but only so many + # per request: the batches past the first are round trips like any + # other, so they are overlapped rather than run one after the next + tasks = list(batched(nchunks, batch)) + return [ + layout + for answers in self._run(self.src.chunk_layouts, tasks, max_concurrency) + for layout in answers + ] return self._run(self.src.chunk_layout, nchunks, max_concurrency) def _write_blocks(self, nchunk: int, payloads: dict[int, bytes], header: bytes) -> None: @@ -1375,7 +1389,9 @@ def index_state(self, keep: Sequence[int] = ()) -> dict: offsets = self._index[0] if self._index is not None else None return { "bpc": self.blocks_per_chunk, - "offsets": b"" if offsets is None else offsets.tobytes(), + # Little-endian whatever the host is: a cache directory outlives the + # machine that filled it, and a stamp cannot tell a byte order + "offsets": b"" if offsets is None else offsets.astype(" None: offsets = state.get("offsets") or b"" if offsets: nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) - array = np.frombuffer(offsets, dtype=np.int64) + array = np.frombuffer(offsets, dtype=" np.dtype: return self._dtype def get_chunk(self, nchunk: int) -> bytes: - offset = int(self._offsets[nchunk]) + # Read the index once: `_offsets` and `_extents` each take the lock, and + # what they hand back does not change after the first read + offsets, extents = self._frame_index() + offset = int(offsets[nchunk]) if offset < 0: return self._special_chunk(offset) - data = self.read_range(offset, int(self._extents[nchunk])) + data = self.read_range(offset, int(extents[nchunk])) return data[: struct.unpack(" bool: and an upper bound on the chunk's compressed size is already in hand from the frame's offsets. See the thresholds at the top of this module. """ - if int(self._offsets[nchunk]) < 0: + offsets, extents = self._frame_index() # once, rather than twice under the lock + if int(offsets[nchunk]) < 0: return False # a run-length chunk has no bytes in the file to skip if nwanted > self.blocks_per_chunk * BLOCK_MAX_FRACTION: return False - return int(self._extents[nchunk]) >= BLOCK_MIN_CBYTES + return int(extents[nchunk]) >= BLOCK_MIN_CBYTES def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | None: """Read where the blocks of a chunk are: its header, bstarts and extents. @@ -1528,8 +1548,9 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: """ section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] + offsets = self._offsets if todo else () # once, not once per span below for batch in batched(todo, max(self.max_ranges, 1)): - spans = [(int(self._offsets[n]), section) for n in batch] + spans = [(int(offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): self._sections[nchunk] = head @@ -1539,6 +1560,11 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: def _parse_layout(self, head: bytes, section: int): """The layout a chunk's header section says it has, or None for no layout.""" nblocks = self.blocks_per_chunk + if len(head) < section: + # A chunk clipped by the end of the frame has no block-offsets section: + # asked for before the header is parsed, so a short one never reaches + # the fields below rather than tripping over their offsets + return None cbytes = struct.unpack(" Date: Thu, 20 Aug 2026 08:24:01 +0200 Subject: [PATCH 26/32] Empty a cache whose remote was replaced, rather than serve it The stamp check now refuses the kept index of a frame that is gone, but the chunks it was fetched into stayed: `blosc2.open` rebuilds a persisted proxy by handing the cache straight over, which is the one route that never passes `_reopen_cache` and its "different remote bytes" refusal. So a remote rewritten between two runs came back as whatever the earlier run had cached, chunk for chunk, with the geometry to make it look right. A cache filled under another stamp now starts as though nothing had been fetched: both bitmaps are cleared and the kept index dropped, before the new stamp is written and never after -- a run that stopped in between would otherwise leave a cache claiming the new bytes while holding the old ones. The chunks themselves are left where they lie, since each is overwritten by the fetch that asks for it again. A cache opened read-only cannot be emptied, and is not believed either: the bitmap it starts with is empty all the same, so `fetch` finds nothing cached, refuses to write, and the read falls through to the source the way an observational proxy already reads past what it cannot fill. The stamp, the index and the bitmap are decided in one place now, which is also what keeps `__init__` under its complexity bar. Co-Authored-By: Claude Opus 5 --- src/blosc2/proxy.py | 89 +++++++++++++++++++++------- tests/ndarray/test_c2array_blocks.py | 60 +++++++++++++++++++ 2 files changed, 128 insertions(+), 21 deletions(-) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 4f3c7e2a3..61ff4be3f 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -316,6 +316,13 @@ def __init__( so a source whose contents changed underneath while its geometry did not is adopted, and the cache keeps serving what the earlier run fetched; pass ``mode="w"`` when that source may have been rewritten. + + A proxy that :func:`blosc2.open` rebuilds over its own cache never + comes through here -- the cache is handed to it as it stands -- so + there a stamp that no longer matches empties the cache instead of + raising: it starts as though nothing had been fetched and fills again + from the bytes served now. Opened read-only there is nothing to + empty, and every read falls through to the source. kwargs: dict, optional Keyword arguments supported: @@ -404,31 +411,11 @@ def __init__( self._hot_payloads = {} # The index as last written to the cache, to write it again only if it moves self._saved_index = None - nchunks = self._schunk_cache.nchunks - nbits = nchunks * self._blocks_per_chunk - self._fetched = bytearray((nbits + 7) // 8) if fresh else self._load_fetched(nchunks) # Evictions the cache has seen, so `_sync_evictions` can spot new ones self._specialized = getattr(self._schunk_cache, "nspecialized", 0) if self.urlpath is None: self.urlpath = getattr(self._schunk_cache, "urlpath", None) - # Geometry alone cannot tell a replaced source from the one the cache was - # filled from, so record whatever identity the source can name itself by - stamp = getattr(self.src, "stamp", None) - # Read before writing, or the check below would compare the stamp with - # the copy of itself just written and pass for every writable cache - stored = None if fresh else self._schunk_cache.vlmeta.get("proxy-stamp") - if stamp is not None and getattr(self._schunk_cache, "mode", None) != "r": - # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` - # hands over for a persisted proxy: nothing may be written there, and a - # proxy over one stays observational anyway (see `__getitem__`) - self._schunk_cache.vlmeta["proxy-stamp"] = stamp - # Where the chunks are, and where the blocks of the partly filled ones - # are, as an earlier run read them. Only from a cache that names the very - # same remote bytes, checked here rather than taken on trust from how the - # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. - adopt = getattr(self.src, "adopt_index", None) - if adopt is not None and not fresh and stamp is not None and stored == stamp: - adopt(self._schunk_cache.vlmeta.get("proxy-index")) + self._fetched = self._adopt_cache(fresh, self._schunk_cache.nchunks) if vlmeta: reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) if reserved: @@ -452,6 +439,66 @@ def _fetched_key(self) -> str: """Where the bitmap lives, which says what it counts: chunks or blocks.""" return "proxy-fetched-blocks" if self._blocks_per_chunk > 1 else "proxy-fetched" + def _adopt_cache(self, fresh: bool, nchunks: int) -> bytearray: + """Take up what the cache holds, as far as it is about the bytes served now. + + Geometry alone cannot tell a replaced source from the one a cache was + filled from, so a source that can name itself has its stamp recorded here + -- read before it is written, or the check would be against the copy of + itself just written and pass for every writable cache. + + A cache filled under another stamp holds the frame that used to be at + that path: the same geometry, and not one byte of it need be the same + data. None of it is reusable -- neither the chunks nor the positions they + were fetched by -- so it starts empty, which for a cache that cannot be + written to means every read falls through to the source (see + `__getitem__`) rather than coming back stale. + """ + stamp = getattr(self.src, "stamp", None) + stored = None if fresh else self._schunk_cache.vlmeta.get("proxy-stamp") + replaced = stamp is not None and stored is not None and stored != stamp + writable = getattr(self._schunk_cache, "mode", None) != "r" + if replaced and writable: + # Before the stamp, never after: the bitmap is what says the chunks + # are there, so a run that stopped in between would leave a cache + # claiming the new bytes and holding the old ones + self._forget_fetched(nchunks) + if stamp is not None and writable: + # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` + # hands over for a persisted proxy: nothing may be written there, and a + # proxy over one stays observational anyway + self._schunk_cache.vlmeta["proxy-stamp"] = stamp + if fresh or replaced: + return bytearray((nchunks * self._blocks_per_chunk + 7) // 8) + # Where the chunks are, and where the blocks of the partly filled ones + # are, as an earlier run read them. Only from a cache that names the very + # same remote bytes, checked here rather than taken on trust from how the + # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. + adopt = getattr(self.src, "adopt_index", None) + if adopt is not None and stamp is not None and stored == stamp: + adopt(self._schunk_cache.vlmeta.get("proxy-index")) + return self._load_fetched(nchunks) + + def _forget_fetched(self, nchunks: int) -> None: + """Empty the cache's record of what it holds, for bytes that are gone. + + The chunks themselves are left where they are: every one of them is + overwritten by the fetch that asks for it again, and rewriting the whole + container here would be a download's worth of work to say `nothing`. + What has to go is every trace of what was fetched, in both the bitmaps a + cache may carry -- the block one this proxy writes, and the chunk one a + run before it may have left -- since either would be believed on the next + open, and where the chunks of the frame that is gone were. + """ + vlmeta = self._schunk_cache.vlmeta + vlmeta[self._fetched_key] = bytes((nchunks * self._blocks_per_chunk + 7) // 8) + if self._fetched_key != "proxy-fetched" and vlmeta.get("proxy-fetched") is not None: + # The chunkwise bitmap a run before the block one left, which + # `_load_fetched` falls back to when there is no block bitmap + vlmeta["proxy-fetched"] = bytes((nchunks + 7) // 8) + if vlmeta.get("proxy-index") is not None: + vlmeta["proxy-index"] = {} + def _load_fetched(self, nchunks: int) -> bytearray: """The bitmap of already fetched blocks that a previous run left behind.""" bpc = self._blocks_per_chunk diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 91a0e3751..ef646f558 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -568,6 +568,66 @@ def test_a_cache_from_other_bytes_is_refused(tmp_path, subscriber, any_chunk_wan assert np.array_equal(p[...], other) +def test_a_cache_of_bytes_that_were_replaced_is_emptied(tmp_path, subscriber, any_chunk_wants_blocks): + # `blosc2.open` rebuilds the proxy over the cache as it stands, with no + # `mode="a"` to refuse it by: the chunks in there were fetched from a frame + # that is gone, so what the cache says it holds is dropped rather than served + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "emptied.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + del p + + other = _incompressible((200, 200), seed=1) + _replace(sub, other, chunks=(100, 200), blocks=(10, 20)) + + reopened = blosc2.open(cache, mode="a") + assert np.array_equal(reopened[0:5, 0:10], other[0:5, 0:10]) + assert np.array_equal(reopened[...], other) + + +def test_a_cache_emptied_of_replaced_bytes_stays_emptied(tmp_path, subscriber, any_chunk_wants_blocks): + # The stamp is written on the way in, so the run after this one finds a cache + # whose stamp fits and believes what it says it holds: emptying it has to + # reach the file, not just the proxy that noticed + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "stillemptied.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert np.array_equal(p[...], data) # every chunk of it, fetched and recorded + del p + + other = _incompressible((200, 200), seed=1) + _replace(sub, other, chunks=(100, 200), blocks=(10, 20)) + noticed = blosc2.open(cache, mode="a") # opened over the new bytes, dropped unread + del noticed + + again = blosc2.open(cache, mode="a") + assert again.schunk.vlmeta["proxy-stamp"] == blosc2.C2Array(array.path, urlbase=array.urlbase).stamp + assert np.array_equal(again[...], other) + + +def test_a_read_only_cache_of_replaced_bytes_reads_past_it(tmp_path, subscriber, any_chunk_wants_blocks): + # Nothing may be written to a cache opened read-only, so it cannot be emptied + # either -- but nothing of it is believed either, and the read falls through + # to the source rather than coming back with the bytes that are gone + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "readonlyreplaced.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert np.array_equal(p[...], data) + del p + + other = _incompressible((200, 200), seed=1) + _replace(sub, other, chunks=(100, 200), blocks=(10, 20)) + + reopened = blosc2.open(cache, mode="r") + assert np.array_equal(reopened[:], other) # read past the cache, off the source + # ... and the cache is left exactly as it was, stamp and chunks alike + assert reopened.schunk.vlmeta["proxy-stamp"] != blosc2.C2Array(array.path, urlbase=array.urlbase).stamp + + def test_a_cache_from_the_same_bytes_is_adopted(tmp_path, subscriber, any_chunk_wants_blocks): data = _incompressible((200, 200)) array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) From 0753fa7dc5e9ea9921b6b173eb29ffcd16e39788 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 08:37:09 +0200 Subject: [PATCH 27/32] Give what a proxy reads from a module of its own `c2array` imported `proxy` at module level, and had to: `C2NDSource` subclasses `ByteRangeNDSource`, which a class statement needs at import time. The suggested fix -- a function-local import -- could not work for that reason, and moving the import in `__init__` could not either: `embed_store` and `dict_store` are bound at lines 565-566 and both reach `c2array` for `C2Array`, while `lazyexpr` reaches `proxy` directly for `convert_dtype`. So `proxy.py` was dragged in ahead of `schunk`, `indexing` and `lazyexpr` by three independent paths, and could import none of them at module level without `import blosc2` failing outright. The half those modules actually want moves to `blosc2/proxy_source.py`: the `ProxySource`/`ProxyNDSource` protocol, the frame reading (header, offsets, chunk and block extents, splicing), `ByteRangeNDSource`, `FsspecNDSource`, `convert_dtype` and the block-granularity thresholds. It reaches into the package for nothing past what `blosc2_ext` binds first. `proxy.py` keeps the `Proxy` itself and imports the rest back, re-exporting the four classes so `blosc2.proxy.X` still answers -- as `blosc2.X` does, unchanged, which is what the docs autoclass. `.proxy` is now bound after `.schunk` and `.indexing`, and the proof is in the diff: `_reopen_cache`'s function-local `from blosc2.schunk import _set_default_dparams`, which was function-local for exactly this reason, is a module-level import again. What moved for good are the tuning constants -- `BLOCK_MIN_CBYTES` and its neighbours are `blosc2.proxy_source.BLOCK_MIN_CBYTES` now, since re-exporting a constant by value would leave anyone setting it on `blosc2.proxy` silently patching a copy nothing reads. The two benches and the tests that set them are updated to say where they live. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 23 +- bench/ndarray/cat2-block-granularity.py | 10 +- bench/ndarray/fsspec-block-granularity.py | 6 +- src/blosc2/__init__.py | 17 +- src/blosc2/c2array.py | 7 +- src/blosc2/lazyexpr.py | 2 +- src/blosc2/ndarray.py | 4 +- src/blosc2/proxy.py | 838 +-------------------- src/blosc2/proxy_source.py | 849 ++++++++++++++++++++++ tests/ndarray/test_c2array_blocks.py | 2 +- tests/ndarray/test_proxy.py | 13 + tests/test_fsspec.py | 10 +- tests/test_fsspec_s3.py | 2 +- 13 files changed, 929 insertions(+), 854 deletions(-) create mode 100644 src/blosc2/proxy_source.py diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 573cdb9b9..9b65e6b25 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -53,7 +53,10 @@ XXX version-specific blurb XXX ranges (`multipart/byteranges`, which no object store offers). A dataset the subscriber *computes* — a lazy expression, an HDF5 leaf, a `.b2z` member — is fetched a whole chunk at a time as before; which it is costs at most one small - request to find out, and is never asked twice. `blosc2.ByteRangeNDSource` is + request to find out, and is not asked again -- unless the subscriber could not + say, which a busy or unreachable one cannot: a 5xx or a connection that failed + is asked again on the next fetch rather than written off, since neither + downloaded anything to find out. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. Opening a remote frame through either of them now costs two requests instead @@ -97,8 +100,22 @@ XXX version-specific blurb XXX bytes they read are held to that as well, so a remote array *replaced* while keeping its shape is noticed rather than served stale: `FsspecNDSource` uses fsspec's token and `C2Array` the subscriber's mtime, both free with metadata - they already fetch. Caches from earlier 4.11.1 development builds are not - adopted (the stamp moved to a `proxy-stamp` entry); pass `mode="w"` once. + they already fetch. A proxy that `blosc2.open` rebuilds over its own cache + gets the same treatment without the raise -- there is no `mode="w"` to offer + it -- so a cache whose stamp no longer matches starts as though nothing had + been fetched and fills again from the bytes served now; opened read-only there + is nothing to empty, and every read falls through to the source. Caches from + earlier 4.11.1 development builds are not adopted (the stamp moved to a + `proxy-stamp` entry); pass `mode="w"` once. + +* The source protocol moved to its own module, `blosc2.proxy_source`: + `ProxySource`, `ProxyNDSource`, `ByteRangeNDSource`, `FsspecNDSource` and the + frame reading behind them. They are still `blosc2.X` and still reachable as + `blosc2.proxy.X`, so nothing outside need change; what moved for good are the + block-granularity knobs, `blosc2.proxy_source.BLOCK_MIN_CBYTES` and its + neighbours. This is what lets `proxy.py` be imported after `schunk` and + `indexing`, rather than being dragged in ahead of them by every module that + wants a source. * Querying a `utf8()` column through its FULL index no longer materializes the index vocabulary. The query literal is turned into an alphabetical rank by diff --git a/bench/ndarray/cat2-block-granularity.py b/bench/ndarray/cat2-block-granularity.py index 7181cc49e..0d2e3ed94 100644 --- a/bench/ndarray/cat2-block-granularity.py +++ b/bench/ndarray/cat2-block-granularity.py @@ -84,7 +84,7 @@ import blosc2 from blosc2 import c2array -CHUNK_HEADER = blosc2.proxy._CHUNK_HEADER_LEN +CHUNK_HEADER = blosc2.proxy_source._CHUNK_HEADER_LEN # @@ -353,8 +353,8 @@ def timed_slice(open_array, item, mode, concurrency, latency, bandwidth): of what it used to do. The frame index is read before the clock starts: it costs four small requests once per C2Array, not once per slice. """ - threshold = blosc2.proxy.BLOCK_MIN_CBYTES - blosc2.proxy.BLOCK_MIN_CBYTES = 1 << 62 if mode == "chunks" else threshold + threshold = blosc2.proxy_source.BLOCK_MIN_CBYTES + blosc2.proxy_source.BLOCK_MIN_CBYTES = 1 << 62 if mode == "chunks" else threshold try: array = open_array() array.max_concurrency = concurrency @@ -367,7 +367,7 @@ def timed_slice(open_array, item, mode, concurrency, latency, bandwidth): proxy[item] return time.perf_counter() - start, tally["requests"], tally["bytes"] finally: - blosc2.proxy.BLOCK_MIN_CBYTES = threshold + blosc2.proxy_source.BLOCK_MIN_CBYTES = threshold def connection_setup(urlbase, path, token, reps): @@ -456,7 +456,7 @@ def open_array(): found_out = ( f"{opening['requests']} request found that out, and it is never asked again" if opening["requests"] - else f"nothing was asked.\n Chunks under the {blosc2.proxy.BLOCK_MIN_CBYTES / 1e6:.0f} MB " + else f"nothing was asked.\n Chunks under the {blosc2.proxy_source.BLOCK_MIN_CBYTES / 1e6:.0f} MB " "a round trip costs are not worth taking apart, and api/info\n tells a computed " "dataset from a stored one for free" ) diff --git a/bench/ndarray/fsspec-block-granularity.py b/bench/ndarray/fsspec-block-granularity.py index 9d5bd6be2..2894757b2 100644 --- a/bench/ndarray/fsspec-block-granularity.py +++ b/bench/ndarray/fsspec-block-granularity.py @@ -223,8 +223,8 @@ def timed_open(urlpath, item, concurrency, blocks, latency=0.0, bandwidth=0.0): *bandwidth* bytes per second across it. Both waits happen inside the proxy's thread pool, so they overlap the way real ones would. """ - old = blosc2.proxy.BLOCK_MIN_CBYTES - blosc2.proxy.BLOCK_MIN_CBYTES = old if blocks else 1 << 62 + old = blosc2.proxy_source.BLOCK_MIN_CBYTES + blosc2.proxy_source.BLOCK_MIN_CBYTES = old if blocks else 1 << 62 try: array = blosc2.open(urlpath, lazy=True, max_concurrency=concurrency) traffic = [0, 0] @@ -246,7 +246,7 @@ def counted(*args, _o=original): array[item] return time.perf_counter() - t0, traffic[0], traffic[1] finally: - blosc2.proxy.BLOCK_MIN_CBYTES = old + blosc2.proxy_source.BLOCK_MIN_CBYTES = old def run_moto(local, concurrency, reps, latency, bandwidth): diff --git a/src/blosc2/__init__.py b/src/blosc2/__init__.py index d6e965463..14433aac1 100644 --- a/src/blosc2/__init__.py +++ b/src/blosc2/__init__.py @@ -588,20 +588,25 @@ def _raise(exc): result_type, can_cast, ) -from .proxy import ( - Proxy, +from .proxy_source import ( ProxySource, ProxyNDSource, - ProxyNDField, ByteRangeNDSource, FsspecNDSource, - SimpleProxy, - jit, - as_simpleproxy, ) from .indexing import Index from .schunk import SChunk, load, open + +# After .schunk and .indexing, which it may now import at module level: what a +# proxy reads from lives in .proxy_source, which the early modules reach instead +from .proxy import ( + Proxy, + ProxyNDField, + SimpleProxy, + jit, + as_simpleproxy, +) from . import linalg from .linalg import tensordot, vecdot, permute_dims, matrix_transpose, matmul, transpose, diagonal, outer from .utils import linalg_funcs as linalg_funcs_list diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index e9260819f..fbf85b4b6 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -23,10 +23,7 @@ import blosc2 from blosc2.b2objects import encode_b2object_payload, make_b2object_carrier, write_b2object_payload from blosc2.info import InfoReporter, format_nbytes_info - -# blosc2/__init__ imports this module before blosc2.proxy, so this pulls proxy in -# early; it is safe because proxy only reaches into the package at call time -from blosc2.proxy import REMOTE_MAX_CONCURRENCY, ByteRangeNDSource +from blosc2.proxy_source import REMOTE_MAX_CONCURRENCY, ByteRangeNDSource _subscriber_data = { "urlbase": os.environ.get("BLOSC_C2URLBASE"), @@ -782,7 +779,7 @@ def _open_block_source(self): # ever use a block, and the dataset keeps the behaviour it had before. # Inside the `try`, since `api/info` need not carry what these read. nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) - if not nchunks or self.cbytes / nchunks < blosc2.proxy.BLOCK_MIN_CBYTES: + if not nchunks or self.cbytes / nchunks < blosc2.proxy_source.BLOCK_MIN_CBYTES: return None source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) source.adopt_index(self._pending_index) diff --git a/src/blosc2/lazyexpr.py b/src/blosc2/lazyexpr.py index f1f4c26ff..2a9cf8473 100644 --- a/src/blosc2/lazyexpr.py +++ b/src/blosc2/lazyexpr.py @@ -54,7 +54,7 @@ write_b2object_user_vlmeta, ) from .dsl_kernel import DSLKernel, DSLSyntaxError, DSLValidator, specialize_miniexpr_inputs -from .proxy import convert_dtype +from .proxy_source import convert_dtype from .utils import ( check_smaller_shape, compute_smaller_slice, diff --git a/src/blosc2/ndarray.py b/src/blosc2/ndarray.py index 914bc5f81..398a9defa 100644 --- a/src/blosc2/ndarray.py +++ b/src/blosc2/ndarray.py @@ -6880,8 +6880,8 @@ def asarray( utf8_out, array = _asarray_string_dispatch(array, copy, kwargs) if utf8_out is not None: return utf8_out - dtype_ = blosc2.proxy.convert_dtype(array.dtype) - dtype = blosc2.proxy.convert_dtype(kwargs.pop("dtype", dtype_)) # check if dtype provided + dtype_ = blosc2.proxy_source.convert_dtype(array.dtype) + dtype = blosc2.proxy_source.convert_dtype(kwargs.pop("dtype", dtype_)) # check if dtype provided kwargs = _check_ndarray_kwargs(**kwargs) chunks = kwargs.pop("chunks", None) blocks = kwargs.pop("blocks", None) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 61ff4be3f..f6cfd57b0 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -11,77 +11,31 @@ import itertools import math import os -import struct import textwrap -import threading -from abc import ABC, abstractmethod from collections.abc import Sequence from concurrent.futures import ThreadPoolExecutor -try: - from itertools import batched -except ImportError: - # Python 3.11 has no itertools.batched - from itertools import islice - - def batched(iterable, n): - it = iter(iterable) - while batch := tuple(islice(it, n)): - yield batch - - -try: - from numpy.typing import DTypeLike -except (ImportError, AttributeError): - # fallback to internal module (use with caution) - from numpy._typing import DTypeLike - import numpy as np import blosc2 from blosc2.dsl_kernel import DSLKernel, DSLSyntaxError -# Default Proxy.afetch concurrency cap for remote sources (e.g. C2Array), -# where fetches are dominated by round-trip latency, not local CPU/IO. -REMOTE_MAX_CONCURRENCY = 8 - -# Block-granular fetching. A chunk is made of blocks that are compressed -# independently, so a slice can fetch only the ones it touches -- at the price of -# one extra round trip, since where the blocks are is written in the chunk header. -# The two thresholds below are what keeps that trade from ever going the wrong -# way; both are decided without reading anything, so a chunk that does not want -# blocks costs exactly what it costs today. Measured against S3 in -# `bench/ndarray/fsspec-block-granularity.py`: 5-17x on multi-MB chunks, and the -# break-even sits at about a megabyte of compressed chunk almost regardless of -# the endpoint, since a round trip buys more bytes exactly where it costs more. -BLOCK_MIN_CBYTES = 1 << 20 -# Above this share of a chunk's blocks the savings no longer pay for the extra -# round trip. Block *count* rather than bytes: it needs no header read, and -# blocks of a chunk are close enough in size for the decision to come out the same. -BLOCK_MAX_FRACTION = 0.5 -# Blocks land in the file roughly, but not exactly, in index order (a -# multithreaded compressor writes them as they finish), so runs of wanted blocks -# are near-adjacent rather than adjacent. Merging across gaps this small trades a -# few unwanted bytes for one less request, which at object-store latencies is -# always the right way round. -BLOCK_GAP = 4096 -# Opening a frame asks two questions whose answers are shorter than the questions -# are dear: how long is the header, and how long is the offsets chunk. Both are -# guessed at generously instead, since over a network a read of a few hundred -# bytes and one of a few kilobytes cost the same. A guess that falls short costs -# the exact read that would have happened anyway, so these are ceilings, not -# promises: an ordinary header is 165-320 bytes, and a frame of 100_000 chunks -# has 4 KB of compressed offsets. -_FRAME_PREFETCH = 8192 -_INDEX_PREFETCH = 1 << 16 - -# How many partly filled chunks keep their blocks in memory as well as in the -# cache. Adding a block to a chunk rewrites that chunk, and the blocks already -# in it have to come from somewhere: from here, or read back out of the cache and -# taken apart again. Keeping the last few saves half the copying for the pattern -# that costs the most -- one chunk filled a slice at a time -- and 8 of them is -# the memory a fetch of 8 whole chunks already peaks at. -BLOCK_HOT_CHUNKS = 8 +# What a proxy reads from, which lived here until the import graph asked for it +# further down (see `proxy_source`). The four classes are re-exported rather +# than used: `blosc2.proxy.X` is where anything outside would look for them +from blosc2.proxy_source import ( # noqa: F401 + BLOCK_HOT_CHUNKS, + REMOTE_MAX_CONCURRENCY, + ByteRangeNDSource, + FsspecNDSource, + ProxyNDSource, + ProxySource, + _chunk_payloads, + _splice_chunk, + batched, + convert_dtype, +) +from blosc2.schunk import _set_default_dparams # vlmeta entries the proxy keeps its own state in: what it has fetched, and which # remote bytes the cache was filled from. A caller cannot write these. @@ -97,188 +51,6 @@ def batched(iterable, n): _JIT_EXECUTION_TUNING_KWARGS = frozenset({"jit", "jit_backend", "fp_accuracy"}) -class ProxyNDSource(ABC): - """ - Base interface for NDim sources in :ref:`Proxy`. - - A source may also serve single *blocks* rather than whole chunks, which is - worth doing when a fetch costs a round trip and a slice touches little of the - chunks it lands in. :ref:`Proxy` uses that path when the source has all four - of ``blocks_per_chunk``, ``wants_blocks(nchunk, nwanted)``, - ``chunk_layout(nchunk)``, ``block_plan(nchunk, nblocks)`` and - ``read_range(offset, size)``; :ref:`FsspecNDSource` implements them over byte - ranges and is the worked example. - - A source whose transport can ask for several ranges at once says so with - ``max_ranges`` and serves ``read_ranges(spans)`` and - ``chunk_layouts(nchunks)`` as well; :ref:`Proxy` then sends a whole wave of - reads as one request. Both are optional, and a source without them is asked - one range at a time exactly as before. - """ - - @property - @abstractmethod - def shape(self) -> tuple: - """ - The shape of the source. - """ - pass - - @property - @abstractmethod - def chunks(self) -> tuple: - """ - The chunk shape of the source. - """ - pass - - @property - @abstractmethod - def blocks(self) -> tuple: - """ - The block shape of the source. - """ - pass - - @property - @abstractmethod - def dtype(self) -> np.dtype: - """ - The dtype of the source. - """ - pass - - @property - def cparams(self) -> blosc2.CParams: - """ - The compression parameters of the source. - - This property is optional and can be overridden if the source has a - different compression configuration. - """ - return blosc2.CParams(typesize=self.dtype.itemsize) - - @abstractmethod - def get_chunk(self, nchunk: int) -> bytes: - """ - Return the compressed chunk in :paramref:`self`. - - Parameters - ---------- - nchunk: int - The unidimensional index of the chunk to retrieve. - - Returns - ------- - out: bytes object - The compressed chunk. - """ - pass - - async def aget_chunk(self, nchunk: int) -> bytes: - """ - Return the compressed chunk in :paramref:`self` asynchronously. - - Parameters - ---------- - nchunk: int - The index of the chunk to retrieve. - - Returns - ------- - out: bytes object - The compressed chunk. - - Notes - ----- - This method is optional, and only available if the source has an async - `aget_chunk` method. - """ - raise NotImplementedError( - "aget_chunk is only available if the source has an async aget_chunk method" - ) - - -class ProxySource(ABC): - """ - Base interface for sources of :ref:`Proxy` that are not NDim objects. - """ - - @property - @abstractmethod - def nbytes(self) -> int: - """ - The total number of bytes in the source. - """ - pass - - @property - @abstractmethod - def chunksize(self) -> tuple: - """ - The chunksize of the source. - """ - pass - - @property - @abstractmethod - def typesize(self) -> int: - """ - The typesize of the source. - """ - pass - - @property - def cparams(self) -> blosc2.CParams: - """ - The compression parameters of the source. - - This property is optional and can be overridden if the source has a - different compression configuration. - """ - return blosc2.CParams(typesize=self.typesize) - - @abstractmethod - def get_chunk(self, nchunk: int) -> bytes: - """ - Return the compressed chunk in :paramref:`self`. - - Parameters - ---------- - nchunk: int - The index of the chunk to retrieve. - - Returns - ------- - out: bytes object - The compressed chunk. - """ - pass - - async def aget_chunk(self, nchunk: int) -> bytes: - """ - Return the compressed chunk in :paramref:`self` asynchronously. - - Parameters - ---------- - nchunk: int - The index of the chunk to retrieve. - - Returns - ------- - out: bytes object - The compressed chunk. - - Notes - ----- - This method is optional and only available if the source has an async - `aget_chunk` method. - """ - raise NotImplementedError( - "aget_chunk is only available if the source has an async aget_chunk method" - ) - - class Proxy(blosc2.Operand): """Proxy (with cache support) for an object following the :ref:`ProxySource` interface. @@ -654,8 +426,6 @@ def _partly_filled(self) -> list[int]: def _reopen_cache(self, urlpath: str): """Adopt the cache container stored at *urlpath*, checking it fits the source.""" - from blosc2.schunk import _set_default_dparams - # Not blosc2.open(): that would rebuild the source we already hold, and # raise outright for sources it cannot reconstruct from the cache metadata kwargs = {} @@ -1153,10 +923,6 @@ def fields(self) -> dict: return {key: ProxyNDField(self, key) for key in _fields} -_FRAME_MAGIC = b"b2frame\0" -_CHUNK_HEADER_LEN = blosc2.MAX_OVERHEAD - - def _item_spans(item, shape) -> list[tuple[int, int]] | None: """The (start, stop) of *item* along every dimension, or None if it is no box. @@ -1173,564 +939,6 @@ def _item_spans(item, shape) -> list[tuple[int, int]] | None: return [(k.start, k.stop) for k in key] -def _block_extents(bstarts: np.ndarray, cbytes: int) -> np.ndarray: - """How many bytes each block of a chunk occupies, given where they start. - - ``bstarts`` is not sorted -- a multithreaded compressor writes blocks in the - order they finish -- so a block ends where the *next one along the file* - begins, which is the same sorted-neighbour rule `_chunk_extents` uses for - chunks inside a frame, one level down. - """ - bounds = np.sort(np.append(bstarts, cbytes)) - return bounds[np.searchsorted(bounds, bstarts, side="right")] - bstarts - - -def _block_streams(header: bytes) -> int: - """How many compressed streams one block of this chunk is stored as. - - One when the chunk is not split, and one per byte of the type when it is - (bit 4 of the flags means *not* split, as blosc1 left it). - """ - return 1 if header[2] & 0x10 else header[3] - - -def _splice_chunk(header: bytes, nblocks: int, payloads: dict[int, bytes]) -> bytes: - """Build a chunk that holds only *payloads*, keyed by block number. - - The blocks that are missing are written as a stream of length zero each, - which the format defines as "made of zeros and stored nowhere", so what comes - back is an ordinary chunk that any reader decodes: exact where the payloads - are, zeros elsewhere. Which is why the proxy has to keep its own record of - what it has actually fetched -- the chunk itself cannot say. - """ - missing = b"\0\0\0\0" * _block_streams(header) - offset = _CHUNK_HEADER_LEN + 4 * nblocks - bstarts, body = np.empty(nblocks, dtype=" dict[int, bytes]: - """The compressed bytes of the *wanted* blocks of an already built chunk.""" - bstarts = np.frombuffer(chunk[_CHUNK_HEADER_LEN : _CHUNK_HEADER_LEN + 4 * nblocks], dtype=" tuple[bytes, list, bytes]: - """Read the header of a contiguous frame. - - *read_range* is ``(offset, size) -> bytes``, so what this costs is round - trips. Returns the raw header bytes, the header decoded as the msgpack - array it is, and the prefetched head those came out of -- which - `_read_frame_offsets` needs, since a frame small enough to have arrived - whole carries its offsets in there too. See ``README_CFRAME_FORMAT.rst`` - in c-blosc2 for the layout. - - The format asks how long the header is before handing it over, and the - question costs as much as the answer. So it is guessed at instead: enough - of the head to hold any ordinary header, and a guess that falls short is - followed by the exact read that would have happened anyway. - - This is everything that says whether a frame can be read this way at all -- - the magic, and the metalayer a caller goes on to decode -- so it is what an - open must do eagerly, and the offsets can wait for the first chunk touched. - """ - import msgpack - - head = read_range(0, _FRAME_PREFETCH) - if head[2:10] != _FRAME_MAGIC: - raise ValueError("not a Blosc2 contiguous frame") - # header_len is the one field that must be located by hand; everything after - # it comes out of unpacking the header, which is plain msgpack - header_len = struct.unpack(">i", head[11:15])[0] - raw = head[:header_len] if header_len <= len(head) else read_range(0, header_len) - # raw=True because the flags field is a msgpack *string* holding four raw - # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up - # that byte is not valid UTF-8 and decoding the header blows up - header = msgpack.unpackb(raw, raw=True, strict_map_key=False) - return raw, header, head - - -def _read_frame_offsets(read_range, header: list, head: bytes, header_len: int) -> np.ndarray: - """The absolute position of every chunk of a frame whose header is in hand. - - A negative position is not a position at all: it encodes a run-length chunk - that was never written to the file. - - Like the header before it, the offsets chunk announces its length in an - answer that costs as much as the chunk does, so the tail the frame's own - length bounds is read instead and the announcement checked against it. A - frame that arrived whole in *head* is already past this and costs nothing. - """ - # An empty frame has no chunks, so it has no offsets chunk either: what sits - # at index_pos is the trailer, and reading it as one fails obscurely - if header[8] == 0: # chunksize - return np.empty(0, dtype=np.int64) - - # The offsets live in a Blosc2 chunk of their own, right after the data ones, - # so all that follows them is the trailer: the frame's own length says how - # much that is, and the cap keeps a large trailer (vlmeta) from being dragged - # along with them. Compressed offsets are small -- 4 KB for a frame of - # 100_000 chunks -- so the cap is reached about as often as the header one is - frame_len, index_pos = header[2], header[1] + header[5] - if len(head) >= frame_len: - index = head[index_pos:] # the whole frame arrived in the first read - else: - index = read_range(index_pos, min(frame_len - index_pos, _INDEX_PREFETCH)) - index_cbytes = struct.unpack(" len(index): - index = read_range(index_pos, index_cbytes) - offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) - # Offsets are relative to the end of the header - return np.where(offsets >= 0, offsets + header_len, offsets) - - -def _frame_metalayer(raw: bytes, header: list, name: str): - """Decode the *name* metalayer out of an already-read frame header.""" - offset = header[13][1][name.encode()] # KeyError if there is no such metalayer - nbytes = struct.unpack(">I", raw[offset + 1 : offset + 5])[0] # msgpack bin32 - import msgpack - - return msgpack.unpackb(raw[offset + 5 : offset + 5 + nbytes], raw=False) - - -def _chunk_extents(offsets: np.ndarray, header: list) -> np.ndarray: - """How many bytes to read at each chunk offset to be sure of covering it. - - A chunk carries its own compressed size in its header, but asking for that - first would cost a second request per chunk. The next thing stored after a - chunk bounds it instead -- another chunk, or the offsets chunk -- capped by - what a chunk can possibly weigh, so a hole left by an updated chunk cannot - turn into an absurd read. The caller truncates to the real size. - """ - index_pos = header[1] + header[5] - bounds = np.sort(np.append(offsets[offsets >= 0], index_pos)) - extents = bounds[np.searchsorted(bounds, offsets, side="right")] - offsets - return np.minimum(extents, header[8] + blosc2.MAX_OVERHEAD) - - -class ByteRangeNDSource(ProxyNDSource): - """A :ref:`Proxy` source that serves parts of a remote Blosc2 frame. - - The frame stays where it is: only its header, its chunk offsets, and what a - slice actually touches ever cross the network. - - A chunk large enough to be worth taking apart is read block by block -- - :meth:`chunk_layout` fetches the offsets of its blocks, :meth:`block_plan` - turns the wanted ones into as few range reads as they fit in -- so a slice - landing in a corner of a multi-megabyte chunk costs a few kilobytes. Small - chunks, memcpyed ones and run-length ones come whole, since there is nothing - to save there; :meth:`wants_blocks` decides which is which without reading - anything. - - Everything above is the Blosc2 frame format and nothing else, so a subclass - only has to say how to read bytes: :meth:`read_range` is the one abstract - method, and the transport behind it decides nothing about the rest. - :ref:`FsspecNDSource` reads them with fsspec, and :ref:`C2Array` reads them - over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. - - A subclass sets its transport up first and then calls this constructor, - which reads the frame's header through it -- one small read, and everything - an open decides: that this is a frame at all, and one holding an NDArray. - Where the chunks are waits for the first one anything asks about, so a - :ref:`Proxy` over a cache that already holds the slice wanted opens the - source without ever fetching its index. It may also set a ``stamp``, - anything that names the exact bytes it reads, so that :ref:`Proxy` can tell - a cache built from other bytes. - - Contiguous frames carrying a ``b2nd`` metalayer only, which is what - :func:`blosc2.asarray` and friends write to a single file. Sparse frames and - ``.b2d`` stores are directories, and cannot be read this way. - - Parameters - ---------- - urlpath: str - Where the frame is, for error messages and for the caller to read back. - max_concurrency: int, optional - How many fetches the enclosing :ref:`Proxy` may run at once. Every chunk - or block costs one range request, so against an object store a slice is - almost entirely round-trip latency and overlapping the requests is the - whole win. Defaults to 8, the same figure :meth:`Proxy.afetch` uses for - remote sources. Pass 1 for a protocol with no latency to hide, where - the thread pool costs about 10 microseconds per chunk and saves nothing. - """ - - stamp = None - - max_ranges = 1 - """How many ranges one request of this transport may carry. - - One means one request each, which is all any object store offers. A - subscriber answering ``multipart/byteranges`` takes more -- see - :meth:`read_ranges` -- and then a slice costs a couple of requests rather - than a couple per chunk it touches. - """ - - def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): - self.max_concurrency = max_concurrency - self.urlpath = urlpath - # Exact ranges, not a file handle: a buffered one reads a whole block per - # seek (50 MiB on s3fs by default), which would undo the point of a lazy - # open. Chunk reads are stateless, so the index below is the only state a - # thread pool shares, and the only thing here that needs a lock - raw, self._header, self._head = _read_frame_header(self.read_range) - self._header_len = len(raw) - self._chunksize = self._header[8] - # Where the chunks are is read on the first one touched, not here: a - # `Proxy` over a cache that already holds the slice asked for fetches - # nothing, and then the offsets are a request spent on nothing at all. - # Everything an open has to decide -- that this is a frame, and one with - # a b2nd metalayer -- is in the header that was just read. - self._index = None - self._index_lock = threading.Lock() - try: - _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, self._header, "b2nd") - except KeyError: - raise NotImplementedError( - f"{urlpath} has no b2nd metalayer, so it is a plain SChunk rather than an " - "NDArray; read it whole or with cache_storage= instead" - ) from None - if dtype_format != 0: - raise NotImplementedError(f"unsupported dtype format {dtype_format} in {urlpath}") - self._shape, self._chunks, self._blocks = tuple(shape), tuple(chunks), tuple(blocks) - try: - self._dtype = np.dtype(dtype) - except TypeError: - # Structured dtypes are stored as their repr, as blosc2_ext does too - self._dtype = np.dtype(ast.literal_eval(dtype)) - # Chunks are padded to whole blocks, so every chunk holds the same number - # of them, edge chunks included. An empty array partitions into nothing - # and has nothing to fetch either, so leave it out of the block path. - self.blocks_per_chunk = ( - math.prod(math.ceil(c / b) for c, b in zip(self._chunks, self._blocks, strict=True)) - if all(self._blocks) - else 1 - ) - # Layouts are memoized for the life of the source, and `_sections` keeps - # the bytes each was read as, so a `Proxy` can hand them to its cache and - # a later run start from them instead of reading them again. - self._layouts = {} - self._sections = {} - - def index_state(self, keep: Sequence[int] = ()) -> dict: - """Where things are, as the bytes they were read as, for a cache to keep. - - The frame's chunk offsets, and the header sections of the chunks in - *keep*. A layout is worth keeping only for a chunk the cache holds some - but not all of: a fetch asks for layouts only where blocks are missing, - so a chunk that is complete is never asked about again, and one that is - empty was never read. - - Kept as read rather than as parsed, so that what comes back goes through - :meth:`_parse_layout` exactly as a fresh read does -- one parser, not a - second one that could disagree with it. - """ - offsets = self._index[0] if self._index is not None else None - return { - "bpc": self.blocks_per_chunk, - # Little-endian whatever the host is: a cache directory outlives the - # machine that filled it, and a stamp cannot tell a byte order - "offsets": b"" if offsets is None else offsets.astype(" None: - """Take up what an earlier run left behind in :meth:`index_state`. - - Only ever called with a state saved against the very same remote bytes -- - :ref:`Proxy` checks the source's ``stamp`` against the one its cache - recorded first -- and that is what makes these safe to reuse. They are - positions *in a file*: a frame replaced underneath keeps its geometry - while every chunk and every block inside it moves, and blocks spliced at - the positions of the frame before it decode to nonsense rather than to - stale data. Anything that does not fit the source as it stands now is - dropped rather than trusted. - """ - if not state or self._index is not None: - return - if state.get("bpc") != self.blocks_per_chunk: - return - section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk - offsets = state.get("offsets") or b"" - if offsets: - nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) - array = np.frombuffer(offsets, dtype=" tuple[np.ndarray, np.ndarray]: - """Where every chunk of the frame is, and how much to read at each. - - Read once, on the first chunk anything asks about. Under a lock because - the fetches this serves run in a thread pool: without one the first wave - of them would each read the index, which is a wasted request apiece and - no worse -- what they read is the same either way. - """ - with self._index_lock: - if self._index is None: - offsets = _read_frame_offsets(self.read_range, self._header, self._head, self._header_len) - self._index = (offsets, _chunk_extents(offsets, self._header)) - self._head = None # the prefetch has nothing left to answer - return self._index - - @property - def _offsets(self) -> np.ndarray: - """Where each chunk begins, negative for one that lives in its offset.""" - return self._frame_index()[0] - - @property - def _extents(self) -> np.ndarray: - """How many bytes to read at each chunk's offset to be sure of covering it.""" - return self._frame_index()[1] - - @property - def shape(self) -> tuple: - return self._shape - - @property - def chunks(self) -> tuple: - return self._chunks - - @property - def blocks(self) -> tuple: - return self._blocks - - @property - def dtype(self) -> np.dtype: - return self._dtype - - def get_chunk(self, nchunk: int) -> bytes: - # Read the index once: `_offsets` and `_extents` each take the lock, and - # what they hand back does not change after the first read - offsets, extents = self._frame_index() - offset = int(offsets[nchunk]) - if offset < 0: - return self._special_chunk(offset) - data = self.read_range(offset, int(extents[nchunk])) - return data[: struct.unpack(" bytes: - """The bytes at [*offset*, *offset* + *size*) of the frame. - - The whole of the transport: everything else here is the frame format. - Fewer bytes may come back only at the end of the frame; anything else is - an error, since the caller has no way to ask for the rest. Must be safe - to call from several threads at once, which is what lets :ref:`Proxy` - overlap the fetches of one slice. - """ - - def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: - """The bytes of every ``(offset, size)`` in *spans*, in that order. - - One request each, unless a transport that can carry several ranges in one - overrides this and raises :attr:`max_ranges` to say how many. Nothing - else has to change for it: this is the only method a batching transport - needs, and everything that reads bytes goes through it. - """ - return [self.read_range(offset, size) for offset, size in spans] - - def wants_blocks(self, nchunk: int, nwanted: int) -> bool: - """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. - - Answered without reading anything, so a chunk that says no costs exactly - what it costs today: the number of blocks a slice touches is geometry, - and an upper bound on the chunk's compressed size is already in hand from - the frame's offsets. See the thresholds at the top of this module. - """ - offsets, extents = self._frame_index() # once, rather than twice under the lock - if int(offsets[nchunk]) < 0: - return False # a run-length chunk has no bytes in the file to skip - if nwanted > self.blocks_per_chunk * BLOCK_MAX_FRACTION: - return False - return int(extents[nchunk]) >= BLOCK_MIN_CBYTES - - def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | None: - """Read where the blocks of a chunk are: its header, bstarts and extents. - - One range read, of a size known in advance since every chunk holds the - same number of blocks. None for a chunk this cannot take apart, which the - header is what says: - - - a chunk of a single block is its own block, and a memcpyed one stores its - blocks raw with no ``bstarts`` at all; - - a chunk that is a run of one value is its header and that value, with no - blocks in the file: `blosc2.full` writes those at a real offset, unlike - the runs of zeros the frame keeps in the offsets themselves; - - a chunk compressed against a codec dictionary keeps it between - ``bstarts`` and the streams, and `_splice_chunk` would drop it while - leaving the flag that promises it; - - a variable-length-block chunk does not use the zero-length stream that - stands in for a block `_splice_chunk` does not have; - - a chunk without the extended header keeps its ``bstarts`` somewhere else - entirely, so reading them at byte 32 would be reading data. - - Each of those is then fetched whole, at the cost of the one header read - that found out. - """ - return self.chunk_layouts((nchunk,))[0] - - def chunk_layouts(self, nchunks: Sequence[int]) -> list: - """:meth:`chunk_layout` for several chunks, in as few requests as they fit. - - One request each unless the transport takes several ranges at once, and - none at all for a chunk already read: a fetch asks for layouts only where - blocks are missing, but the same chunk comes up again as a slice fills it - in. - """ - section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk - todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] - offsets = self._offsets if todo else () # once, not once per span below - for batch in batched(todo, max(self.max_ranges, 1)): - spans = [(int(offsets[n]), section) for n in batch] - heads = self.read_ranges(spans) - for nchunk, head in zip(batch, heads, strict=True): - self._sections[nchunk] = head - self._layouts[nchunk] = self._parse_layout(head, section) - return [self._layouts[n] for n in nchunks] - - def _parse_layout(self, head: bytes, section: int): - """The layout a chunk's header section says it has, or None for no layout.""" - nblocks = self.blocks_per_chunk - if len(head) < section: - # A chunk clipped by the end of the frame has no block-offsets section: - # asked for before the header is parsed, so a short one never reaches - # the fields below rather than tripping over their offsets - return None - cbytes = struct.unpack("> 4) & 0x7 # a run of one value: header and value, no blocks - or head[31] & 0x01 # compressed against a dictionary - or head[30] & 0x01 # variable-length blocks - or not extended - # Whatever else a chunk may be, one too small to hold a block-offsets - # section has none: reading 32 bytes in would be reading the next chunk - or cbytes < section - ): - return None - bstarts = np.frombuffer(head[_CHUNK_HEADER_LEN:section], dtype=" list[tuple[int, int, tuple]]: - """The range reads that cover *nblocks*, near-adjacent ones merged. - - Each is ``(offset, size, members)`` in frame coordinates, where members - says which block each piece of the answer is, as - ``(nblock, offset within the read, size)``. - """ - _, bstarts, extents = self._layouts[nchunk] - base = int(self._offsets[nchunk]) - plan = [] - for nblock in sorted(nblocks, key=lambda n: bstarts[n]): - start, size = int(bstarts[nblock]), int(extents[nblock]) - if plan and start - (plan[-1][0] + plan[-1][1]) <= BLOCK_GAP: - offset, length, members = plan[-1] - plan[-1] = (offset, max(length, start + size - offset), (*members, (nblock, start, size))) - else: - plan.append((start, size, ((nblock, start, size),))) - return [ - (base + offset, length, tuple((n, s - offset, size) for n, s, size in members)) - for offset, length, members in plan - ] - - async def aget_chunk(self, nchunk: int) -> bytes: - """Same as :meth:`get_chunk`, but without blocking the caller's event loop. - - This is what makes :meth:`Proxy.afetch` worth using against an object - store, where a slice spanning many chunks is nearly all round-trip - latency. - - The fetch goes to a worker thread rather than being awaited directly. - Awaiting an async filesystem's coroutine looks like the obvious thing to - do and does not work: fsspec drives those on a private event loop of its - own, so a client created there and awaited here raises "got Future - attached to a different loop" (seen with s3fs). Its blocking API is the - supported way in, and it hands off to that same private loop, so the - thread parks on a queue rather than on a socket. - """ - offset = int(self._offsets[nchunk]) - if offset < 0: - return self._special_chunk(offset) - loop = asyncio.get_running_loop() - return await loop.run_in_executor(None, self.get_chunk, nchunk) - - def _special_chunk(self, offset: int) -> bytes: - """Rebuild a run-length chunk, which lives in its offset instead of the file.""" - kind = ((offset & 0xFFFFFFFFFFFFFFFF) >> 56) & 0x7 - nitems = self._chunksize // self._dtype.itemsize - if kind == 2: - data = np.full(nitems, np.nan, dtype=self._dtype) - else: - # A run of zeros (1); uninitialized chunks (4) have no defined - # content, and zeros is what reading them locally hands back too - data = np.zeros(nitems, dtype=self._dtype) - # The blocksize has to be the container's: left to choose, blosc2 takes - # the whole chunk, and the cache then rejects the chunk we hand it - return blosc2.compress2( - data, - typesize=self._dtype.itemsize, - blocksize=int(np.prod(self._blocks)) * self._dtype.itemsize, - ) - - -class FsspecNDSource(ByteRangeNDSource): - """A :ref:`ByteRangeNDSource` reading its frame through fsspec. - - This is what ``blosc2.open(url, lazy=True)`` builds; wrap it in a - :ref:`Proxy` by hand when the cache belongs at a path of your choosing - rather than inside ``cache_storage``:: - - src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") - a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") - - Parameters - ---------- - urlpath: str - The fsspec URL of the frame. - max_concurrency: int, optional - As in :ref:`ByteRangeNDSource`. - """ - - def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): - from blosc2.core import _import_fsspec - - fsspec = _import_fsspec(urlpath) - fs, path = fsspec.url_to_fs(urlpath) - if fs.isdir(path): - raise NotImplementedError( - f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " - "chunk by chunk; open it with cache_storage= instead" - ) - self._fs, self._path = fs, path - # Identifies the remote bytes, so a cache built against them can tell it - # has gone stale -- and chunk offsets from a replaced frame are garbage. - # fsspec's own token, rather than a tuple of the metadata fields we guess - # a backend exposes: memory:// has no mtime, which left it size-only. - self.stamp = fs.ukey(path) - super().__init__(urlpath, max_concurrency) - - def read_range(self, offset: int, size: int) -> bytes: - return self._fs.cat_file(self._path, start=offset, end=offset + size) - - class ProxyNDField(blosc2.Operand): def __init__(self, proxy: Proxy, field: str): self.proxy = proxy @@ -1781,20 +989,6 @@ def __getitem__(self, item: slice | list[slice]) -> np.ndarray: return nparr[self.field] -def convert_dtype(dt: str | DTypeLike): - """ - Attempts to convert to blosc2.dtype (i.e. numpy dtype) - """ - if hasattr(dt, "as_numpy_dtype"): - dt = dt.as_numpy_dtype - try: - return np.dtype(dt) - except TypeError: # likely passed e.g. a torch.float64 - return np.dtype(str(dt).split(".")[1]) - except Exception as e: - raise TypeError(f"Could not parse dtype arg {dt}.") from e - - class SimpleProxy(blosc2.Operand): """ Simple proxy for any data container to be used with the compute engine. diff --git a/src/blosc2/proxy_source.py b/src/blosc2/proxy_source.py new file mode 100644 index 000000000..8fa69f6a9 --- /dev/null +++ b/src/blosc2/proxy_source.py @@ -0,0 +1,849 @@ +####################################################################### +# Copyright (c) 2019-present, Blosc Development Team +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause +####################################################################### + +"""What a :ref:`Proxy` reads from: the source protocol, and a frame over byte ranges. + +Kept apart from `proxy.py` for the sake of the import graph. `blosc2/__init__` +binds its submodules in an order, and `embed_store`, `dict_store` and `lazyexpr` +are bound early and reach in here: the first two through `c2array`, whose +`C2NDSource` subclasses :ref:`ByteRangeNDSource` at import time, and `lazyexpr` +for `convert_dtype`. Were these still in `proxy.py`, that module would be +dragged in with them, ahead of `schunk`, `indexing` and `lazyexpr` -- and could +then import none of those at module level without `import blosc2` failing +outright. Nothing here reaches into the package for more than what +`blosc2_ext` binds first. +""" + +import ast +import asyncio +import math +import struct +import threading +from abc import ABC, abstractmethod +from collections.abc import Sequence + +try: + from itertools import batched +except ImportError: + # Python 3.11 has no itertools.batched + from itertools import islice + + def batched(iterable, n): + it = iter(iterable) + while batch := tuple(islice(it, n)): + yield batch + + +try: + from numpy.typing import DTypeLike +except (ImportError, AttributeError): + # fallback to internal module (use with caution) + from numpy._typing import DTypeLike + +import numpy as np + +import blosc2 + +# Default Proxy.afetch concurrency cap for remote sources (e.g. C2Array), +# where fetches are dominated by round-trip latency, not local CPU/IO. +REMOTE_MAX_CONCURRENCY = 8 + +# Block-granular fetching. A chunk is made of blocks that are compressed +# independently, so a slice can fetch only the ones it touches -- at the price of +# one extra round trip, since where the blocks are is written in the chunk header. +# The two thresholds below are what keeps that trade from ever going the wrong +# way; both are decided without reading anything, so a chunk that does not want +# blocks costs exactly what it costs today. Measured against S3 in +# `bench/ndarray/fsspec-block-granularity.py`: 5-17x on multi-MB chunks, and the +# break-even sits at about a megabyte of compressed chunk almost regardless of +# the endpoint, since a round trip buys more bytes exactly where it costs more. +BLOCK_MIN_CBYTES = 1 << 20 +# Above this share of a chunk's blocks the savings no longer pay for the extra +# round trip. Block *count* rather than bytes: it needs no header read, and +# blocks of a chunk are close enough in size for the decision to come out the same. +BLOCK_MAX_FRACTION = 0.5 +# Blocks land in the file roughly, but not exactly, in index order (a +# multithreaded compressor writes them as they finish), so runs of wanted blocks +# are near-adjacent rather than adjacent. Merging across gaps this small trades a +# few unwanted bytes for one less request, which at object-store latencies is +# always the right way round. +BLOCK_GAP = 4096 +# Opening a frame asks two questions whose answers are shorter than the questions +# are dear: how long is the header, and how long is the offsets chunk. Both are +# guessed at generously instead, since over a network a read of a few hundred +# bytes and one of a few kilobytes cost the same. A guess that falls short costs +# the exact read that would have happened anyway, so these are ceilings, not +# promises: an ordinary header is 165-320 bytes, and a frame of 100_000 chunks +# has 4 KB of compressed offsets. +_FRAME_PREFETCH = 8192 +_INDEX_PREFETCH = 1 << 16 + +# How many partly filled chunks keep their blocks in memory as well as in the +# cache. Adding a block to a chunk rewrites that chunk, and the blocks already +# in it have to come from somewhere: from here, or read back out of the cache and +# taken apart again. Keeping the last few saves half the copying for the pattern +# that costs the most -- one chunk filled a slice at a time -- and 8 of them is +# the memory a fetch of 8 whole chunks already peaks at. +BLOCK_HOT_CHUNKS = 8 + + +class ProxyNDSource(ABC): + """ + Base interface for NDim sources in :ref:`Proxy`. + + A source may also serve single *blocks* rather than whole chunks, which is + worth doing when a fetch costs a round trip and a slice touches little of the + chunks it lands in. :ref:`Proxy` uses that path when the source has all four + of ``blocks_per_chunk``, ``wants_blocks(nchunk, nwanted)``, + ``chunk_layout(nchunk)``, ``block_plan(nchunk, nblocks)`` and + ``read_range(offset, size)``; :ref:`FsspecNDSource` implements them over byte + ranges and is the worked example. + + A source whose transport can ask for several ranges at once says so with + ``max_ranges`` and serves ``read_ranges(spans)`` and + ``chunk_layouts(nchunks)`` as well; :ref:`Proxy` then sends a whole wave of + reads as one request. Both are optional, and a source without them is asked + one range at a time exactly as before. + """ + + @property + @abstractmethod + def shape(self) -> tuple: + """ + The shape of the source. + """ + pass + + @property + @abstractmethod + def chunks(self) -> tuple: + """ + The chunk shape of the source. + """ + pass + + @property + @abstractmethod + def blocks(self) -> tuple: + """ + The block shape of the source. + """ + pass + + @property + @abstractmethod + def dtype(self) -> np.dtype: + """ + The dtype of the source. + """ + pass + + @property + def cparams(self) -> blosc2.CParams: + """ + The compression parameters of the source. + + This property is optional and can be overridden if the source has a + different compression configuration. + """ + return blosc2.CParams(typesize=self.dtype.itemsize) + + @abstractmethod + def get_chunk(self, nchunk: int) -> bytes: + """ + Return the compressed chunk in :paramref:`self`. + + Parameters + ---------- + nchunk: int + The unidimensional index of the chunk to retrieve. + + Returns + ------- + out: bytes object + The compressed chunk. + """ + pass + + async def aget_chunk(self, nchunk: int) -> bytes: + """ + Return the compressed chunk in :paramref:`self` asynchronously. + + Parameters + ---------- + nchunk: int + The index of the chunk to retrieve. + + Returns + ------- + out: bytes object + The compressed chunk. + + Notes + ----- + This method is optional, and only available if the source has an async + `aget_chunk` method. + """ + raise NotImplementedError( + "aget_chunk is only available if the source has an async aget_chunk method" + ) + + +class ProxySource(ABC): + """ + Base interface for sources of :ref:`Proxy` that are not NDim objects. + """ + + @property + @abstractmethod + def nbytes(self) -> int: + """ + The total number of bytes in the source. + """ + pass + + @property + @abstractmethod + def chunksize(self) -> tuple: + """ + The chunksize of the source. + """ + pass + + @property + @abstractmethod + def typesize(self) -> int: + """ + The typesize of the source. + """ + pass + + @property + def cparams(self) -> blosc2.CParams: + """ + The compression parameters of the source. + + This property is optional and can be overridden if the source has a + different compression configuration. + """ + return blosc2.CParams(typesize=self.typesize) + + @abstractmethod + def get_chunk(self, nchunk: int) -> bytes: + """ + Return the compressed chunk in :paramref:`self`. + + Parameters + ---------- + nchunk: int + The index of the chunk to retrieve. + + Returns + ------- + out: bytes object + The compressed chunk. + """ + pass + + async def aget_chunk(self, nchunk: int) -> bytes: + """ + Return the compressed chunk in :paramref:`self` asynchronously. + + Parameters + ---------- + nchunk: int + The index of the chunk to retrieve. + + Returns + ------- + out: bytes object + The compressed chunk. + + Notes + ----- + This method is optional and only available if the source has an async + `aget_chunk` method. + """ + raise NotImplementedError( + "aget_chunk is only available if the source has an async aget_chunk method" + ) + + +_FRAME_MAGIC = b"b2frame\0" +_CHUNK_HEADER_LEN = blosc2.MAX_OVERHEAD + + +def _block_extents(bstarts: np.ndarray, cbytes: int) -> np.ndarray: + """How many bytes each block of a chunk occupies, given where they start. + + ``bstarts`` is not sorted -- a multithreaded compressor writes blocks in the + order they finish -- so a block ends where the *next one along the file* + begins, which is the same sorted-neighbour rule `_chunk_extents` uses for + chunks inside a frame, one level down. + """ + bounds = np.sort(np.append(bstarts, cbytes)) + return bounds[np.searchsorted(bounds, bstarts, side="right")] - bstarts + + +def _block_streams(header: bytes) -> int: + """How many compressed streams one block of this chunk is stored as. + + One when the chunk is not split, and one per byte of the type when it is + (bit 4 of the flags means *not* split, as blosc1 left it). + """ + return 1 if header[2] & 0x10 else header[3] + + +def _splice_chunk(header: bytes, nblocks: int, payloads: dict[int, bytes]) -> bytes: + """Build a chunk that holds only *payloads*, keyed by block number. + + The blocks that are missing are written as a stream of length zero each, + which the format defines as "made of zeros and stored nowhere", so what comes + back is an ordinary chunk that any reader decodes: exact where the payloads + are, zeros elsewhere. Which is why the proxy has to keep its own record of + what it has actually fetched -- the chunk itself cannot say. + """ + missing = b"\0\0\0\0" * _block_streams(header) + offset = _CHUNK_HEADER_LEN + 4 * nblocks + bstarts, body = np.empty(nblocks, dtype=" dict[int, bytes]: + """The compressed bytes of the *wanted* blocks of an already built chunk.""" + bstarts = np.frombuffer(chunk[_CHUNK_HEADER_LEN : _CHUNK_HEADER_LEN + 4 * nblocks], dtype=" tuple[bytes, list, bytes]: + """Read the header of a contiguous frame. + + *read_range* is ``(offset, size) -> bytes``, so what this costs is round + trips. Returns the raw header bytes, the header decoded as the msgpack + array it is, and the prefetched head those came out of -- which + `_read_frame_offsets` needs, since a frame small enough to have arrived + whole carries its offsets in there too. See ``README_CFRAME_FORMAT.rst`` + in c-blosc2 for the layout. + + The format asks how long the header is before handing it over, and the + question costs as much as the answer. So it is guessed at instead: enough + of the head to hold any ordinary header, and a guess that falls short is + followed by the exact read that would have happened anyway. + + This is everything that says whether a frame can be read this way at all -- + the magic, and the metalayer a caller goes on to decode -- so it is what an + open must do eagerly, and the offsets can wait for the first chunk touched. + """ + import msgpack + + head = read_range(0, _FRAME_PREFETCH) + if head[2:10] != _FRAME_MAGIC: + raise ValueError("not a Blosc2 contiguous frame") + # header_len is the one field that must be located by hand; everything after + # it comes out of unpacking the header, which is plain msgpack + header_len = struct.unpack(">i", head[11:15])[0] + raw = head[:header_len] if header_len <= len(head) else read_range(0, header_len) + # raw=True because the flags field is a msgpack *string* holding four raw + # bytes, and codec_flags packs clevel into its high nibble: from clevel 8 up + # that byte is not valid UTF-8 and decoding the header blows up + header = msgpack.unpackb(raw, raw=True, strict_map_key=False) + return raw, header, head + + +def _read_frame_offsets(read_range, header: list, head: bytes, header_len: int) -> np.ndarray: + """The absolute position of every chunk of a frame whose header is in hand. + + A negative position is not a position at all: it encodes a run-length chunk + that was never written to the file. + + Like the header before it, the offsets chunk announces its length in an + answer that costs as much as the chunk does, so the tail the frame's own + length bounds is read instead and the announcement checked against it. A + frame that arrived whole in *head* is already past this and costs nothing. + """ + # An empty frame has no chunks, so it has no offsets chunk either: what sits + # at index_pos is the trailer, and reading it as one fails obscurely + if header[8] == 0: # chunksize + return np.empty(0, dtype=np.int64) + + # The offsets live in a Blosc2 chunk of their own, right after the data ones, + # so all that follows them is the trailer: the frame's own length says how + # much that is, and the cap keeps a large trailer (vlmeta) from being dragged + # along with them. Compressed offsets are small -- 4 KB for a frame of + # 100_000 chunks -- so the cap is reached about as often as the header one is + frame_len, index_pos = header[2], header[1] + header[5] + if len(head) >= frame_len: + index = head[index_pos:] # the whole frame arrived in the first read + else: + index = read_range(index_pos, min(frame_len - index_pos, _INDEX_PREFETCH)) + index_cbytes = struct.unpack(" len(index): + index = read_range(index_pos, index_cbytes) + offsets = np.frombuffer(blosc2.decompress2(index[:index_cbytes]), dtype=np.int64) + # Offsets are relative to the end of the header + return np.where(offsets >= 0, offsets + header_len, offsets) + + +def _frame_metalayer(raw: bytes, header: list, name: str): + """Decode the *name* metalayer out of an already-read frame header.""" + offset = header[13][1][name.encode()] # KeyError if there is no such metalayer + nbytes = struct.unpack(">I", raw[offset + 1 : offset + 5])[0] # msgpack bin32 + import msgpack + + return msgpack.unpackb(raw[offset + 5 : offset + 5 + nbytes], raw=False) + + +def _chunk_extents(offsets: np.ndarray, header: list) -> np.ndarray: + """How many bytes to read at each chunk offset to be sure of covering it. + + A chunk carries its own compressed size in its header, but asking for that + first would cost a second request per chunk. The next thing stored after a + chunk bounds it instead -- another chunk, or the offsets chunk -- capped by + what a chunk can possibly weigh, so a hole left by an updated chunk cannot + turn into an absurd read. The caller truncates to the real size. + """ + index_pos = header[1] + header[5] + bounds = np.sort(np.append(offsets[offsets >= 0], index_pos)) + extents = bounds[np.searchsorted(bounds, offsets, side="right")] - offsets + return np.minimum(extents, header[8] + blosc2.MAX_OVERHEAD) + + +class ByteRangeNDSource(ProxyNDSource): + """A :ref:`Proxy` source that serves parts of a remote Blosc2 frame. + + The frame stays where it is: only its header, its chunk offsets, and what a + slice actually touches ever cross the network. + + A chunk large enough to be worth taking apart is read block by block -- + :meth:`chunk_layout` fetches the offsets of its blocks, :meth:`block_plan` + turns the wanted ones into as few range reads as they fit in -- so a slice + landing in a corner of a multi-megabyte chunk costs a few kilobytes. Small + chunks, memcpyed ones and run-length ones come whole, since there is nothing + to save there; :meth:`wants_blocks` decides which is which without reading + anything. + + Everything above is the Blosc2 frame format and nothing else, so a subclass + only has to say how to read bytes: :meth:`read_range` is the one abstract + method, and the transport behind it decides nothing about the rest. + :ref:`FsspecNDSource` reads them with fsspec, and :ref:`C2Array` reads them + over HTTP ranges from a Caterva2 subscriber, carrying its auth cookie. + + A subclass sets its transport up first and then calls this constructor, + which reads the frame's header through it -- one small read, and everything + an open decides: that this is a frame at all, and one holding an NDArray. + Where the chunks are waits for the first one anything asks about, so a + :ref:`Proxy` over a cache that already holds the slice wanted opens the + source without ever fetching its index. It may also set a ``stamp``, + anything that names the exact bytes it reads, so that :ref:`Proxy` can tell + a cache built from other bytes. + + Contiguous frames carrying a ``b2nd`` metalayer only, which is what + :func:`blosc2.asarray` and friends write to a single file. Sparse frames and + ``.b2d`` stores are directories, and cannot be read this way. + + Parameters + ---------- + urlpath: str + Where the frame is, for error messages and for the caller to read back. + max_concurrency: int, optional + How many fetches the enclosing :ref:`Proxy` may run at once. Every chunk + or block costs one range request, so against an object store a slice is + almost entirely round-trip latency and overlapping the requests is the + whole win. Defaults to 8, the same figure :meth:`Proxy.afetch` uses for + remote sources. Pass 1 for a protocol with no latency to hide, where + the thread pool costs about 10 microseconds per chunk and saves nothing. + """ + + stamp = None + + max_ranges = 1 + """How many ranges one request of this transport may carry. + + One means one request each, which is all any object store offers. A + subscriber answering ``multipart/byteranges`` takes more -- see + :meth:`read_ranges` -- and then a slice costs a couple of requests rather + than a couple per chunk it touches. + """ + + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + self.max_concurrency = max_concurrency + self.urlpath = urlpath + # Exact ranges, not a file handle: a buffered one reads a whole block per + # seek (50 MiB on s3fs by default), which would undo the point of a lazy + # open. Chunk reads are stateless, so the index below is the only state a + # thread pool shares, and the only thing here that needs a lock + raw, self._header, self._head = _read_frame_header(self.read_range) + self._header_len = len(raw) + self._chunksize = self._header[8] + # Where the chunks are is read on the first one touched, not here: a + # `Proxy` over a cache that already holds the slice asked for fetches + # nothing, and then the offsets are a request spent on nothing at all. + # Everything an open has to decide -- that this is a frame, and one with + # a b2nd metalayer -- is in the header that was just read. + self._index = None + self._index_lock = threading.Lock() + try: + _, _, shape, chunks, blocks, dtype_format, dtype = _frame_metalayer(raw, self._header, "b2nd") + except KeyError: + raise NotImplementedError( + f"{urlpath} has no b2nd metalayer, so it is a plain SChunk rather than an " + "NDArray; read it whole or with cache_storage= instead" + ) from None + if dtype_format != 0: + raise NotImplementedError(f"unsupported dtype format {dtype_format} in {urlpath}") + self._shape, self._chunks, self._blocks = tuple(shape), tuple(chunks), tuple(blocks) + try: + self._dtype = np.dtype(dtype) + except TypeError: + # Structured dtypes are stored as their repr, as blosc2_ext does too + self._dtype = np.dtype(ast.literal_eval(dtype)) + # Chunks are padded to whole blocks, so every chunk holds the same number + # of them, edge chunks included. An empty array partitions into nothing + # and has nothing to fetch either, so leave it out of the block path. + self.blocks_per_chunk = ( + math.prod(math.ceil(c / b) for c, b in zip(self._chunks, self._blocks, strict=True)) + if all(self._blocks) + else 1 + ) + # Layouts are memoized for the life of the source, and `_sections` keeps + # the bytes each was read as, so a `Proxy` can hand them to its cache and + # a later run start from them instead of reading them again. + self._layouts = {} + self._sections = {} + + def index_state(self, keep: Sequence[int] = ()) -> dict: + """Where things are, as the bytes they were read as, for a cache to keep. + + The frame's chunk offsets, and the header sections of the chunks in + *keep*. A layout is worth keeping only for a chunk the cache holds some + but not all of: a fetch asks for layouts only where blocks are missing, + so a chunk that is complete is never asked about again, and one that is + empty was never read. + + Kept as read rather than as parsed, so that what comes back goes through + :meth:`_parse_layout` exactly as a fresh read does -- one parser, not a + second one that could disagree with it. + """ + offsets = self._index[0] if self._index is not None else None + return { + "bpc": self.blocks_per_chunk, + # Little-endian whatever the host is: a cache directory outlives the + # machine that filled it, and a stamp cannot tell a byte order + "offsets": b"" if offsets is None else offsets.astype(" None: + """Take up what an earlier run left behind in :meth:`index_state`. + + Only ever called with a state saved against the very same remote bytes -- + :ref:`Proxy` checks the source's ``stamp`` against the one its cache + recorded first -- and that is what makes these safe to reuse. They are + positions *in a file*: a frame replaced underneath keeps its geometry + while every chunk and every block inside it moves, and blocks spliced at + the positions of the frame before it decode to nonsense rather than to + stale data. Anything that does not fit the source as it stands now is + dropped rather than trusted. + """ + if not state or self._index is not None: + return + if state.get("bpc") != self.blocks_per_chunk: + return + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + offsets = state.get("offsets") or b"" + if offsets: + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) + array = np.frombuffer(offsets, dtype=" tuple[np.ndarray, np.ndarray]: + """Where every chunk of the frame is, and how much to read at each. + + Read once, on the first chunk anything asks about. Under a lock because + the fetches this serves run in a thread pool: without one the first wave + of them would each read the index, which is a wasted request apiece and + no worse -- what they read is the same either way. + """ + with self._index_lock: + if self._index is None: + offsets = _read_frame_offsets(self.read_range, self._header, self._head, self._header_len) + self._index = (offsets, _chunk_extents(offsets, self._header)) + self._head = None # the prefetch has nothing left to answer + return self._index + + @property + def _offsets(self) -> np.ndarray: + """Where each chunk begins, negative for one that lives in its offset.""" + return self._frame_index()[0] + + @property + def _extents(self) -> np.ndarray: + """How many bytes to read at each chunk's offset to be sure of covering it.""" + return self._frame_index()[1] + + @property + def shape(self) -> tuple: + return self._shape + + @property + def chunks(self) -> tuple: + return self._chunks + + @property + def blocks(self) -> tuple: + return self._blocks + + @property + def dtype(self) -> np.dtype: + return self._dtype + + def get_chunk(self, nchunk: int) -> bytes: + # Read the index once: `_offsets` and `_extents` each take the lock, and + # what they hand back does not change after the first read + offsets, extents = self._frame_index() + offset = int(offsets[nchunk]) + if offset < 0: + return self._special_chunk(offset) + data = self.read_range(offset, int(extents[nchunk])) + return data[: struct.unpack(" bytes: + """The bytes at [*offset*, *offset* + *size*) of the frame. + + The whole of the transport: everything else here is the frame format. + Fewer bytes may come back only at the end of the frame; anything else is + an error, since the caller has no way to ask for the rest. Must be safe + to call from several threads at once, which is what lets :ref:`Proxy` + overlap the fetches of one slice. + """ + + def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: + """The bytes of every ``(offset, size)`` in *spans*, in that order. + + One request each, unless a transport that can carry several ranges in one + overrides this and raises :attr:`max_ranges` to say how many. Nothing + else has to change for it: this is the only method a batching transport + needs, and everything that reads bytes goes through it. + """ + return [self.read_range(offset, size) for offset, size in spans] + + def wants_blocks(self, nchunk: int, nwanted: int) -> bool: + """Whether fetching *nwanted* blocks of a chunk beats fetching all of it. + + Answered without reading anything, so a chunk that says no costs exactly + what it costs today: the number of blocks a slice touches is geometry, + and an upper bound on the chunk's compressed size is already in hand from + the frame's offsets. See the thresholds at the top of this module. + """ + offsets, extents = self._frame_index() # once, rather than twice under the lock + if int(offsets[nchunk]) < 0: + return False # a run-length chunk has no bytes in the file to skip + if nwanted > self.blocks_per_chunk * BLOCK_MAX_FRACTION: + return False + return int(extents[nchunk]) >= BLOCK_MIN_CBYTES + + def chunk_layout(self, nchunk: int) -> tuple[bytes, np.ndarray, np.ndarray] | None: + """Read where the blocks of a chunk are: its header, bstarts and extents. + + One range read, of a size known in advance since every chunk holds the + same number of blocks. None for a chunk this cannot take apart, which the + header is what says: + + - a chunk of a single block is its own block, and a memcpyed one stores its + blocks raw with no ``bstarts`` at all; + - a chunk that is a run of one value is its header and that value, with no + blocks in the file: `blosc2.full` writes those at a real offset, unlike + the runs of zeros the frame keeps in the offsets themselves; + - a chunk compressed against a codec dictionary keeps it between + ``bstarts`` and the streams, and `_splice_chunk` would drop it while + leaving the flag that promises it; + - a variable-length-block chunk does not use the zero-length stream that + stands in for a block `_splice_chunk` does not have; + - a chunk without the extended header keeps its ``bstarts`` somewhere else + entirely, so reading them at byte 32 would be reading data. + + Each of those is then fetched whole, at the cost of the one header read + that found out. + """ + return self.chunk_layouts((nchunk,))[0] + + def chunk_layouts(self, nchunks: Sequence[int]) -> list: + """:meth:`chunk_layout` for several chunks, in as few requests as they fit. + + One request each unless the transport takes several ranges at once, and + none at all for a chunk already read: a fetch asks for layouts only where + blocks are missing, but the same chunk comes up again as a slice fills it + in. + """ + section = _CHUNK_HEADER_LEN + 4 * self.blocks_per_chunk + todo = [n for n in dict.fromkeys(nchunks) if n not in self._layouts] + offsets = self._offsets if todo else () # once, not once per span below + for batch in batched(todo, max(self.max_ranges, 1)): + spans = [(int(offsets[n]), section) for n in batch] + heads = self.read_ranges(spans) + for nchunk, head in zip(batch, heads, strict=True): + self._sections[nchunk] = head + self._layouts[nchunk] = self._parse_layout(head, section) + return [self._layouts[n] for n in nchunks] + + def _parse_layout(self, head: bytes, section: int): + """The layout a chunk's header section says it has, or None for no layout.""" + nblocks = self.blocks_per_chunk + if len(head) < section: + # A chunk clipped by the end of the frame has no block-offsets section: + # asked for before the header is parsed, so a short one never reaches + # the fields below rather than tripping over their offsets + return None + cbytes = struct.unpack("> 4) & 0x7 # a run of one value: header and value, no blocks + or head[31] & 0x01 # compressed against a dictionary + or head[30] & 0x01 # variable-length blocks + or not extended + # Whatever else a chunk may be, one too small to hold a block-offsets + # section has none: reading 32 bytes in would be reading the next chunk + or cbytes < section + ): + return None + bstarts = np.frombuffer(head[_CHUNK_HEADER_LEN:section], dtype=" list[tuple[int, int, tuple]]: + """The range reads that cover *nblocks*, near-adjacent ones merged. + + Each is ``(offset, size, members)`` in frame coordinates, where members + says which block each piece of the answer is, as + ``(nblock, offset within the read, size)``. + """ + _, bstarts, extents = self._layouts[nchunk] + base = int(self._offsets[nchunk]) + plan = [] + for nblock in sorted(nblocks, key=lambda n: bstarts[n]): + start, size = int(bstarts[nblock]), int(extents[nblock]) + if plan and start - (plan[-1][0] + plan[-1][1]) <= BLOCK_GAP: + offset, length, members = plan[-1] + plan[-1] = (offset, max(length, start + size - offset), (*members, (nblock, start, size))) + else: + plan.append((start, size, ((nblock, start, size),))) + return [ + (base + offset, length, tuple((n, s - offset, size) for n, s, size in members)) + for offset, length, members in plan + ] + + async def aget_chunk(self, nchunk: int) -> bytes: + """Same as :meth:`get_chunk`, but without blocking the caller's event loop. + + This is what makes :meth:`Proxy.afetch` worth using against an object + store, where a slice spanning many chunks is nearly all round-trip + latency. + + The fetch goes to a worker thread rather than being awaited directly. + Awaiting an async filesystem's coroutine looks like the obvious thing to + do and does not work: fsspec drives those on a private event loop of its + own, so a client created there and awaited here raises "got Future + attached to a different loop" (seen with s3fs). Its blocking API is the + supported way in, and it hands off to that same private loop, so the + thread parks on a queue rather than on a socket. + """ + offset = int(self._offsets[nchunk]) + if offset < 0: + return self._special_chunk(offset) + loop = asyncio.get_running_loop() + return await loop.run_in_executor(None, self.get_chunk, nchunk) + + def _special_chunk(self, offset: int) -> bytes: + """Rebuild a run-length chunk, which lives in its offset instead of the file.""" + kind = ((offset & 0xFFFFFFFFFFFFFFFF) >> 56) & 0x7 + nitems = self._chunksize // self._dtype.itemsize + if kind == 2: + data = np.full(nitems, np.nan, dtype=self._dtype) + else: + # A run of zeros (1); uninitialized chunks (4) have no defined + # content, and zeros is what reading them locally hands back too + data = np.zeros(nitems, dtype=self._dtype) + # The blocksize has to be the container's: left to choose, blosc2 takes + # the whole chunk, and the cache then rejects the chunk we hand it + return blosc2.compress2( + data, + typesize=self._dtype.itemsize, + blocksize=int(np.prod(self._blocks)) * self._dtype.itemsize, + ) + + +class FsspecNDSource(ByteRangeNDSource): + """A :ref:`ByteRangeNDSource` reading its frame through fsspec. + + This is what ``blosc2.open(url, lazy=True)`` builds; wrap it in a + :ref:`Proxy` by hand when the cache belongs at a path of your choosing + rather than inside ``cache_storage``:: + + src = blosc2.FsspecNDSource("s3://bucket/big.b2nd") + a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a") + + Parameters + ---------- + urlpath: str + The fsspec URL of the frame. + max_concurrency: int, optional + As in :ref:`ByteRangeNDSource`. + """ + + def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): + from blosc2.core import _import_fsspec + + fsspec = _import_fsspec(urlpath) + fs, path = fsspec.url_to_fs(urlpath) + if fs.isdir(path): + raise NotImplementedError( + f"{urlpath} is a directory (a sparse frame or a store), which cannot be read " + "chunk by chunk; open it with cache_storage= instead" + ) + self._fs, self._path = fs, path + # Identifies the remote bytes, so a cache built against them can tell it + # has gone stale -- and chunk offsets from a replaced frame are garbage. + # fsspec's own token, rather than a tuple of the metadata fields we guess + # a backend exposes: memory:// has no mtime, which left it size-only. + self.stamp = fs.ukey(path) + super().__init__(urlpath, max_concurrency) + + def read_range(self, offset: int, size: int) -> bytes: + return self._fs.cat_file(self._path, start=offset, end=offset + size) + + +def convert_dtype(dt: str | DTypeLike): + """ + Attempts to convert to blosc2.dtype (i.e. numpy dtype) + """ + if hasattr(dt, "as_numpy_dtype"): + dt = dt.as_numpy_dtype + try: + return np.dtype(dt) + except TypeError: # likely passed e.g. a torch.float64 + return np.dtype(str(dt).split(".")[1]) + except Exception as e: + raise TypeError(f"Could not parse dtype arg {dt}.") from e diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index ef646f558..6fb490800 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -228,7 +228,7 @@ def build(*args, **kwargs): @pytest.fixture def any_chunk_wants_blocks(monkeypatch): """Take the size threshold out of the way, so small test arrays use blocks.""" - monkeypatch.setattr(blosc2.proxy, "BLOCK_MIN_CBYTES", 0) + monkeypatch.setattr(blosc2.proxy_source, "BLOCK_MIN_CBYTES", 0) def _incompressible(shape, seed=0): diff --git a/tests/ndarray/test_proxy.py b/tests/ndarray/test_proxy.py index 3b611a84d..8e6d88657 100644 --- a/tests/ndarray/test_proxy.py +++ b/tests/ndarray/test_proxy.py @@ -375,3 +375,16 @@ def test_vlmeta_cannot_overwrite_proxy_state(): # Anything else still goes through proxy = blosc2.Proxy(source, vlmeta={"mine": "ok"}) assert proxy.vlmeta["mine"] == "ok" + + +def test_the_proxy_module_still_answers_for_the_source_names(): + # They live in `blosc2.proxy_source` now, so that the modules bound early in + # `blosc2/__init__` can reach them without dragging `proxy.py` in ahead of + # `schunk` and `indexing`. `blosc2.proxy.X` is where anything outside would + # look for them, so that keeps working: the imports there are not dead. + import blosc2.proxy + import blosc2.proxy_source + + for name in ("ProxySource", "ProxyNDSource", "ByteRangeNDSource", "FsspecNDSource"): + assert getattr(blosc2.proxy, name) is getattr(blosc2.proxy_source, name) + assert getattr(blosc2, name) is getattr(blosc2.proxy_source, name) diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index 1dc74cd35..a8eacc471 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -729,7 +729,7 @@ def test_handbuilt_proxy_rejects_a_stale_cache(tmp_path): @pytest.fixture def any_chunk_wants_blocks(monkeypatch): """Take the size threshold out of the way, so small test arrays exercise blocks.""" - monkeypatch.setattr(blosc2.proxy, "BLOCK_MIN_CBYTES", 0) + monkeypatch.setattr(blosc2.proxy_source, "BLOCK_MIN_CBYTES", 0) def _traffic(monkeypatch): @@ -783,7 +783,7 @@ def test_lazy_open_of_a_small_frame_never_reads_twice(monkeypatch): # A frame that fits in the first read is wholly in hand: the offsets chunk # is in those bytes too, so there is nothing left to ask for, then or later a = blosc2.arange(0, 100, dtype="i4", chunks=(10,)) - assert a.schunk.cbytes < blosc2.proxy._FRAME_PREFETCH + assert a.schunk.cbytes < blosc2.proxy_source._FRAME_PREFETCH url = _put("smallframe.b2nd", a) reads, _ = _traffic(monkeypatch) @@ -797,7 +797,7 @@ def test_lazy_open_of_a_small_frame_never_reads_twice(monkeypatch): def test_lazy_open_reads_a_header_that_did_not_fit(monkeypatch): # A metalayer big enough to push the header past the guess: the exact read # the format asks for happens after all, and nothing is misread - monkeypatch.setattr(blosc2.proxy, "_FRAME_PREFETCH", 256) + monkeypatch.setattr(blosc2.proxy_source, "_FRAME_PREFETCH", 256) data = np.arange(1000, dtype="i4") a = blosc2.asarray(data, chunks=(100,), meta={"big": {"pad": "x" * 4096}}) url = _put("bigheader.b2nd", a) @@ -935,7 +935,7 @@ def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path) def test_lazy_open_reads_an_index_that_did_not_fit(monkeypatch): # The same for the offsets chunk, which is bounded by the frame's own length # but capped in case a large trailer sits behind it - monkeypatch.setattr(blosc2.proxy, "_INDEX_PREFETCH", 16) + monkeypatch.setattr(blosc2.proxy_source, "_INDEX_PREFETCH", 16) data, a = _incompressible((600, 600), (30, 600), (30, 600)) url = _put("bigindex.b2nd", a) reads, _ = _traffic(monkeypatch) @@ -952,7 +952,7 @@ def test_lazy_fetches_only_touched_blocks(monkeypatch): # Chunks big enough to be worth taking apart, at the real threshold data, a = _incompressible((600, 600), (300, 600), (30, 600)) cbytes = a.schunk.cbytes // a.schunk.nchunks - assert cbytes > blosc2.proxy.BLOCK_MIN_CBYTES + assert cbytes > blosc2.proxy_source.BLOCK_MIN_CBYTES p = blosc2.open(_put("blocks.b2nd", a), lazy=True) reads, chunks = _traffic(monkeypatch) # after the open, which reads the header diff --git a/tests/test_fsspec_s3.py b/tests/test_fsspec_s3.py index 001cef1d9..f4f8d583a 100644 --- a/tests/test_fsspec_s3.py +++ b/tests/test_fsspec_s3.py @@ -122,7 +122,7 @@ def blocky(s3_endpoint): """An array whose chunks are big enough to be worth reading block by block.""" data = np.random.default_rng(0).random((600, 600)) a = blosc2.asarray(data, chunks=(300, 600), blocks=(30, 600)) - assert a.schunk.cbytes / a.schunk.nchunks > blosc2.proxy.BLOCK_MIN_CBYTES + assert a.schunk.cbytes / a.schunk.nchunks > blosc2.proxy_source.BLOCK_MIN_CBYTES urlpath = f"s3://{BUCKET}/blocky.b2nd" a.save(urlpath, mode="w") return urlpath, data From 7254129d897ac001293202e78d44aa3e238e8e48 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 10:05:21 +0200 Subject: [PATCH 28/32] Leave a cache a proxy fits exactly as it found it Opening a proxy over a cache it already fits wrote to that cache twice over: the stamp went in whether or not it was already what was there, and the kept index started every run as `None`, so the first fetch wrote the offsets back over an identical copy of themselves. Writing a vlmeta entry rewrites the container, which for a frame of 100_000 chunks is 800 KB of offsets rebuilt, serialized and written to say nothing at all. Both now write only what moves. And the constructor's arguments are refused before its cache is touched. Adopting a cache whose stamp has moved on empties it as its first act, and the check for reserved vlmeta names came after that -- so a call that raised had already wiped the bitmaps and the index it was refusing to run. Co-Authored-By: Claude Opus 5 --- src/blosc2/proxy.py | 42 +++++++++++++++++++++++++--------------- tests/test_fsspec.py | 46 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+), 16 deletions(-) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index f6cfd57b0..13bf5c742 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -118,6 +118,19 @@ def __init__( self._cache = kwargs.pop("_cache", None) vlmeta = kwargs.pop("vlmeta", None) caterva2_env = kwargs.pop("caterva2_env", False) + # Before anything is built or emptied: a call that is going to be refused + # must leave the cache at `urlpath` exactly as it found it, and adopting + # one whose stamp has moved on empties it as its first act + reserved = sorted(_RESERVED_VLMETA & set(vlmeta or ())) + if reserved: + # Writing these would hand the proxy a bitmap or an identity it never + # earned: a caller's `proxy-fetched` makes it skip chunks it has not + # fetched, and a caller's `proxy-stamp` makes a good cache fail its + # identity check (or a stale one pass it) + raise ValueError( + f"{', '.join(reserved)} {'is' if len(reserved) == 1 else 'are'} reserved " + f"for the proxy's own bookkeeping and cannot be set through vlmeta" + ) if self._cache is None and mode == "a" and urlpath is not None and os.path.exists(urlpath): # Reuse the cache left by an earlier run: whatever was fetched then is @@ -188,19 +201,8 @@ def __init__( if self.urlpath is None: self.urlpath = getattr(self._schunk_cache, "urlpath", None) self._fetched = self._adopt_cache(fresh, self._schunk_cache.nchunks) - if vlmeta: - reserved = sorted(_RESERVED_VLMETA & set(vlmeta)) - if reserved: - # Writing these would hand the proxy a bitmap or an identity it - # never earned: a caller's `proxy-fetched` makes it skip chunks it - # has not fetched, and a caller's `proxy-stamp` makes a good cache - # fail its identity check (or a stale one pass it) - raise ValueError( - f"{', '.join(reserved)} {'is' if len(reserved) == 1 else 'are'} reserved " - f"for the proxy's own bookkeeping and cannot be set through vlmeta" - ) - for key in vlmeta: - self._schunk_cache.vlmeta[key] = vlmeta[key] + for key in vlmeta or (): + self._schunk_cache.vlmeta[key] = vlmeta[key] def __enter__(self) -> "Proxy": """Enter a context manager and return this proxy.""" @@ -235,10 +237,12 @@ def _adopt_cache(self, fresh: bool, nchunks: int) -> bytearray: # are there, so a run that stopped in between would leave a cache # claiming the new bytes and holding the old ones self._forget_fetched(nchunks) - if stamp is not None and writable: + if stamp is not None and writable and stored != stamp: # Not into a cache opened read-only, which `blosc2.open(path, mode="r")` # hands over for a persisted proxy: nothing may be written there, and a - # proxy over one stays observational anyway + # proxy over one stays observational anyway. Nor when it is already + # what is there: writing a vlmeta entry rewrites the cache file, and + # merely opening a proxy over a cache it fits should not touch it. self._schunk_cache.vlmeta["proxy-stamp"] = stamp if fresh or replaced: return bytearray((nchunks * self._blocks_per_chunk + 7) // 8) @@ -248,7 +252,13 @@ def _adopt_cache(self, fresh: bool, nchunks: int) -> bytearray: # cache was come by: a `_cache=` handed in never passed `_reopen_cache`. adopt = getattr(self.src, "adopt_index", None) if adopt is not None and stamp is not None and stored == stamp: - adopt(self._schunk_cache.vlmeta.get("proxy-index")) + index = self._schunk_cache.vlmeta.get("proxy-index") + adopt(index) + # What is on disk, so that a run which reads nothing new out of the + # source writes it back over itself for nothing: the offsets are the + # bulk of a large frame's index, and this is the first fetch's + # comparison, not just the second's + self._saved_index = index return self._load_fetched(nchunks) def _forget_fetched(self, nchunks: int) -> None: diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index a8eacc471..2b4655748 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -908,6 +908,52 @@ def test_a_kept_index_is_dropped_when_the_bytes_behind_it_were_replaced(tmp_path assert np.array_equal(p[300:330, 0:10], other[300:330, 0:10]) +def test_a_cache_a_proxy_fits_is_not_written_to_by_opening_it(tmp_path): + # Writing a vlmeta entry rewrites the cache file, so a proxy that adopts a + # cache and reads nothing new out of the source must write nothing back: + # neither the stamp it just checked nor the index it just read + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("untouched.b2nd", a) + cache = str(tmp_path / "untouched-cache.b2nd") + item = (slice(0, 30), slice(None)) + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch(item) + # The bytes and when they were last written: a stamp written over the same + # stamp leaves the first alone and moves the second, and both are the cost + before = (pathlib.Path(cache).read_bytes(), os.stat(cache).st_mtime_ns) + + p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") + assert (pathlib.Path(cache).read_bytes(), os.stat(cache).st_mtime_ns) == before + # What the cache holds is what the proxy would write back, so the first fetch + # of this run has something to compare against rather than writing to find out + assert p._saved_index == p._schunk_cache.vlmeta["proxy-index"] + p.fetch(item) # already cached, in full + assert (pathlib.Path(cache).read_bytes(), os.stat(cache).st_mtime_ns) == before + assert np.array_equal(p[...], data) + + +def test_a_refused_vlmeta_leaves_the_cache_as_it_was(tmp_path): + # Adopting a cache whose bytes were replaced empties it, and a constructor + # that goes on to refuse its arguments must not have done that: a call that + # reports failure has to leave the cache for the next one to use + data, a = _incompressible((600, 600), (300, 600), (30, 600)) + url = _put("refusedvlmeta.b2nd", a) + cache = str(tmp_path / "refusedvlmeta-cache.b2nd") + blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="w").fetch((slice(0, 30), slice(None))) + + other = np.zeros((600, 600)) + other[300:] = np.random.default_rng(1).random((300, 600)) + _put("refusedvlmeta.b2nd", blosc2.asarray(other, chunks=(300, 600), blocks=(30, 600))) + before = pathlib.Path(cache).read_bytes() + + kwargs = {} + blosc2.schunk._set_default_dparams(kwargs) + holder = blosc2.blosc2_ext.open(cache, "a", 0, **kwargs) + with pytest.raises(ValueError, match="reserved"): + blosc2.Proxy(blosc2.FsspecNDSource(url), _cache=holder, vlmeta={"proxy-fetched": b"nonsense"}) + del holder + assert pathlib.Path(cache).read_bytes() == before + + def test_a_cache_that_holds_the_slice_asks_for_no_offsets(monkeypatch, tmp_path): # What the deferral is for: a later run over a cache that already covers the # slice fetches nothing, and so has no use for where the chunks are either From 410f89f0ce181906bf43da1864355fd593cf8cfb Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 10:05:40 +0200 Subject: [PATCH 29/32] Read a .b2z leaf only where the zip kept it whole A zip store is read where its members lie: `blosc2_ext.open(path, offset=...)` at each member's data offset. That is only a frame for a member the archive *stored*, and an archive repacked by another tool may well have deflated them -- `member_window` learned to say so, but the three places that open a member at its offset still went by the suffix and would have read deflate output as a frame. So the fact goes where the invariant lives: a deflated member is not mapped as a leaf at all, which is the same answer `member_window` gives for one. A store whose own super-chunk was deflated says that, rather than handing compressed bytes to a frame reader and failing somewhere inside it. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 4 +++- src/blosc2/dict_store.py | 20 ++++++++++++++++-- tests/test_dict_store.py | 45 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+), 3 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 9b65e6b25..74e196f8a 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -90,7 +90,9 @@ XXX version-specific blurb XXX container leaf from it, so a `Proxy` over `@public/tree.b2z/leaf` reads blocks exactly as it does over a `.b2nd` (2.8x on a point read, 20x fewer bytes). None where there is no window: a directory-backed store, an embedded leaf, a - `C2Array` reference. + `C2Array` reference, and a member some other tool repacked *compressed* -- + which is now also left out of the store's keys rather than read as the deflate + output it is, and said plainly when it is the store's own super-chunk. * `blosc2.Proxy(src, urlpath=..., mode="a")` now adopts the cache left by an earlier run instead of failing on the existing file, so a proxy's cache can diff --git a/src/blosc2/dict_store.py b/src/blosc2/dict_store.py index 4435de09a..4f0931dda 100644 --- a/src/blosc2/dict_store.py +++ b/src/blosc2/dict_store.py @@ -246,6 +246,14 @@ def _init_read_mode(self, dparams: blosc2.DParams | None = None): self.offsets = self._get_zip_offsets() if "embed.b2e" not in self.offsets: raise FileNotFoundError("Embed file embed.b2e not found in store.") + if not self.offsets["embed.b2e"]["stored"]: + # The store is read where its members lie, so a repack that + # deflated them left no store to read: say so, rather than hand + # compressed bytes to a frame reader and fail somewhere inside it + raise ValueError( + f"{self.b2z_path} holds a compressed embed.b2e; a .b2z is read in place, so its " + f"members must be stored uncompressed. Unzip it and open the directory instead." + ) estore_offset = self.offsets["embed.b2e"]["offset"] schunk = blosc2.blosc2_ext.open( self.b2z_path, @@ -431,10 +439,18 @@ def _update_map_tree_from_offsets(self): leaf suffixes. Trusting those suffixes avoids opening every member just to classify it, which is especially important for compact CTable stores with many columns. + + A suffix is worth trusting only where the member is *stored*, which is + how this writes one: every read of a leaf here opens the archive at the + member's data offset, and what lies there for a deflated member is + compressed bytes rather than the frame its name promises. A `.b2z` + repacked by another tool may hold such members, and they are left + unmapped -- there is no frame at that offset to serve, which is the same + answer :meth:`member_window` gives for one. """ external_exts = {".b2nd", ".b2f", ".b2b"} - for filepath in self.offsets: - if filepath == "embed.b2e": + for filepath, window in self.offsets.items(): + if filepath == "embed.b2e" or not window["stored"]: continue if os.path.splitext(filepath)[1] in external_exts or self._probe_external_leaf_offset(filepath): self.map_tree[self._logical_key_from_relpath(filepath)] = filepath diff --git a/tests/test_dict_store.py b/tests/test_dict_store.py index 7405c5dd5..99b636dbf 100644 --- a/tests/test_dict_store.py +++ b/tests/test_dict_store.py @@ -894,3 +894,48 @@ def test_member_window_of_a_tree_group(tmp_path): tstore = blosc2.open(path) assert tstore.member_window("/g/leaf") is not None assert tstore.member_window("/g") is None # a group is not a leaf + + +def _repack(source: str, target: str, deflate: set[str]) -> str: + """Rewrite a .b2z with the named members compressed, as another tool might.""" + with zipfile.ZipFile(source) as src, zipfile.ZipFile(target, "w") as dst: + for info in src.infolist(): + kind = zipfile.ZIP_DEFLATED if info.filename in deflate else zipfile.ZIP_STORED + dst.writestr(info.filename, src.read(info.filename), compress_type=kind) + return target + + +def test_a_deflated_member_is_not_read_as_a_frame(tmp_path): + """A .b2z is read where its members lie, so a deflated one is no leaf. + + Every read of an external leaf opens the archive at the member's data + offset, and what lies there for a deflated member is deflate output rather + than the frame its suffix promises. A store written here keeps them stored; + one repacked by another tool need not, and then the leaf is not there to be + served -- which is what `member_window` answers, and what everything that + opens a member at its offset has to agree with. + """ + path = str(tmp_path / "stored.b2z") + with blosc2.TreeStore(path, mode="w") as tstore: + tstore["/a"] = np.arange(1000, dtype="i4") + tstore["/b"] = np.arange(1000, dtype="i4") + + mixed = _repack(path, str(tmp_path / "mixed.b2z"), deflate={"a.b2nd"}) + store = blosc2.open(mixed) + assert "/a" not in store # nothing at that offset for a frame reader + assert store.member_window("/a") is None + assert "/b" in store # ... and the stored leaf beside it is untouched + assert np.array_equal(store["/b"][:], np.arange(1000, dtype="i4")) + + +def test_a_deflated_store_says_so_rather_than_misreading_it(tmp_path): + # The store's own super-chunk is read at its offset too, so a repack that + # compressed it left nothing to open: say that, rather than hand deflate + # output to a frame reader and fail somewhere inside it + path = str(tmp_path / "wholestore.b2z") + with blosc2.TreeStore(path, mode="w") as tstore: + tstore["/a"] = np.arange(1000, dtype="i4") + + deflated = _repack(path, str(tmp_path / "deflated.b2z"), deflate={"a.b2nd", "embed.b2e"}) + with pytest.raises(ValueError, match=r"compressed embed\.b2e"): + blosc2.open(deflated) From 92b02b310187eca4ead6550ba5c4187900e40278 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 10:05:57 +0200 Subject: [PATCH 30/32] Keep the layouts of a cache whose offsets do not fit, and keep them once Two things about the index a source hands its cache. An offsets blob that does not fit the frame was dropping the per-chunk layouts saved beside it, though those are checked on their own and keyed by chunk number: a cache that lost its offsets was paying a header read per partly filled chunk as well. And the bytes a layout was read as are no longer kept beside the parsed one. `_sections` grew for the life of the source -- 32 + 4 bytes per block for every chunk ever laid out, including the ones with no layout to give back -- to be read back a handful of chunks at a time. The header and the `bstarts` are exactly what the layout already holds, so `index_state` rebuilds the section from it. Co-Authored-By: Claude Opus 5 --- src/blosc2/proxy_source.py | 43 +++++++++++++++++++++++++------------- tests/test_fsspec.py | 4 +++- 2 files changed, 31 insertions(+), 16 deletions(-) diff --git a/src/blosc2/proxy_source.py b/src/blosc2/proxy_source.py index 8fa69f6a9..2949a96f4 100644 --- a/src/blosc2/proxy_source.py +++ b/src/blosc2/proxy_source.py @@ -518,11 +518,10 @@ def __init__(self, urlpath: str, max_concurrency: int = REMOTE_MAX_CONCURRENCY): if all(self._blocks) else 1 ) - # Layouts are memoized for the life of the source, and `_sections` keeps - # the bytes each was read as, so a `Proxy` can hand them to its cache and - # a later run start from them instead of reading them again. + # Layouts are memoized for the life of the source; `index_state` hands + # them back as the bytes they were read as, so a `Proxy` can keep them in + # its cache and a later run start from them instead of reading again. self._layouts = {} - self._sections = {} def index_state(self, keep: Sequence[int] = ()) -> dict: """Where things are, as the bytes they were read as, for a cache to keep. @@ -533,9 +532,13 @@ def index_state(self, keep: Sequence[int] = ()) -> dict: so a chunk that is complete is never asked about again, and one that is empty was never read. - Kept as read rather than as parsed, so that what comes back goes through - :meth:`_parse_layout` exactly as a fresh read does -- one parser, not a - second one that could disagree with it. + Handed back as the bytes a layout was read as rather than as parsed, so + that what comes back goes through :meth:`_parse_layout` exactly as a + fresh read does -- one parser, not a second one that could disagree with + it. The bytes are rebuilt from the layout instead of being kept beside + it: they are the header and the ``bstarts`` it already holds, and a + second copy of every chunk ever laid out would grow for the life of the + source to be read back a handful of chunks at a time. """ offsets = self._index[0] if self._index is not None else None return { @@ -543,9 +546,19 @@ def index_state(self, keep: Sequence[int] = ()) -> dict: # Little-endian whatever the host is: a cache directory outlives the # machine that filled it, and a stamp cannot tell a byte order "offsets": b"" if offsets is None else offsets.astype(" bytes: + """A chunk's header section, as the read that found its layout saw it. + + A chunk with no layout has none to give back, and none is wanted: a + `Proxy` keeps layouts for the chunks it holds some blocks of, and a chunk + that cannot be taken apart was fetched whole. + """ + head, bstarts, _ = self._layouts[nchunk] + return head + bstarts.astype(" None: """Take up what an earlier run left behind in :meth:`index_state`. @@ -567,14 +580,15 @@ def adopt_index(self, state: dict | None) -> None: if offsets: nchunks = math.prod(math.ceil(s / c) for s, c in zip(self._shape, self._chunks, strict=True)) array = np.frombuffer(offsets, dtype=" tuple[np.ndarray, np.ndarray]: @@ -704,7 +718,6 @@ def chunk_layouts(self, nchunks: Sequence[int]) -> list: spans = [(int(offsets[n]), section) for n in batch] heads = self.read_ranges(spans) for nchunk, head in zip(batch, heads, strict=True): - self._sections[nchunk] = head self._layouts[nchunk] = self._parse_layout(head, section) return [self._layouts[n] for n in nchunks] diff --git a/tests/test_fsspec.py b/tests/test_fsspec.py index 2b4655748..b799d24c1 100644 --- a/tests/test_fsspec.py +++ b/tests/test_fsspec.py @@ -876,7 +876,9 @@ def test_a_kept_index_of_the_wrong_shape_is_dropped(tmp_path, monkeypatch): reads, _ = _traffic(monkeypatch) p = blosc2.Proxy(blosc2.FsspecNDSource(url), urlpath=cache, mode="a") assert np.array_equal(p[60:90, 0:10], data[60:90, 0:10]) - assert len(reads) > 2 # the offsets were read again + # The offsets were read again -- and only they: the layout of the chunk half + # held is checked on its own, so bad offsets do not throw it away as well + assert len(reads) == 3 # the header, the offsets, and the blocks wanted assert np.array_equal(p[...], data) From 8fa4b8f759430012b1acaa03aa3d98561b0ab751 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 10:06:14 +0200 Subject: [PATCH 31/32] Let the source batch the layout reads it already batches `Proxy._chunk_layouts` split on `max_ranges` to either batch the reads itself or map them one at a time, returning a list from one branch and a generator from the other -- but `chunk_layouts` batches internally and works fine for a batch of one, and `chunk_layout` is defined as a call to it. What is the proxy's own business is the concurrency: the batches past the first are round trips like any other. A source that serves layouts one at a time and nothing else keeps its branch. Co-Authored-By: Claude Opus 5 --- src/blosc2/proxy.py | 27 +++++++++++++-------------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 13bf5c742..571b29314 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -634,20 +634,19 @@ def fetch(task): return self._cache - def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None): - """Where the blocks of every one of *nchunks* are, in as few requests as fit.""" - batch = max(getattr(self.src, "max_ranges", 1), 1) - if batch > 1: - # The source reads a batch of them in one request, but only so many - # per request: the batches past the first are round trips like any - # other, so they are overlapped rather than run one after the next - tasks = list(batched(nchunks, batch)) - return [ - layout - for answers in self._run(self.src.chunk_layouts, tasks, max_concurrency) - for layout in answers - ] - return self._run(self.src.chunk_layout, nchunks, max_concurrency) + def _chunk_layouts(self, nchunks: list[int], max_concurrency: int | None) -> list: + """Where the blocks of every one of *nchunks* are, in as few requests as fit. + + How many go in one request is the source's business (`chunk_layouts` is + what batches them, down to a batch of one); what is this proxy's is that + the requests past the first are round trips like any other, and so are + overlapped rather than run one after the next. + """ + layouts = getattr(self.src, "chunk_layouts", None) + if layouts is None: # a source that only serves them one at a time + return list(self._run(self.src.chunk_layout, nchunks, max_concurrency)) + tasks = list(batched(nchunks, max(getattr(self.src, "max_ranges", 1), 1))) + return [layout for answers in self._run(layouts, tasks, max_concurrency) for layout in answers] def _write_blocks(self, nchunk: int, payloads: dict[int, bytes], header: bytes) -> None: """Put the blocks just fetched into the cache, keeping those already there. From f81016b16356acef1e4a359ecdc47926e88efa64 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Thu, 20 Aug 2026 10:07:12 +0200 Subject: [PATCH 32/32] Read whole chunks when a subscriber stops answering ranges A refusal after the block source exists was fatal to the fetch: a 503, a 416, or a dataset the subscriber has started computing came out of `Proxy.fetch` as a private exception nothing catches, though the same 503 at probe time is deliberately survivable. The refusal now has a type of its own -- `blosc2.proxy_source.NotRanged`, a `ValueError`, with `PartsMissing` under it -- and it is answered rather than raised: the proxy fetches whatever is still missing as whole chunks, which every dataset can be read as. A refusal that says the dataset is no longer served from a file retires the block source with it, so the array goes back to `get_chunk` for good; one that says the server was busy leaves it alone, since the next request may well be answered. Both read the same `_is_transient`, rather than each writing the rule out. Whether blocks are worth asking for at all is now a question the source answers -- `serves_blocks` -- rather than one the proxy guesses at by looking for methods. Every `C2Array` has those methods, so a computed dataset, a streamed one and one below the size threshold all got a per-block bitmap they would never fill, the block path on every fetch, and, for a dataset whose `api/info` carries no geometry, a `KeyError` where a proxy rebuilt over its own cache used to open. Two answers that were being read wrongly: * A part that ends where the frame does is what `read_range` promises may come back, and refusing it wrote a frame shorter than the prefetch off as streamed. Parts now carry the frame length their `Content-Range` gives, and a short read is accepted only against it. * A multipart body is cut where each part's own `Content-Range` says it ends, not at the next thing that looks like a boundary or a trailing CRLF. Compressed payloads hold arbitrary bytes, those among them. And an answer that does not carry its parts is retried a range at a time before batching is given up on, which one truncated answer used to end for the life of the process -- 0.136 s against 1.530 s for 32 spans. Co-Authored-By: Claude Opus 5 --- RELEASE_NOTES.md | 5 +- doc/reference/byterangendsource.rst | 16 ++ src/blosc2/c2array.py | 234 ++++++++++++++++++--------- src/blosc2/proxy.py | 29 +++- src/blosc2/proxy_source.py | 68 +++++++- tests/ndarray/test_c2array_blocks.py | 191 +++++++++++++++++++++- 6 files changed, 456 insertions(+), 87 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 74e196f8a..d17e4a92e 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -56,7 +56,10 @@ XXX version-specific blurb XXX request to find out, and is not asked again -- unless the subscriber could not say, which a busy or unreachable one cannot: a 5xx or a connection that failed is asked again on the next fetch rather than written off, since neither - downloaded anything to find out. `blosc2.ByteRangeNDSource` is + downloaded anything to find out. The same holds once blocks are being read: a + dataset that stops being served from a file, or a subscriber too busy to serve + it, costs the granularity and not the fetch — whatever is still missing comes + as whole chunks, which every dataset can be read as. `blosc2.ByteRangeNDSource` is the frame reader `FsspecNDSource` and the new `C2NDSource` share: subclass it with a `read_range(offset, size)` to give any transport the same treatment. Opening a remote frame through either of them now costs two requests instead diff --git a/doc/reference/byterangendsource.rst b/doc/reference/byterangendsource.rst index af57c43f2..82970ff78 100644 --- a/doc/reference/byterangendsource.rst +++ b/doc/reference/byterangendsource.rst @@ -17,3 +17,19 @@ and :ref:`ProxySource`. :members: :exclude-members: all, any, max, mean, min, prod, std, sum, var :member-order: groupwise + +When a range read is refused +---------------------------- + +A transport that reads byte ranges may be answered with something other than the +bytes asked for: a subscriber that now streams the dataset, a server that is too +busy to serve it, a body that cannot be taken apart. Those raise +``blosc2.proxy_source.NotRanged``, which a :ref:`Proxy` catches for itself -- +whatever the fetch is still missing comes as whole chunks -- and which a caller +reading ranges directly can catch by name. + +.. autoclass:: blosc2.proxy_source.NotRanged + :members: + +.. autoclass:: blosc2.proxy_source.PartsMissing + :members: diff --git a/src/blosc2/c2array.py b/src/blosc2/c2array.py index fbf85b4b6..4a160980c 100644 --- a/src/blosc2/c2array.py +++ b/src/blosc2/c2array.py @@ -23,7 +23,13 @@ import blosc2 from blosc2.b2objects import encode_b2object_payload, make_b2object_carrier, write_b2object_payload from blosc2.info import InfoReporter, format_nbytes_info -from blosc2.proxy_source import REMOTE_MAX_CONCURRENCY, ByteRangeNDSource +from blosc2.proxy_source import ( + REMOTE_MAX_CONCURRENCY, + ByteRangeNDSource, + NotRanged, + PartsMissing, + _is_transient, +) _subscriber_data = { "urlbase": os.environ.get("BLOSC_C2URLBASE"), @@ -276,48 +282,51 @@ def slice_to_string(slice_): """ -class _NotRanged(Exception): - """The subscriber answered a range request with something other than a 206.""" - - def __init__(self, message: str, status: int | None = None): - super().__init__(message) - self.status = status - - @property - def transient(self) -> bool: - """Whether asking again could be answered differently. - - A 200 is the dataset itself, streamed, and no amount of asking again will - make it a file; a server that is busy or broken says nothing at all about - how the dataset is served, and costs no download to ask twice. - """ - return self.status is not None and (self.status >= 500 or self.status == 429) +MULTIPART_STRIKES = 2 +"""How many answers that did not carry their parts end multi-range requests. +Batching is worth an order of magnitude (see :meth:`C2NDSource.read_ranges`), so +one truncated or unreadable answer is worth retrying a range at a time and +asking again; a server that cannot do it says so twice. +""" -class _PartsMissing(Exception): - """A multi-range answer did not carry all the bytes that were asked for.""" +def _content_range(value: str) -> tuple[int, int, int | None]: + """A `Content-Range: bytes start-end/total` header, as (start, nbytes, total). -def _content_range(value: str) -> int: - """Where a `Content-Range: bytes start-end/total` header says its part starts. + The length comes from the header rather than from the bytes that follow it, + which is what lets a multipart body be cut where its parts really end rather + than at whatever looks like a boundary inside a compressed payload. Malformed answers are refused rather than guessed at: `bytes */1234` for a range that could not be satisfied has no start to read, and a part written to no shape at all cannot be placed in the frame. """ try: - return int(value.split()[1].split("-")[0]) + span, _, total = value.split()[1].partition("/") + start, _, end = span.partition("-") + nbytes = int(end) - int(start) + 1 except (IndexError, ValueError) as exc: - raise _NotRanged(f"a 206 carried an unreadable Content-Range: {value!r}") from exc + raise NotRanged(f"a 206 carried an unreadable Content-Range: {value!r}") from exc + if nbytes < 0: + raise NotRanged(f"a 206 carried a Content-Range that ends before it starts: {value!r}") + # `*` for a server that will not say how long the whole is, which is allowed + # and which only costs the tolerance for a read clipped by the end of the frame + return int(start), nbytes, int(total) if total.isdigit() else None -def _byteranges(response) -> list[tuple[int, bytes]]: - """The parts of a 206, as (where each starts in the frame, its bytes). +def _byteranges(response) -> list[tuple[int, bytes, int | None]]: + """The parts of a 206, as (where each starts, its bytes, how long the frame is). One part for an ordinary 206, several for a `multipart/byteranges` body: a boundary line, the part's own headers, a blank line and its bytes, over and over, ending in the boundary followed by two dashes. `email` would parse it, at the price of decoding a megabyte of compressed data as text. + + Each part is cut to the length its own `Content-Range` gives, and the next + boundary looked for after it: compressed payloads hold arbitrary bytes, the + boundary and a trailing CRLF among them, so a body split on either would slice + parts short wherever the data happened to spell one. """ content_type = response.headers.get("content-type", "") if "multipart/byteranges" not in content_type: @@ -326,35 +335,52 @@ def _byteranges(response) -> list[tuple[int, bytes]]: # otherwise have its bytes placed wherever they were asked for single = response.headers.get("content-range") if single is None: - raise _NotRanged("a 206 arrived without a Content-Range to place it by") - return [(_content_range(single), response.content)] + raise NotRanged("a 206 arrived without a Content-Range to place it by") + start, _, total = _content_range(single) + return [(start, response.content, total)] _, sep, boundary = content_type.partition("boundary=") if not sep: - raise _NotRanged(f"a multipart answer named no boundary: {content_type!r}") - boundary = boundary.strip().strip('"').encode() + raise NotRanged(f"a multipart answer named no boundary: {content_type!r}") + marker = b"--" + boundary.strip().strip('"').encode() + body = response.content parts = [] - for chunk in response.content.split(b"--" + boundary): - head, sep, body = chunk.partition(b"\r\n\r\n") - if not sep: # the preamble before the first boundary, and the closing -- - continue - for line in head.split(b"\r\n"): - name, _, value = line.partition(b":") - if name.strip().lower() == b"content-range": - # The body ends with the CRLF that belongs to the next boundary - parts.append((_content_range(value.decode()), body[: body.rfind(b"\r\n")])) + pos = body.find(marker) + while pos != -1: + pos += len(marker) + if body[pos : pos + 2] == b"--": + break # the closing boundary, and nothing after it belongs to a part + head_end = body.find(b"\r\n\r\n", pos) + if head_end == -1: + raise NotRanged("a multipart part arrived without a blank line to end its headers") + placed = None + for line in body[pos:head_end].split(b"\r\n"): + name, sep, value = line.partition(b":") + if sep and name.strip().lower() == b"content-range": + placed = _content_range(value.decode()) break + if placed is None: + raise NotRanged("a multipart part arrived without a Content-Range to place it by") + start, nbytes, total = placed + data = body[head_end + 4 : head_end + 4 + nbytes] + parts.append((start, data, total)) + pos = body.find(marker, head_end + 4 + nbytes) return parts -def _span_of(parts: list[tuple[int, bytes]], offset: int, size: int, url: str) -> bytes: +def _span_of(parts: list[tuple[int, bytes, int | None]], offset: int, size: int, url: str) -> bytes: """The bytes of one requested span, out of whichever part covers it.""" - for start, data in parts: + for start, data, total in parts: + if start > offset: + continue # The whole span, not just its first byte: a part that begins inside it # and ends early would otherwise slice short, and a short block payload - # is spliced against a `bstarts` that promises the full length - if start <= offset and offset + size <= start + len(data): + # is spliced against a `bstarts` that promises the full length. Unless + # it ends where the frame does, which `read_range` allows and an open of + # a frame shorter than the prefetch relies on. + end = start + len(data) + if offset + size <= end or (end == total and offset < end): return data[offset - start : offset - start + size] - raise _PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") + raise PartsMissing(f"{url} answered without the bytes at {offset}, which were asked for") class C2NDSource(ByteRangeNDSource): @@ -380,6 +406,8 @@ class C2NDSource(ByteRangeNDSource): def __init__(self, array: C2Array, max_concurrency: int = REMOTE_MAX_CONCURRENCY): self._url = _sub_url(array.urlbase, f"api/fetch/{array.path}") self._auth_token = array.auth_token + # Answers that did not carry their parts, in a row; see `read_ranges` + self._misses = 0 super().__init__(self._url, max_concurrency) # A `Proxy` mixes the two: the block grid and the fetched bitmap come from # the array's `api/info`, while the header sections and `bstarts` come from @@ -412,16 +440,23 @@ def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: spans and merges the ones that touch, and answers a single 206 when they all merge into one. So the answer is taken apart by what each part says it holds, and each span read out of the part that covers it, rather than - by trusting the order. A server that answers a multi-range request with - bytes that do not cover the whole of it is not one to ask again: this - keeps to a range per request from there on. + by trusting the order. An answer that does not carry the whole of what + was asked for is retried a range at a time, and only a server that does + that `MULTIPART_STRIKES` times is written off as unable to batch: an + order of magnitude for the rest of the process is too much to pay for one + truncated answer. """ spans = list(spans) if len(spans) > 1 and self.max_ranges > 1: try: - return self._get(spans) - except _PartsMissing: - self.max_ranges = 1 + answers = self._get(spans) + except PartsMissing: + self._misses += 1 + if self._misses >= MULTIPART_STRIKES: + self.max_ranges = 1 + else: + self._misses = 0 # what counts is a server that cannot do this + return answers return [self.read_range(*span) for span in spans] def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: @@ -432,7 +467,7 @@ def _get(self, spans: list[tuple[int, int]]) -> list[bytes]: # Whatever this is, it is not the bytes that were asked for: a 200 # carries the whole dataset, which is the download this exists to # avoid, so leave the body unread on the socket - raise _NotRanged( + raise NotRanged( f"{self._url} answered {response.status_code} to a Range request", response.status_code, ) @@ -742,6 +777,31 @@ def blocks_per_chunk(self) -> int: return 1 return math.prod(math.ceil(c / b) for c, b in zip(self.chunks, blocks, strict=True)) + @property + def serves_blocks(self) -> bool: + """Whether blocks are worth asking this dataset for, as far as info can say. + + What `api/info` already carries, and no request of its own: a dataset the + subscriber *computes* reports an expression where a stored one reports a + geometry, and a frame of small chunks would never have one taken apart -- + blosc2 declines to split a chunk below ``BLOCK_MIN_CBYTES``, so the block + path would end in whole chunks anyway, by the longer road. + + False is the whole answer; True is only that it is worth one request to + find out, which :meth:`block_source` spends. A :ref:`Proxy` reads this + when it is built, to decide whether its cache records blocks or chunks, + so it must cost nothing and must not depend on what has been fetched. + """ + if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): + return False + try: + nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) + return bool(nchunks) and self.cbytes / nchunks >= blosc2.proxy_source.BLOCK_MIN_CBYTES + except (KeyError, TypeError, ValueError, ZeroDivisionError): + # An `api/info` without the fields these read describes a dataset + # whole chunks work for, as they do for every dataset there is + return False + def block_source(self) -> C2NDSource | None: """The frame reader behind the block methods, or None if there is none. @@ -765,36 +825,29 @@ def _open_block_source(self): good; `_UNTRIED` for a subscriber that could not say, which is not. """ httpx = _httpx() - # `api/info` rules out a dataset the subscriber computes for nothing: a - # stored one reports its geometry where a lazy expression reports - # `expression` and `operands` - if not all(key in self.meta for key in ("chunks", "blocks", "schunk")): + # What `api/info` alone rules out -- a dataset the subscriber computes, a + # frame of chunks too small to take apart -- costs no request to find out + if not self.serves_blocks: return None # Whether a dataset that reports a geometry is *served* from a file is # something only the answer to a range request can say: an HDF5 leaf or a # `.b2z` member reports one and is streamed all the same try: - # Nor is a frame of small chunks worth an index read: blosc2 declines - # to take a chunk below BLOCK_MIN_CBYTES apart, so nothing here would - # ever use a block, and the dataset keeps the behaviour it had before. - # Inside the `try`, since `api/info` need not carry what these read. - nchunks = math.prod(math.ceil(s / c) for s, c in zip(self.shape, self.chunks, strict=True)) - if not nchunks or self.cbytes / nchunks < blosc2.proxy_source.BLOCK_MIN_CBYTES: - return None source = C2NDSource(self, max_concurrency=REMOTE_MAX_CONCURRENCY) source.adopt_index(self._pending_index) return source - except _NotRanged as exc: + except NotRanged as exc: + # PartsMissing among them, which carries no status and so is not + # transient: an answer that cannot be taken apart is one to stop + # asking, and whole chunks read the dataset either way return _UNTRIED if exc.transient else None except httpx.HTTPStatusError as exc: # A busy or broken subscriber said nothing about how this is served - status = exc.response.status_code - return _UNTRIED if status >= 500 or status == 429 else None + return _UNTRIED if _is_transient(exc.response.status_code) else None except httpx.TransportError: # Nothing was downloaded to find this out, so asking again is cheap return _UNTRIED except ( - _PartsMissing, ValueError, NotImplementedError, RuntimeError, @@ -840,29 +893,62 @@ def max_ranges(self) -> int: def chunk_layout(self, nchunk: int): """Where the blocks of a chunk are; see :meth:`ByteRangeNDSource.chunk_layout`.""" - return self._ranged().chunk_layout(nchunk) + with self._ranged() as source: + return source.chunk_layout(nchunk) def chunk_layouts(self, nchunks: Sequence[int]) -> list: """The same for several chunks; see :meth:`ByteRangeNDSource.chunk_layouts`.""" - return self._ranged().chunk_layouts(nchunks) + with self._ranged() as source: + return source.chunk_layouts(nchunks) def block_plan(self, nchunk: int, nblocks: Sequence[int]) -> list[tuple[int, int, tuple]]: """The range reads covering *nblocks*; see :meth:`ByteRangeNDSource.block_plan`.""" - return self._ranged().block_plan(nchunk, nblocks) + with self._ranged() as source: + return source.block_plan(nchunk, nblocks) def read_range(self, offset: int, size: int) -> bytes: """The bytes at [*offset*, *offset* + *size*) of the remote frame.""" - return self._ranged().read_range(offset, size) + with self._ranged() as source: + return source.read_range(offset, size) def read_ranges(self, spans: Sequence[tuple[int, int]]) -> list[bytes]: """The bytes of every span, in one request where the subscriber allows it.""" - return self._ranged().read_ranges(spans) - - def _ranged(self) -> C2NDSource: + with self._ranged() as source: + return source.read_ranges(spans) + + @contextmanager + def _ranged(self): + """The block source, retired if it turns out to serve ranges no longer. + + The subscriber can stop serving a dataset from a file between one fetch + and the next -- replaced by a lazy expression, moved into a container it + streams out of -- and the answer to a range request is where that shows. + A refusal that says so for good puts the array back where it was before + any of this: `get_chunk`, which works for every dataset there is. One + that says the server was busy leaves the source alone, since the next + request may well be answered. + + The exception is raised either way: a `Proxy` catches it and fetches the + chunks it was after whole, and a caller reading ranges directly is + entitled to hear that the ranges are gone. + """ source = self.block_source() if source is None: - raise ValueError(f"{self.path} is not served in byte ranges by {self.urlbase}") - return source + # A `NotRanged`, which is a `ValueError`: a fetch that finds the + # source retired under it -- by another thread of the same wave -- + # falls back to whole chunks rather than failing, exactly as the + # thread that retired it does + raise NotRanged(f"{self.path} is not served in byte ranges by {self.urlbase}") + try: + yield source + except PartsMissing: + # One answer that did not carry its bytes, which the source itself + # answers by asking for a range at a time; nothing about the dataset + raise + except NotRanged as exc: + if not exc.transient: + self._block_source = None + raise @property def shape(self) -> tuple[int]: diff --git a/src/blosc2/proxy.py b/src/blosc2/proxy.py index 571b29314..005539127 100644 --- a/src/blosc2/proxy.py +++ b/src/blosc2/proxy.py @@ -28,6 +28,7 @@ REMOTE_MAX_CONCURRENCY, ByteRangeNDSource, FsspecNDSource, + NotRanged, ProxyNDSource, ProxySource, _chunk_payloads, @@ -186,9 +187,15 @@ def __init__( # empty ones the cache starts life with. Hence an explicit bitmap. A # source that serves blocks needs one bit per block, since a chunk in the # cache may hold only some of them and reads zeros for the rest. - serves_blocks = all( - callable(getattr(self.src, name, None)) - for name in ("wants_blocks", "chunk_layout", "block_plan", "read_range") + # Having the block methods is not the same as being able to use them for + # this dataset: a source that knows it is served whole says so with + # `serves_blocks`, and then the block path is never taken and the bitmap + # is the chunkwise one an older blosc2 also reads. Asked rather than + # tried, since trying costs a request at the moment a proxy is built -- + # which a proxy over a cache that already holds what is wanted never pays. + serves_blocks = getattr(self.src, "serves_blocks", True) and all( + getattr(self.src, name, None) is not None + for name in ("blocks_per_chunk", "wants_blocks", "chunk_layout", "block_plan", "read_range") ) self._blocks_per_chunk = self.src.blocks_per_chunk if serves_blocks else 1 # Blocks of the last few partly filled chunks, so a rewrite need not read @@ -522,8 +529,11 @@ def fetch( A source that can serve individual blocks (:ref:`FsspecNDSource`) is asked only for the blocks the slice touches, rather than for whole chunks, whenever that is the cheaper way round -- see the thresholds in - `blosc2.proxy`. The chunks left in the cache then hold only those - blocks, and read as zeros elsewhere until the rest are fetched. + `blosc2.proxy_source`. The chunks left in the cache then hold only those + blocks, and read as zeros elsewhere until the rest are fetched. A source + that stops answering range reads partway (a subscriber that now computes + the dataset, or is too busy to serve it from its file) does not fail the + fetch: the chunks it was asked for come whole instead. Examples -------- @@ -539,7 +549,14 @@ def fetch( [4 5]] """ if self._blocks_per_chunk > 1: - return self._fetch_by_block(item, max_concurrency) + try: + return self._fetch_by_block(item, max_concurrency) + except NotRanged: + # The transport stopped answering range reads partway: whatever + # arrived before it did is in the cache and stays there, and what + # is still missing falls through to whole chunks below, which is + # the one way of reading a source that never goes away + pass missing = self._missing_chunks(item) try: diff --git a/src/blosc2/proxy_source.py b/src/blosc2/proxy_source.py index 2949a96f4..18648d3b0 100644 --- a/src/blosc2/proxy_source.py +++ b/src/blosc2/proxy_source.py @@ -91,23 +91,80 @@ def batched(iterable, n): BLOCK_HOT_CHUNKS = 8 +def _is_transient(status: int | None) -> bool: + """Whether a status says the server was busy, rather than answering. + + Both places that decide what a refused range read means -- whether to try + again at all, and whether the dataset is streamed for good -- ask this one + question, so that adding a status to it cannot leave the two disagreeing. + """ + return status is not None and (status >= 500 or status == 429) + + +class NotRanged(ValueError): + """A transport that reads byte ranges answered with something other than one. + + Raised out of :meth:`ByteRangeNDSource.read_range` and its neighbours when + the answer is not the bytes that were asked for: an HTTP 200 carrying the + whole dataset, a busy server, a body that cannot be taken apart. It is not + fatal to a fetch -- :meth:`Proxy.fetch` catches it and asks for the chunks + it wanted whole, which every source can serve -- so what it costs is the + block granularity, not the data. + + A `ValueError`, which is what a source that cannot be read in ranges raised + before this had a type of its own. + """ + + def __init__(self, message: str, status: int | None = None): + super().__init__(message) + self.status = status + + @property + def transient(self) -> bool: + """Whether asking again could be answered differently. + + A 200 is the dataset itself, streamed, and no amount of asking again + will make it a file; a server that is busy or broken says nothing at all + about how the dataset is served, and costs no download to ask twice. + """ + return _is_transient(self.status) + + +class PartsMissing(NotRanged): + """A multi-range answer did not carry all the bytes that were asked for. + + A :class:`NotRanged` for the caller that only wants to know the read failed, + and its own type for the transport, which answers it by asking for one range + at a time rather than by giving up on ranges. + """ + + class ProxyNDSource(ABC): """ Base interface for NDim sources in :ref:`Proxy`. A source may also serve single *blocks* rather than whole chunks, which is worth doing when a fetch costs a round trip and a slice touches little of the - chunks it lands in. :ref:`Proxy` uses that path when the source has all four + chunks it lands in. :ref:`Proxy` uses that path when the source has all five of ``blocks_per_chunk``, ``wants_blocks(nchunk, nwanted)``, ``chunk_layout(nchunk)``, ``block_plan(nchunk, nblocks)`` and ``read_range(offset, size)``; :ref:`FsspecNDSource` implements them over byte ranges and is the worked example. + Having them is not the same as being able to use them for the dataset in + hand: a source that knows, without asking anything, that this one is served + whole says so with ``serves_blocks``, and :ref:`Proxy` then keeps to chunks + from the start rather than to a per-block bitmap it would never fill. It is + read once, when the proxy is built, and a source without it counts as True. + A source whose transport can ask for several ranges at once says so with ``max_ranges`` and serves ``read_ranges(spans)`` and ``chunk_layouts(nchunks)`` as well; :ref:`Proxy` then sends a whole wave of reads as one request. Both are optional, and a source without them is asked one range at a time exactly as before. + + A block read that the transport cannot answer raises ``NotRanged``, and + :ref:`Proxy` then fetches the chunks it was after whole. """ @property @@ -469,6 +526,15 @@ class ByteRangeNDSource(ProxyNDSource): stamp = None + serves_blocks = True + """That blocks are worth asking this source for, which reading a frame is. + + The frame is there to be read in pieces -- that is what an open of one + settles -- so a :ref:`Proxy` over it goes straight to the block path. A + source that only sometimes serves blocks (:ref:`C2Array`, whose subscriber + may compute the dataset rather than store it) overrides this. + """ + max_ranges = 1 """How many ranges one request of this transport may carry. diff --git a/tests/ndarray/test_c2array_blocks.py b/tests/ndarray/test_c2array_blocks.py index 6fb490800..e8a967e1b 100644 --- a/tests/ndarray/test_c2array_blocks.py +++ b/tests/ndarray/test_c2array_blocks.py @@ -46,6 +46,8 @@ def __init__( multipart=True, merge_ranges=True, fetch_failures=0, + bad_parts=0, + geometry=True, ): self.fetch_failures = fetch_failures # answer this many fetches 503 first self.path = str(path) @@ -53,7 +55,10 @@ def __init__( self.ranges = ranges # False: stream the body and ignore Range, as a self.cookie = cookie # computed dataset does self.multipart = multipart # False: answer only the first range asked for + self.bad_parts = bad_parts # ... and this many multi-range answers do too self.merge_ranges = merge_ranges # as Starlette does with ranges that touch + self.geometry = geometry # False: report a dataset the subscriber computes, + # which has no chunks or blocks of its own to report self.log = [] # (endpoint, status, bytes served) self.reload() @@ -76,6 +81,16 @@ def reload(self): @property def meta(self): schunk = self.array.schunk + if not self.geometry: + # What a lazy expression looks like: an expression and its operands + # where a stored dataset has a partitioning + return { + "shape": list(self.array.shape), + "dtype": str(self.array.dtype), + "expression": "a + 1", + "operands": {"a": "@public/other.b2nd"}, + "schunk": {"cparams": {"typesize": self.array.dtype.itemsize}}, + } return { "shape": list(self.array.shape), "chunks": list(self.array.chunks), @@ -154,7 +169,11 @@ def _fetch(self, sub): merged[-1] = (merged[-1][0], max(merged[-1][1], end)) else: merged.append((start, end)) - if len(merged) == 1 or not sub.multipart: + partial = not sub.multipart + if len(merged) > 1 and sub.bad_parts: + sub.bad_parts -= 1 # an answer that carries only the first part, once + partial = True + if len(merged) == 1 or partial: start, end = merged[0] # ... and answers a plain 206 when one is left self._send( 206, @@ -333,6 +352,23 @@ def test_small_chunks_are_fetched_whole(subscriber): assert [kind for kind, _, _ in sub.log] == ["chunk"] +def test_a_dataset_that_serves_no_blocks_keeps_the_chunkwise_bitmap(tmp_path, subscriber): + # Nothing will ever ask this one for a block, so its cache records chunks: + # the bitmap an older blosc2 also reads, and none of the per-block + # bookkeeping that would be kept only to say `all of them` every time + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + assert not array.serves_blocks # decided from api/info, without a request + cache = str(tmp_path / "chunkwise-cache.b2nd") + p = blosc2.Proxy(array, urlpath=cache, mode="w") + assert p._blocks_per_chunk == 1 + + p.fetch((slice(0, 5), slice(0, 10))) + kept = p.schunk.vlmeta.getall() + assert "proxy-fetched" in kept + assert "proxy-fetched-blocks" not in kept + + def test_blocks_accumulate_in_a_chunk(subscriber, any_chunk_wants_blocks): data = _incompressible((200, 200)) array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) @@ -723,18 +759,163 @@ def test_a_streamed_dataset_is_not_asked_again(subscriber, any_chunk_wants_block assert not any(status == 206 for _, status, _ in sub.log) -def test_a_part_that_ends_early_is_refused(subscriber, any_chunk_wants_blocks): +def test_a_dataset_that_stops_being_stored_falls_back_to_chunks(subscriber, any_chunk_wants_blocks): + # A subscriber can stop serving a dataset from a file between one fetch and + # the next -- replaced by a lazy expression, moved into a container it + # streams out of. The fetch that runs into it reads the chunks it was after + # whole rather than failing, and nothing asks for a range again + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is not None + + sub.ranges = False + sub.log.clear() + assert np.array_equal(p[50:55, 0:10], data[50:55, 0:10]) + assert array.block_source() is None # retired: whole chunks read every dataset + assert _bytes(sub, "chunk") + assert np.array_equal(p[...], data) + # The one refused request is the whole of what the change cost: the body it + # answered with was never read, and nothing asked for a range again + assert [status for kind, status, _ in sub.log if kind == "fetch"] == [200] + + +def test_a_subscriber_too_busy_for_a_range_keeps_its_source(subscriber, any_chunk_wants_blocks): + # The other half of the rule that governs the probe: a 503 says nothing about + # how the dataset is served, so the fetch falls back for now and the next one + # asks for blocks again + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + p = blosc2.Proxy(array, mode="w") + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + source = array.block_source() + + sub.fetch_failures = 1 + sub.log.clear() + assert np.array_equal(p[50:55, 0:10], data[50:55, 0:10]) + assert _bytes(sub, "chunk") # served whole, since the range was refused + assert array.block_source() is source + + sub.log.clear() + assert np.array_equal(p[0:5, 100:110], data[0:5, 100:110]) + assert not _bytes(sub, "chunk") # ... and blocks are asked for again + + +def test_a_proxy_over_a_cache_survives_a_dataset_that_became_computed( + tmp_path, subscriber, any_chunk_wants_blocks +): + # `blosc2.open` rebuilds the proxy over its own cache, and the source it + # rebuilds may by then be a dataset the subscriber computes -- which reports + # no partitioning at all, so nothing may go asking one for it + data = _incompressible((200, 200)) + array, sub = subscriber(data, chunks=(100, 200), blocks=(10, 20)) + cache = str(tmp_path / "computed-cache.b2nd") + blosc2.Proxy(array, urlpath=cache, mode="w").fetch((slice(0, 5), slice(0, 10))) + + sub.geometry = False + assert not blosc2.C2Array(array.path, urlbase=array.urlbase).serves_blocks + reopened = blosc2.open(cache) + assert isinstance(reopened, blosc2.Proxy) + assert np.array_equal(reopened[0:5, 0:10], data[0:5, 0:10]) # out of the cache + + +def test_one_answer_that_misses_its_parts_does_not_end_batching(subscriber, any_chunk_wants_blocks): + # Batching is worth an order of magnitude, so a single truncated answer is + # worth retrying a range at a time rather than giving up the whole of it + data = _incompressible((400, 400)) + array, sub = subscriber(data, chunks=(200, 200), blocks=(10, 20), bad_parts=1) + p = blosc2.Proxy(array, mode="w") + + item = (slice(190, 210), slice(190, 210)) # a corner of each of the four chunks + assert np.array_equal(p[item], data[item]) + assert not sub.bad_parts # the answer that carried one part was asked for ... + assert array.max_ranges > 1 # ... and cost the batching nothing + assert np.array_equal(p[...], data) + + +def test_a_part_that_ends_early_is_refused(): # A part that starts inside a span and stops short of its end would slice to # a short payload, which is spliced against a `bstarts` promising the whole # of it -- so it is refused, and the fetch falls back to a range per request - parts = [(100, b"12345")] + parts = [(100, b"12345", 200)] # (where it starts, its bytes, the frame's length) assert blosc2.c2array._span_of(parts, 100, 5, "url") == b"12345" - with pytest.raises(blosc2.c2array._PartsMissing): + with pytest.raises(blosc2.proxy_source.PartsMissing): blosc2.c2array._span_of(parts, 100, 6, "url") - with pytest.raises(blosc2.c2array._PartsMissing): + with pytest.raises(blosc2.proxy_source.PartsMissing): blosc2.c2array._span_of(parts, 103, 4, "url") +class _Answer: + """The little of a response that taking a multipart body apart reads.""" + + def __init__(self, content, **headers): + self.content = content + self.headers = {name.replace("_", "-"): value for name, value in headers.items()} + + +def test_a_payload_that_spells_the_boundary_is_read_whole(): + # Compressed bytes hold whatever they hold, the boundary and a trailing CRLF + # among them: each part is cut to the length its own Content-Range gives, so + # what the data happens to spell decides nothing + payload = b"--c2boundary\r\n\r\nnot a boundary at all\r\ntail" + body = ( + b"--c2boundary\r\nContent-Type: application/octet-stream\r\n" + + f"Content-Range: bytes 64-{64 + len(payload) - 1}/1000\r\n\r\n".encode() + + payload + + b"\r\n--c2boundary--\r\n" + ) + answer = _Answer(body, content_type="multipart/byteranges; boundary=c2boundary") + assert blosc2.c2array._byteranges(answer) == [(64, payload, 1000)] + assert blosc2.c2array._span_of([(64, payload, 1000)], 64, len(payload), "url") == payload + + +def test_a_multipart_answer_is_taken_apart_in_order(): + parts = [(0, b"first-part-bytes"), (500, b"second\r\npart")] + body = b"" + for start, data in parts: + body += b"--sep\r\n" + f"Content-Range: bytes {start}-{start + len(data) - 1}/1000\r\n\r\n".encode() + body += data + b"\r\n" + body += b"--sep--\r\n" + answer = _Answer(body, content_type='multipart/byteranges; boundary="sep"') + assert blosc2.c2array._byteranges(answer) == [(0, parts[0][1], 1000), (500, parts[1][1], 1000)] + + +def test_a_part_without_a_content_range_is_refused(): + body = b"--sep\r\nContent-Type: application/octet-stream\r\n\r\nbytes\r\n--sep--\r\n" + answer = _Answer(body, content_type="multipart/byteranges; boundary=sep") + with pytest.raises(blosc2.proxy_source.NotRanged, match="Content-Range"): + blosc2.c2array._byteranges(answer) + + +def test_a_part_that_ends_where_the_frame_does_is_kept(): + # ... unless it ends because the frame does, which is the one short read + # `read_range` allows: a frame shorter than the prefetch an open asks for + # comes back clipped, and refusing it would write the dataset off as streamed + parts = [(100, b"12345", 105)] + assert blosc2.c2array._span_of(parts, 100, 8192, "url") == b"12345" + assert blosc2.c2array._span_of(parts, 103, 8192, "url") == b"45" + with pytest.raises(blosc2.proxy_source.PartsMissing): + blosc2.c2array._span_of(parts, 105, 1, "url") # nothing there to be short of + # A server that will not say how long the whole is leaves nothing to check a + # short answer against, so a short answer is a missing one + with pytest.raises(blosc2.proxy_source.PartsMissing): + blosc2.c2array._span_of([(100, b"12345", None)], 100, 6, "url") + + +def test_a_small_frame_is_read_over_ranges(subscriber, any_chunk_wants_blocks): + # The whole of the frame arrives in the first read an open asks for, which + # is the clipped answer above, and the dataset is served in blocks all the same + data = np.arange(200, dtype="i4").reshape(20, 10) + array, sub = subscriber(data, chunks=(10, 10), blocks=(5, 10)) + assert len(sub.frame) < blosc2.proxy_source._FRAME_PREFETCH + p = blosc2.Proxy(array, mode="w") + + assert np.array_equal(p[0:5, 0:10], data[0:5, 0:10]) + assert array.block_source() is not None + assert not _bytes(sub, "chunk") + + def test_the_shared_client_keeps_no_cookies(): # One client serves every C2Array, so a `Set-Cookie` kept from any of them # would start authorizing requests that asked for none